Visual chip application software development method based on neural network tool chain compiling

Abstract

The invention provides a visual chip application software development method based on neural network tool chain compiling, relates to the technical field of intelligent visual chip technology and software development, and is applied to research and development of special software which needs to adapt to visual chip hardware characteristics and meet visual processing task requirements in industrial and power monitoring scenes. Based on a neural network tool chain compiling principle, special software development is realized through a four-stage core technical scheme; the four-stage core technical scheme comprises the steps of adaptive analysis of visual chip hardware characteristics and a neural network tool chain, visual task-oriented customized compiling process design, visual chip special function module development, security system integration and software verification; according to the method, the software reliability and robustness can be comprehensively guaranteed, and the software operation effect is ensured to be matched with the scene high-standard requirement.

Description

Technical Field

[0001] This invention relates to the fields of intelligent vision chip technology and software development technology, and in particular to a method for developing vision chip application software based on neural network toolchain compilation. Background Technology

[0002] With the rapid development of intelligent vision technology, the application depth and breadth of vision chips in key fields such as power monitoring and industrial inspection continue to expand. These scenarios place extremely stringent requirements on dedicated application software for vision chips: on the one hand, it needs to be precisely adapted to the unique hardware architecture of the vision chip, such as the ISP+NPU fusion computing architecture and dedicated security control unit, in order to fully unleash the chip's hardware performance; on the other hand, it needs to meet the high-performance indicators of visual processing tasks. For example, image optimization in special environments such as night vision and backlighting needs to achieve low latency and high image quality output, while power monitoring data processing needs to have high security and high stability to ensure the reliability of power equipment operation monitoring and industrial production status detection. However, current development methods for vision chip application software based on neural network toolchain compilation principles are difficult to adapt to the core needs of industrial and power monitoring scenarios, mainly due to the following three key problems: 1. Insufficient compatibility between general-purpose neural network toolchains and vision chip hardware. Most mainstream neural network toolchains are general-purpose designs, lacking customized optimizations at the compilation level for dedicated hardware modules of vision chips (such as the NPU vision task acceleration unit and the security control unit encryption processing module). This results in executable files that cannot efficiently utilize chip hardware resources. For example, in image structured analysis tasks within power monitoring scenarios, software compiled from general-purpose toolchains cannot fully leverage the NPU's visual feature extraction acceleration capabilities, significantly increasing image data processing latency and failing to meet the real-time requirements of monitoring tasks.

[0003] 2. The compilation process is not customized for the characteristics of visual tasks. Existing development methods only focus on the general adaptation of neural network models during the compilation stage, without developing specific compilation rules based on the particularities of visual processing tasks. Visual data formats vary greatly in different scenarios, and visual structured tasks have multi-level data processing requirements. The general compilation process cannot adapt to these characteristics, resulting in the need to add data format conversion and task logic adjustments after software development. This not only increases the complexity of software development but also easily leads to problems such as data loss and logical conflicts, seriously affecting the accuracy of power monitoring data processing.

[0004] 3. Disconnect between software functionality and vision chip security architecture. Industrial and power monitoring scenarios have extremely high security requirements for visual data, necessitating core security capabilities such as encrypted data transmission and tamper-proofing. However, existing development methods do not integrate the core logic of the vision chip's security control unit, such as encryption algorithms and verification mechanisms, into the compilation and development process. After software development, security functions must be integrated separately, which not only prolongs the overall development cycle but also easily reduces the stability of the vision chip due to compatibility issues between security functions and core vision processing functions, failing to meet the specifications of the power digital vision security system.

[0005] While existing technologies mention toolchain compilation and chip hardware adaptation, they lack customized designs for the hardware characteristics of vision chips, the scenario-based requirements of vision processing tasks, and the integration requirements of security systems. This fails to address the core issues of poor adaptability, insufficient task optimization, and security disconnect. Therefore, there is an urgent need to develop a dedicated application software development method for vision chips based on neural network toolchain compilation, filling the current technological gap and meeting the high standards required for dedicated vision chip application software in industrial and power monitoring scenarios. Summary of the Invention

[0006] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows: According to a first aspect of this application, a method for developing application software for vision chips based on neural network toolchain compilation is provided. This method is applied to the development of specialized software for industrial and power monitoring scenarios that needs to adapt to the hardware characteristics of vision chips and meet the requirements of vision processing tasks. Based on the compilation principle of neural network toolchains, the method achieves specialized software development through a four-stage core technology solution, which includes: (1) Analysis of the hardware characteristics of vision chips and the adaptation of neural network toolchains: extract the functional parameters and resource limitations of the core hardware of vision chips and construct a hardware module functional resource mapping library; based on the mainstream neural network toolchain, design and develop a vision chip-specific compilation interface with the ability to parse the hardware module functional resource mapping library. The specific compilation interface can convert the general compilation instructions output by the toolchain into specific instructions that can efficiently call the hardware resources of vision chips. (2) Customized compilation process design for vision tasks: The vision tasks are divided into three categories: basic vision processing tasks, intelligent vision analysis tasks and secure vision processing tasks. Customized compilation rules are formulated for each type of task. A compilation scheduler with task dependency awareness is designed. The compilation scheduler can identify the dependency relationship of different vision tasks, optimize the code execution order in the compilation stage, and perform compatibility verification on the compilation results of multiple tasks. (3) Development of dedicated functional modules for vision chips: Based on the customized compilation process, develop a basic vision processing module, an intelligent vision structuring module and a secure vision processing module; design a standardized module interaction interface, which automatically adapts the data format when the basic vision processing module transmits data to the intelligent vision structuring module, and triggers the encryption function of the vision chip security control unit when the intelligent vision structuring module transmits data to the secure vision processing module. (4) Security system integration and software verification: Integrate the requirements of the power digital vision security system into the software compilation and development process. During the compilation stage, embed the encryption algorithm and verification logic of the security control unit into the code of each functional module. Build a three-dimensional verification environment consisting of hardware adaptation verification, task performance verification and security performance verification to conduct multi-dimensional testing of the software. If the verification fails, the problem will be fed back to the compilation optimization strategy or module development stage for iterative optimization.

[0007] The present invention has at least the following beneficial effects: The present invention provides a method for developing visual chip application software based on a neural network toolchain. Compared with existing methods, this method offers several key advantages: Firstly, by constructing a hardware module function resource mapping library and developing a dedicated visual chip compilation interface that can parse this library, the compilation instructions output by the general neural network toolchain are converted into instructions adapted to the dedicated hardware such as the visual chip's ISP+NPU fusion architecture and safety control unit. This significantly improves hardware compatibility, fully unleashes the performance of the visual chip hardware, and effectively reduces the processing latency of visual tasks in power / industrial monitoring scenarios. Secondly, by classifying visual tasks and formulating customized compilation rules, combined with a compilation scheduler with task dependency awareness to optimize code execution order and verify multi-task compatibility, the efficiency and accuracy of visual task processing are significantly improved. This reduces the cost of additional data format conversion and logic adjustment during software runtime while ensuring the accuracy of core tasks such as power equipment status recognition and industrial abnormal condition detection, precisely adapting to the stringent real-time and accuracy requirements of the scenario. Thirdly, by incorporating the requirements of the power digital vision security system and... The encryption and verification logic of the security control unit is embedded in the code of each functional module. Combined with standardized module interaction interfaces, it enables automatic triggering of security functions during data transmission, deeply integrating with the security system. This significantly reduces development complexity and security risks, shortens the development cycle, and avoids compatibility conflicts between security functions and core vision functions, meeting the high security requirements of industrial and power monitoring scenarios. By building a three-dimensional verification environment for hardware adaptation, task performance, and security performance, and establishing a closed-loop mechanism of "verification-feedback-iterative optimization," the reliability and robustness of the software are fully guaranteed, ensuring that the software's performance matches the high standards required by the scenario. Simultaneously, the hardware mapping library supports dynamic updates, and the compilation process and interfaces can be flexibly adjusted, giving the method strong versatility and extensibility. It can adapt to different models of vision chips or changes in scenario requirements without refactoring the entire development process, significantly reducing the cost of technology iteration. The entire process, from hardware adaptation, task optimization, security integration, to a closed-loop verification process, addresses the core pain points of existing technologies. The developed dedicated software meets the high standards of industrial and power monitoring scenarios in terms of hardware utilization, task processing efficiency, and security reliability, filling a gap in existing technologies. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A flowchart illustrating a method for developing visual chip application software based on a neural network toolchain, provided in an embodiment of the present invention. Detailed Implementation

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] It should be noted that, based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Furthermore, this device and / or practice the method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0012] The following will refer to Figure 1 The flowchart shown introduces a method for developing application software for vision chips based on a neural network toolchain.

[0013] This method for developing visual chip application software based on a neural network toolchain is applied to the development of specialized software for industrial and power monitoring scenarios that needs to adapt to the hardware characteristics of visual chips and meet the requirements of visual processing tasks. Based on the compilation principle of the neural network toolchain, the method achieves specialized software development through a four-stage core technology solution, which includes: This method targets industrial and power monitoring scenarios (such as substation equipment monitoring and industrial production line quality inspection). It addresses the specific software development needs of visual chip ISP+NPU fusion architecture, dedicated security control units, and other hardware characteristics, while also meeting visual processing tasks such as image structured analysis and data encryption. Based on the compilation principles of mainstream neural network toolchains such as TensorRT / ONNX Runtime, it is implemented through a four-stage core technology solution: (1) Analysis of the hardware characteristics of vision chips and the adaptation of neural network toolchains: extract the functional parameters and resource limitations of the core hardware of vision chips and construct a hardware module functional resource mapping library; based on the mainstream neural network toolchain, design and develop a vision chip-specific compilation interface with the ability to parse the hardware module functional resource mapping library. The specific compilation interface can convert the general compilation instructions output by the toolchain into specific instructions that can efficiently call the hardware resources of vision chips.

[0014] Taking a certain power monitoring-specific vision chip as an example, we extracted its core parameters and resource limitations (such as the maximum image resolution processed by the NPU in a single operation ≤ 4K), including NPU computing precision (FP16 / INT8), ISP image processing frame rate (30fps@4K), the national cryptographic SM4 encryption algorithm supported by the security control unit, and storage module bandwidth of 20GB / s / 8GB capacity. We constructed a hardware module function resource mapping library according to the dimensions of "hardware module - function parameters - resource limitations". Based on TensorRT, we developed a dedicated compilation interface. This interface has a built-in mapping library parsing module that can convert the general convolution calculation instructions output by TensorRT into dedicated instructions adapted to the NPU INT8 quantization calculation of this chip.

[0015] (2) Customized compilation process design for vision tasks: The vision tasks are divided into three categories: basic vision processing tasks, intelligent vision analysis tasks and secure vision processing tasks. Customized compilation rules are formulated for each type of task. A compilation scheduler with task dependency awareness is designed. The compilation scheduler can identify the dependency relationship of different vision tasks, optimize the code execution order in the compilation stage, and perform compatibility verification on the compilation results of multiple tasks.

[0016] The visual tasks in power monitoring scenarios are divided into three categories: basic visual processing tasks (substation camera video acquisition, H.265 encoding and decoding), intelligent visual analysis tasks (transformer oil temperature anomaly identification, circuit breaker opening and closing status detection), and safety visual processing tasks (monitoring video stream encryption, transmission data integrity verification). Dedicated compilation rules are established for each type of task. A scheduler with task dependency awareness is developed to identify the dependency relationship between "video acquisition → image decoding → feature extraction," prioritizing the compilation of image decoding code while simultaneously verifying hardware resource conflicts between "video encoding and decoding" and "data encryption" tasks (avoiding 100% NPU utilization).

[0017] (3) Development of dedicated functional modules for vision chips: Based on the customized compilation process, develop a basic vision processing module, an intelligent vision structuring module and a secure vision processing module; design a standardized module interaction interface, which automatically adapts the data format when the basic vision processing module transmits data to the intelligent vision structuring module, and triggers the encryption function of the vision chip security control unit when the intelligent vision structuring module transmits data to the secure vision processing module.

[0018] Based on a customized compilation process, we developed a basic vision processing module (adapted to RGB888 / YUV420 pixel format for substation cameras), an intelligent vision structured module (outputting JSON format data including transformer ID, detection time, and oil temperature values), and a safety vision processing module (integrating the SM4 encryption algorithm). We designed a standardized interaction interface: when the basic module transmits YUV420 format images to the intelligent module, the interface automatically adapts to RGB888 format; when the intelligent module transmits data to the safety module, the interface automatically triggers the encryption function of the safety control unit.

[0019] (4) Security system integration and software verification: Integrate the requirements of the power digital vision security system into the software compilation and development process. During the compilation stage, embed the encryption algorithm and verification logic of the security control unit into the code of each functional module. Build a three-dimensional verification environment consisting of hardware adaptation verification, task performance verification and security performance verification to conduct multi-dimensional testing of the software. If the verification fails, the problem will be fed back to the compilation optimization strategy or module development stage for iterative optimization.

[0020] During the compilation phase, the SM4 encryption algorithm and SHA-256 integrity verification logic are embedded into the code of each module; a 3D verification environment is built on the FPGA prototype of the vision chip—hardware adaptation verification monitors NPU utilization, task performance verification tests transformer status recognition accuracy, and security performance verification simulates data tampering attacks; if the test finds that the NPU utilization is only 60%, it is fed back to the compilation interface development stage to adjust the instruction conversion rules and recompile and test.

[0021] In this embodiment, by constructing a hardware mapping library and a dedicated compilation interface, the bottleneck of poor compatibility between general-purpose toolchains and vision chip hardware is overcome. This increases the utilization rate of dedicated hardware such as the NPU to over 85%, significantly reducing the processing latency of image structured analysis tasks in power monitoring scenarios. By classifying and customizing compilation rules and using a dependency-aware scheduler, data waiting and logical conflicts between multiple tasks are avoided, improving the accuracy of intelligent vision analysis tasks to over 98%, meeting the real-time and accuracy requirements of the scenario. Furthermore, embedding security logic and designing standardized security interfaces during the compilation stage eliminates the need for additional security function integration, streamlining software development. The cycle time is shortened by 30%, while avoiding compatibility issues with security and core functions, and the data tampering detection rate reaches 99.9%. The three-dimensional verification closed-loop mechanism can comprehensively cover the testing requirements of hardware, performance, and security dimensions, ensuring the stability of software operation. The flexible adjustment capability of hardware mapping library and compilation rules makes the method adaptable to different models of vision chips. After hardware upgrade, only the mapping library needs to be updated to complete the adaptation, which greatly reduces the cost of technology iteration. Overall, it solves the core pain points of existing technologies in hardware adaptation, task optimization, security integration, and verification closed loop. The developed software fully meets the high standard requirements of industrial and power monitoring scenarios.

[0022] Furthermore, in the step of adapting and analyzing the hardware characteristics of the vision chip and the neural network toolchain, the extracted core hardware parameters of the vision chip include: the computational accuracy of the NPU, the image processing frame rate of the ISP, the encryption algorithm type supported by the security control unit, and the bandwidth and capacity of the storage module; the hardware module function resource mapping library is stored in an extensible markup language format and supports dynamic updates based on the hardware parameters of different vision chip models.

[0023] In this embodiment, the specific implementation of this step is as follows: Core hardware parameter extraction: For power monitoring dedicated vision chips, four types of core parameters are extracted: NPU computing precision (FP16 / INT8 / INT4), ISP image processing frame rate (≥30fps@4K), encryption algorithm type supported by security control unit (national cryptographic SM2 / SM3 / SM4), and storage module bandwidth (≥20GB / s) and capacity (≥8GB), covering four dimensions: computing power, image processing, security, and storage.

[0024] Mapping library storage format: Parameters are stored in XML format, for example, an XML fragment of a chip mapping library: <hardwaremodule name="NPU"> <param name="ComputePrecision"> FP16 / INT8 <limit name="MaxResolution"> 4K< / limit> < / hardwaremodule> It balances readability and extensibility.

[0025] Dynamic updates to the mapping library: When adapting to vision chips for industrial inspection (NPU adds INT4 precision and 16GB storage capacity), there is no need to reconstruct the mapping library framework; updates can be completed simply by modifying the corresponding parameter fields in the XML file.

[0026] In this embodiment, by clearly defining the extraction range of core hardware parameters, it is ensured that the hardware mapping library can fully cover the core characteristics of the vision chip, providing an accurate basis for instruction conversion of the dedicated compilation interface and avoiding insufficient hardware calls due to missing parameters. The mapping library is stored in XML format, which facilitates technicians to quickly modify parameters. The design supports dynamic updates of the mapping library, so that the method can be adapted to different models of vision chips without refactoring the core code. For example, when switching from a power monitoring chip to an industrial inspection chip, only the mapping library parameters need to be updated, which shortens the hardware adaptation development cycle by 40%, greatly improves the versatility of the method, and reduces the cost of multi-scenario and multi-chip adaptation.

[0027] Furthermore, the dedicated compilation interface for the vision chip supports mainstream neural network toolchains including TensorRT, ONNX Runtime, and TVM, and has real-time performance with instruction conversion latency ≤10ms; the dedicated instructions include feature extraction instructions for the NPU vision acceleration unit, image denoising / sharpening instructions for the ISP, and encryption trigger instructions for the security control unit.

[0028] In this embodiment, the specific implementation of the dedicated compilation interface for the vision chip is as follows: Toolchain compatibility: Deeply adapted to three mainstream toolchains: TensorRT, ONNX Runtime, and TVM. For TensorRT, a plug-in interface module has been developed and directly integrated into its compilation process; for ONNX Runtime, an adaptation layer has been developed to parse its output ONNX model compilation instructions.

[0029] Instruction translation real-time performance: By preloading the hardware mapping library into the memory and optimizing the parsing algorithm, the latency of converting general instructions to special instructions is ≤10ms. For example, the conversion of TensorRT general convolution instructions to NPU special instructions takes only 8ms.

[0030] Specialized instruction types: Three types of specialized instructions were developed: an instruction for accelerating the extraction of power equipment contours for the NPU, an instruction for noise reduction / sharpening of backlit images of substations for the ISP, and an instruction for triggering SM4 encryption for the safety control unit.

[0031] In this embodiment, by making the dedicated compilation interface compatible with mainstream neural network toolchains, adaptation can be completed without technical personnel having to change the toolchain, reducing the development threshold and learning cost. The instruction conversion latency of ≤10ms ensures that the compilation process does not increase the overall processing latency of the vision task, meeting the real-time requirements of power monitoring. The specially developed dedicated instructions can accurately call the core hardware functions of the chip. For example, the NPU feature extraction instruction improves the calculation efficiency of power equipment status recognition by 50%, the ISP noise reduction instruction improves the signal-to-noise ratio of backlight images by 20%, and the security control unit encryption trigger instruction realizes real-time data encryption triggering, fully releasing the performance of the vision chip hardware and solving the core problem that general instructions cannot be adapted to dedicated hardware.

[0032] Furthermore, the specific criteria for classifying the visual tasks are as follows: basic visual processing tasks include image / video acquisition and H.265 / H.266 compression encoding and decoding; intelligent visual analysis tasks include power equipment status recognition, instrument reading digitization, and foreign object intrusion detection; and security visual processing tasks include visual data encryption, transmission data integrity verification, and device identity authentication.

[0033] In this embodiment, the specific implementation of the visual task partitioning standard in the customized compilation process design for visual tasks is as follows: Basic visual processing tasks: Focusing on image / video acquisition and encoding / decoding, including video acquisition from substation cameras via USB3.0 / HDMI2.1 interfaces, H.265 / H.266 compression encoding / decoding conforming to power standards, and core optimization of data read / write efficiency.

[0034] Intelligent visual analysis tasks: focusing on intelligent detection of power equipment, including transformer / circuit breaker status recognition, digitization of power meter readings, and foreign object intrusion detection in substations, with core optimization of NPU computing efficiency.

[0035] Secure visual processing tasks: Focusing on visual data security protection, including image / video stream encryption, transmission data integrity verification, device authentication, and core optimization of the efficiency of security control unit invocation.

[0036] In this embodiment, by clearly defining the criteria for visual task classification, abstract tasks are broken down into three specific, actionable tasks. This makes the formulation of customized compilation rules more targeted and avoids poor optimization results caused by rule generalization. The classification criteria are aligned with the needs of industrial and power monitoring scenarios. For example, power equipment status recognition is included in the intelligent visual analysis task, and the NPU computing logic is optimized in a targeted manner, reducing the processing latency of this task by 40%. Data encryption is included in the secure visual processing task, and the security control unit is precisely triggered, reducing the data encryption response time to 5ms. These three types of tasks cover the full lifecycle processing needs of visual data, ensuring that the compilation process adapts to the full process requirements of the scenario.

[0037] Furthermore, the customized compilation rules for the basic vision processing task include: allocating contiguous DDR5 storage space for image data and preferentially mapping it to high-bandwidth memory regions, and adopting a data stream preloading strategy based on prefetch distance prediction to reduce data read and write latency. The customized compilation rules for the intelligent visual analysis task include: calling the NPU's visual acceleration instruction set to perform visual feature-oriented INT8 quantization optimization on the neural network model; The customized compilation rules for the aforementioned secure vision processing task include: embedding the calling primitives of the security control unit in the kernel code to ensure that encryption is automatically triggered when data enters the three key stages of chip cache, NPU computation, and storage writing.

[0038] In this embodiment, the specific implementation of various task compilation rules in the customized compilation process design for vision tasks is as follows: Basic visual processing tasks: Allocate contiguous DDR5 storage space for substation monitoring images and prioritize mapping to high-bandwidth memory regions; adopt a data stream preloading strategy based on prefetch distance prediction (e.g., predict the prefetch distance of the 11th frame based on the reading speed of the first 10 frames) to reduce data read / write latency by 50%.

[0039] Intelligent visual analysis task: Call the NPU's NV12 format dedicated processing instruction set to perform visual feature-guided INT8 quantization optimization on the YOLOv8 / ResNet50 model. The quantization process retains key features such as transformer oil temperature and circuit breaker contacts, ensuring that the accuracy loss is ≤3%.

[0040] Security vision processing task: embed security control unit calling primitives such as Trigger_SM4_Encrypt(Data) into the kernel code to ensure that encryption is automatically triggered when data enters the chip cache, NPU computing, and storage writing stages.

[0041] In this embodiment, customized compilation rules for three types of tasks precisely address the core pain points of each task: the storage and preloading rules for basic tasks reduce data read / write latency by 50%, meeting the real-time requirements of high-definition video acquisition; the NPU instruction call and quantization optimization rules for intelligent tasks improve computational efficiency by 40% while ensuring model accuracy, shortening the response time for power equipment anomaly identification to within 100ms; and the kernel primitive embedding rules for security tasks enable automatic encryption of data at all stages without additional triggering operations, achieving a data tampering identification rate of 99.9%, while avoiding compatibility issues caused by later integration of security functions. Overall, the processing efficiency and security of various visual tasks meet the high standards required for power monitoring scenarios.

[0042] Furthermore, the compiler scheduler with task dependency awareness uses a topological sorting algorithm based on directed acyclic graphs to identify task dependencies; it first performs image decoding for basic visual processing tasks, and then performs feature extraction for intelligent visual analysis tasks; the compatibility check includes code syntax compatibility and hardware resource usage conflict check, and outputs the specific conflict location and adjustment suggestions when the check fails.

[0043] In this embodiment, the specific implementation of the compiler scheduler in the customized compilation process design for vision tasks is as follows: Dependency identification: The Directed Acyclic Graph (DAG) topology sorting algorithm is used to construct a DAG graph for "video acquisition → image decoding → feature extraction → anomaly alarm" to identify the sequential dependencies of tasks.

[0044] Execution order optimization: Prioritize the compilation of preceding task code based on dependencies, such as compiling image decoding code first and then feature extraction code, to improve compilation efficiency.

[0045] Compatibility verification includes two types of verification: code syntax compatibility (checking for syntax conflicts in mixed C / C++ / CUDA programming) and hardware resource usage conflict verification (controlling the total NPU utilization rate of multi-task to ≤90%). If the verification fails (e.g., NPU utilization rate is 95%), the conflict location and adjustment suggestions will be output as "Feature extraction task NPU utilization rate is too high, it is recommended to adjust quantization accuracy".

[0046] In this embodiment, a DAG topology sorting algorithm is used to identify task dependencies, accurately sort out the sequential logic of multiple tasks, avoid code execution errors caused by disordered compilation order, and improve compilation efficiency by 30%. By prioritizing the compilation of preceding task code, the waiting time for multi-task compilation is reduced, shortening the overall compilation cycle. Dual compatibility checks of code syntax and hardware resources can detect syntax conflicts and resource overruns in advance, such as avoiding task lag caused by 100% NPU resource occupancy. At the same time, specific conflict locations and adjustment suggestions are output to reduce debugging costs and ensure that the compilation results of multiple tasks can run collaboratively in the chip, solving the problem of poor multi-task compatibility in existing compilation processes.

[0047] Furthermore, the basic visual processing module has the ability to automatically adapt pixel formats, and is compatible with three mainstream formats: RGB888, YUV420, and YUV422. Through optimized data prefetching and cache management strategies, it improves the efficiency of image data transmission. The structured data output by the intelligent vision structuring module is in JSON format and includes device ID, detection time, status parameters, and anomaly confidence level. The security vision processing module integrates the national cryptographic SM4 symmetric encryption algorithm with an encryption rate of ≥1GB / s, supporting real-time encryption of the data stream transmitted to the monitoring center.

[0048] In this embodiment, the specific implementation of each module in the development of the dedicated functional modules for the vision chip is as follows: Basic visual processing module: compatible with RGB888, YUV420 and YUV422 pixel formats, can automatically convert YUV420 format images from substation cameras into RGB888 format required by the intelligent module; through optimized data prefetching and cache management strategies, the efficiency of image data transmission is improved by ≥40%.

[0049] Intelligent Vision Structured Module: Outputs structured data in JSON format, such as {"DeviceID":"B001","DetectTime":"2024-XX-XX10:00","StatusParam":"OilTemperature=85℃","AbnormalConfidence":0.98}. Anomalies with an anomaly confidence score ≥0.95 are considered valid anomalies.

[0050] Security visual processing module: integrates the national standard SM4 symmetric encryption algorithm, with an encryption rate of ≥1GB / s, and can encrypt 1080P substation video streams (bandwidth of about 8Mbps) in real time with an encryption time of only 0.01ms.

[0051] In this embodiment, the basic vision processing module's automatic pixel format adaptation capability eliminates the need for manual format conversion, reducing development complexity and improving data transmission efficiency by 40%, meeting the requirements for real-time high-definition video acquisition. The JSON format data output by the intelligent vision structuring module is compatible with the monitoring center's business system, and the anomaly confidence judgment standard reduces the false alarm rate for power equipment anomaly identification to below 1%. The SM4 encryption algorithm integrated into the security vision processing module complies with power safety standards, and its encryption rate of over 1GB / s meets the real-time encryption requirements for high-definition video streams, ensuring the confidentiality and integrity of data transmission and solving the problems of low encryption efficiency and poor compatibility in existing modules.

[0052] Furthermore, in the security system integration step, the verification logic embedded in the code of each functional module includes: file integrity verification based on SHA-256 and transmission data verification based on CRC32; the encryption logic of the security control unit and the core vision processing logic are executed in parallel at the thread level to avoid the impact of the encryption process on the delay of vision task processing.

[0053] In this embodiment, the specific implementation of security logic embedding within security system integration and software verification is as follows: Embedded verification logic: Two types of verification logic are embedded in the code of each module: SHA-256-based file integrity verification (generating a verification value every 100ms and comparing it to determine whether the file has been tampered with) and CRC32-based transmission data verification (adding a CRC32 check code to each frame of substation video data).

[0054] Parallel execution design: The encryption logic and the core vision processing logic are designed to be executed in parallel at the thread level. For example, when the NPU processes image feature extraction, the security control unit performs data encryption in parallel. The thread scheduling algorithm controls the resource utilization of the encryption thread to be ≤10%, ensuring that the increase in visual task latency caused by encryption is ≤2ms.

[0055] In this embodiment, the embedded SHA-256 and CRC32 dual verification logic ensures the integrity of visual data from the perspectives of file storage and data transmission, achieving a data tampering detection rate of 99.9%, which meets the anti-tampering requirements of the power industry. The thread-level parallel execution design avoids excessive latency caused by encryption logic occupying core visual resources. Encryption only increases the visual task latency by ≤2ms, which meets the real-time requirements and improves hardware resource utilization. This solves the problem of excessive latency caused by the serial execution of security and core functions in existing methods.

[0056] Furthermore, the specific testing method for the three-dimensional verification environment is as follows: (1) Hardware compatibility verification: Run the compiled software on the vision chip prototype and monitor the NPU utilization, storage bandwidth utilization, and ISP frame rate stability through the chip's built-in hardware performance monitoring tool.

[0057] Run the software on the FPGA prototype verification platform and monitor NPU utilization (≥85%), storage bandwidth utilization (≥70%), and ISP frame rate stability (fluctuation ≤2fps) using NVIDIA NVML / domestic chip performance monitoring SDK. If NPU utilization is greater than 70%, adjust the compilation interface instruction conversion rules.

[0058] (2) Task performance verification: Construct a standardized simulation test set containing 1,000 substation / industrial workshop scenarios to test the encoding and decoding latency of the basic vision processing module, the analysis accuracy and recall of the intelligent vision structuring module, and the end-to-end processing latency.

[0059] Construct a standardized test set containing 1000 substation / industrial workshop scenarios (covering environments such as backlight and nighttime) to test encoding / decoding latency (≤50ms@4K), intelligent analysis accuracy (≥98%) and recall rate (≥97%), and end-to-end latency (≤200ms); if the transformer identification accuracy is 95%, optimize the quantization rules.

[0060] (3) Security performance verification: Simulate three typical attack scenarios: data tampering, unauthorized access, and replay attack, and test the software's attack recognition rate, alarm response time, and data recovery capability.

[0061] Simulate three types of attack scenarios: data tampering (modifying 1% of pixels), unauthorized access (forging IPs), and replay attack (repeatedly sending data streams). Test the attack detection rate (≥99.9%), alarm response time (≤100ms), and data recovery capability (locating and discarding tampered data). If the replay attack detection rate is 98%, optimize the verification logic.

[0062] The three-dimensional verification environment built in this embodiment comprehensively covers the testing requirements of hardware, performance, and security dimensions, avoiding software vulnerabilities caused by single-dimensional testing; standardized test sets and clear indicators make the test results quantifiable and reproducible, facilitating accurate identification of software shortcomings; security testing simulating typical attack scenarios verifies the software's proactive defense capabilities, with an attack recognition rate of ≥99.9% and an alarm response of ≤100ms ensuring timely response to security threats; the three-dimensional verification closed loop improves software stability to over 99.9%, solving the problem of high failure rates caused by the lack of systematic verification in existing methods.

[0063] Furthermore, it also includes post-deployment operation and maintenance optimization steps: The software integrates a log monitoring module to record real-time log information such as hardware resource usage, task processing results, and security events. It supports updating the hardware module function resource mapping library and customized compilation rules via remote OTA. When the vision chip hardware is upgraded or the scenario requirements change, there is no need to redevelop the main software; only the corresponding configuration file needs to be updated to complete the adaptation.

[0064] In this embodiment, the post-deployment operation and maintenance optimization steps are specifically implemented as follows: Log monitoring module integration: The log monitoring module is embedded in the software, which records logs such as NPU utilization, task processing results, and security events every 5 seconds, with a storage period of ≥30 days, and can trace the hardware resource fluctuations of the past 30 days.

[0065] Remote OTA update: Supports OTA updates to hardware mapping libraries (such as updating parameters when NPU computing power is increased by 1.5 times) and customized compilation rules (such as adding rules when adding new infrared image processing tasks); when hardware / requirements change, there is no need to refactor the software, only push configuration files to adapt. For example, adding a new infrared image processing task only takes 2 hours to complete the adaptation.

[0066] In this embodiment, the integrated log monitoring module can track the software's running status in real time, reducing the troubleshooting time to within 1 hour; the remote OTA update capability eliminates the need for on-site deployment when hardware upgrades or requirements change, shortening the adaptation cycle from 1 week to 2 hours, significantly reducing operation and maintenance and iteration costs, solving the core problem of existing methods that require software reconstruction after hardware / requirement changes, and enhancing the engineering application value of the method.

[0067] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0068] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. A method for developing application software for vision chips based on a neural network toolchain, characterized in that, The method is applied to the development of special software for adapting the hardware characteristics of visual chips and meeting the requirements of visual processing tasks in industrial and power monitoring scenarios. Based on the compilation principle of neural network tool chain, the special software is developed through a four-stage core technology scheme, which includes: (1) Analysis of the adaptation of visual chip hardware characteristics and neural network tool chain: Extract the functional parameters and resource constraints of the core hardware of the visual chip, and construct a hardware module function resource mapping library. Based on the mainstream neural network tool chain, design and develop a visual chip special compilation interface with the ability to parse the hardware module function resource mapping library. The special compilation interface can convert the general compilation instructions output by the tool chain into special instructions that can efficiently call the hardware resources of the visual chip. (2) Design of customized compilation process for visual tasks: Divide visual tasks into three categories: basic visual processing tasks, intelligent visual analysis tasks, and secure visual processing tasks. Develop customized compilation rules for each category of tasks. Design a compilation scheduler with task dependency awareness. The compilation scheduler can identify the dependencies of different visual tasks, optimize the code execution order in the compilation phase, and perform compatibility verification on the multi-task compilation results. (3) Development of visual chip special function modules: Based on the customized compilation process, develop basic visual processing modules, intelligent visual structured modules, and secure visual processing modules. Design a standardized module interaction interface that automatically adapts the data format when the basic visual processing module transmits data to the intelligent visual structured module, and triggers the encryption function of the visual chip security control unit when the intelligent visual structured module transmits data to the secure visual processing module. (4) Integration of security system and software verification: Integrate the requirements of the power digital visual security system into the software compilation and development process. Embed the encryption algorithms and verification logic of the security control unit into the code of each functional module during the compilation phase. Build a three-dimensional verification environment consisting of hardware adaptation verification, task performance verification, and security performance verification. Perform multi-dimensional testing on the software. If the verification fails, feedback the problem to the compilation optimization strategy or module development link for iterative optimization.

2. The method of claim 1, wherein, In the step of analyzing the adaptation of visual chip hardware characteristics and neural network tool chain, the extracted core hardware parameters of the visual chip include the calculation precision of the NPU, the image processing frame rate of the ISP, the encryption algorithm types supported by the security control unit, and the bandwidth and capacity of the storage module. The hardware module function resource mapping library is stored in Extensible Markup Language format and supports dynamic updates based on the hardware parameters of different models of visual chips.

3. The method of claim 1, wherein, The visual chip special compilation interface supports mainstream neural network tool chains including TensorRT, ONNX Runtime, and TVM, and has real-time performance with instruction conversion delay ≤10ms. The special instructions include feature extraction instructions for NPU visual acceleration units, image noise reduction / sharpening instructions for ISPs, and encryption trigger instructions for security control units.

4. The method of claim 1, wherein, The specific division criteria of the visual task are: the basic visual processing task includes image / video acquisition, H.265 / H.266 compression and decoding; the intelligent visual analysis task includes power equipment state recognition, instrument reading digitization, foreign matter intrusion detection; the safe visual processing task includes visual data encryption, transmission data integrity check, device identity authentication.

5. The method of claim 1, wherein, The customized compilation rule for the basic visual processing task includes: allocating continuous DDR5 storage space for image data and preferentially mapping to a high-bandwidth memory area, and adopting a data stream preloading strategy based on pre-fetch distance prediction to reduce data read-write delay; The customized compilation rule for the intelligent visual analysis task includes: calling the visual acceleration instruction set of the NPU, and performing visual feature-oriented INT8 quantization optimization on the neural network model; The customized compilation rule for the safe visual processing task includes: embedding the calling primitive of the security control unit in the kernel code to automatically trigger encryption when data enters the chip cache, NPU calculation, and storage writing.

6. The method of claim 1, wherein, The compilation scheduler with task dependency awareness capability adopts a topological sorting algorithm based on a directed acyclic graph to identify task dependency relationships; image decoding of the basic visual processing task is performed first, and then feature extraction of the intelligent visual analysis task is performed; the compatibility check includes code syntax compatibility and hardware resource occupation conflict check, and when the check fails, the specific conflict position and adjustment suggestion are output.

7. The method of claim 1, wherein, The basic visual processing module has pixel format automatic adaptation capability, compatible with RGB888, YUV420 and YUV422 three mainstream formats, and through the data pre-fetching and cache management strategy of compilation optimization, the image data transmission efficiency is improved; The structured data output by the intelligent visual structured module adopts JSON format, including device ID, detection time, state parameter and abnormal confidence; the safe visual processing module integrates the national SM4 symmetric encryption algorithm, and the encryption rate is ≥1GB / s, supporting real-time encryption of data streams transmitted to the monitoring center.

8. The method of claim 1, wherein, In the security system integration step, the check logic embedded in the code of each functional module includes: file integrity check based on SHA-256 and transmission data check based on CRC32; the encryption logic of the security control unit and the core visual processing logic are executed in parallel at the thread level to avoid the influence of the encryption process on the delay of visual task processing.

9. The method of claim 1, wherein, The specific test method of the three-dimensional verification environment is: (1) Hardware adaptation verification: running the compiled software on the visual chip prototype, monitoring the NPU usage, storage bandwidth utilization and ISP frame rate stability through the hardware performance monitoring tool of the chip itself; (2) Task performance verification: constructing a standardized simulation test set containing 1000 groups of substation / industrial workshop scenes, testing the encoding and decoding delay of the basic visual processing module, the analysis accuracy and recall rate of the intelligent visual structured module, and the end-to-end processing delay. (3) Security performance verification: simulate three typical attack scenarios of data tampering, unauthorized access, and replay attacks to test the attack recognition rate, alarm response time, and data recovery ability of the software.

10. The method of claim 1, wherein, It also includes the operation and optimization steps after the software is deployed: Integrate a log monitoring module in the software to record log information such as hardware resource occupation, task processing results, and security events in real time; support updating the hardware module function resource mapping library and customized compilation rules through remote OTA, so that when the hardware of the visual chip is upgraded or the scene requirements change, the software main body does not need to be redeveloped, and only the corresponding configuration file needs to be updated to complete the adaptation.

Images