Hardware-software co-verification system and method

Abstract

The application provides a software and hardware cooperative verification system and method, and is applied to the technical field of integrated circuits. The software and hardware cooperative verification system of the application realizes real physical closed loop between a software environment and a hardware platform through a physical loop channel, and avoids the physical isolation problem in the traditional verification method. Through the tracking module, the collection and uploading of the hardware running state data can significantly improve the debugging visibility without reducing the hardware running speed, and solves the time and space mismatch problem existing in the traditional software and hardware cooperative verification. Based on the cooperative verification framework, the complex end-to-end system level task (such as long time video stream processing) can be verified in engineering, the risk of excessive hardware redundancy design and on-site trial and error is reduced, the verification gap between software and hardware is effectively reduced, and the verification efficiency and reliability of the whole end-to-end system are improved.

Description

Technical Field

[0001] This application relates to the field of integrated circuit technology, and more specifically, to a software and hardware co-verification system and method. Background Technology

[0002] End-to-End Intelligent Systems (E2E) are a design paradigm for intelligent systems that directly map the entire process from raw input to final output using a unified model. With the rapid development of artificial intelligence, integrated circuits, and embedded systems technologies, E2E is playing an increasingly important role in scenarios such as drones, autonomous driving, intelligent robots, and edge sensing terminals. E2E integrates complex hardware and software structures as well as complex interconnect structures to achieve real-time perception, decision-making, and control of the external environment. For example, common E2E systems integrate heterogeneous sensors (such as frame-based RGB sensors and dynamic vision sensors (DVS)) and heterogeneous computing units (such as convolutional neural networks (CNN) and spiking neural networks (SNN)), and there are complex interconnect structures between these heterogeneous sensors and computing units.

[0003] Because the behavior of E2E systems is highly dependent on the collaborative working method between software and hardware, effective performance verification typically requires functional verification and testing of both the software and hardware within the E2E system. Traditional verification methods usually involve verifying the software and hardware independently. However, this approach suffers from several drawbacks. Software verification lacks real hardware interfaces, bandwidth limitations, and timing constraints, while hardware verification lacks complex software loads and real software control logic. Consequently, potential problems arising during software-hardware collaboration may go undetected, reducing verification accuracy and efficiency. Summary of the Invention

[0004] In view of this, this application provides a software and hardware co-verification system and method to achieve co-verification of software and hardware in an end-to-end system, thereby improving verification accuracy and efficiency.

[0005] Specifically, this application is implemented through the following technical solution: In a first aspect, embodiments of this disclosure provide a software-hardware co-verification system, including a host computer and a hardware verification chip. The host computer constructs a software runtime environment related to the software function to be verified in the end-to-end system to be verified, and deploys the software function to be verified. The hardware verification chip deploys the hardware function to be verified corresponding to the system hardware in the end-to-end system to be verified. A physical loopback channel is pre-established between the host computer and the hardware verification chip. This physical loopback channel has data transmission characteristics that match those of the hardware verification chip, wherein: The host computer is used to receive input data, process the input data in the software running environment according to the function of the software to be verified, obtain a first processing result, and transmit the first processing result to the hardware verification chip. The hardware verification chip is used to receive the first processing result, process the first processing result according to the hardware function to be verified, and obtain a second processing result; and use the physical loopback channel to send the second processing result back to the host computer as new input data; the first processing result and the second processing result are used to verify and debug the end-to-end system to be verified.

[0006] In one possible implementation, a tracking module is also included, used to collect hardware operating status data during the process of the hardware verification chip processing the first processing result according to the function of the hardware to be verified.

[0007] In one possible implementation, the hardware operating status data includes at least the data path occupancy status, storage and interconnect bandwidth usage, task scheduling and arbitration results, the operating status of the control state machine, and operating result information.

[0008] In one possible implementation, the tracking module is further configured to send the hardware operating status data to the host computer; The host computer is further configured to verify and analyze the end-to-end system to be verified based on the second processing result, the hardware operating status data, and the first processing result, and obtain the software and hardware verification results and software and hardware analysis results of the end-to-end system to be verified.

[0009] In one possible implementation, the software and hardware verification results include software operation problems, hardware operation problems, and collaborative operation problems; the software and hardware analysis results include at least the problem location results and optimization suggestions related to the software and hardware verification results.

[0010] In one possible implementation, the host computer is further configured to optimize the software configuration parameters and hardware configuration parameters of the end-to-end system to be verified based on the software and hardware verification results and software and hardware analysis results.

[0011] In one possible implementation, the hardware verification chip includes a physical layer receiver module and a physical layer transmitter module; The physical loopback channel is located between the physical layer receiver module and the physical layer receiver module.

[0012] In one possible implementation, the hardware verification chip includes a Field Programmable Gate Array (FPGA) chip.

[0013] In one possible implementation, the input data also includes image information continuously acquired by heterogeneous sensors.

[0014] Secondly, this disclosure also provides a software-hardware co-verification method, including: Configure the target parameters corresponding to the end-to-end system to be verified into the software and hardware co-verification system described in any of the first aspects above; the target parameters include the software configuration parameters and hardware configuration parameters of the end-to-end system to be verified. The software and hardware collaborative verification system is obtained, and a first processing result and a second processing result are output for the input data; the first processing result and the second processing result are used to verify and debug the end-to-end system to be verified.

[0015] The software and hardware co-verification system and method provided in this disclosure, by designing a software and hardware co-verification system for an end-to-end system, includes a host computer for verifying the functions of the software to be verified and a hardware verification chip for verifying the functions of the hardware. A physical loopback channel matching the data transmission characteristics of the hardware verification chip is constructed between the host computer and the hardware verification chip. This not only allows the first processing result of software verification to be transmitted to the hardware in a timely manner, thereby enabling hardware verification under real complex software load and real software control logic, but also, by using the physical loopback channel to transmit the hardware verification result back, the second processing result of hardware verification can be transmitted back to the software in a timely manner for software verification under conditions close to real hardware interface, bandwidth limitations, and timing constraints. Thus, without sacrificing the physical authenticity of the hardware, rapid closed-loop co-verification between software and hardware is achieved. In summary, this application, by pre-constructing a physical loopback channel between the host computer and the hardware verification chip, enables closed-loop collaborative verification of the hardware and software in an end-to-end system without losing the physical authenticity of the hardware and ensuring the participation of complex software loads and real software control logic. This facilitates the timely detection of potential problems during hardware and software collaboration, thereby improving verification accuracy and efficiency.

[0016] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] Figure 1 This is a framework diagram of a hardware and software co-verification system illustrated in an exemplary embodiment of this application; Figure 2 This is a schematic diagram illustrating the specific architecture of a hardware and software co-verification system according to an exemplary embodiment of this application; Figure 3 This is a flowchart illustrating a hardware-software co-verification method according to an exemplary embodiment of this application; Figure 4 This is an implementation framework diagram of a software and hardware co-verification method illustrated in an exemplary embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of this application. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0019] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0021] Research has revealed that end-to-end intelligent systems are increasingly exhibiting a highly heterogeneous development trend at the architectural level. On the one hand, the sensing layer is expanding from traditional single-frame image sensors to multimodal sensing systems, such as the coexistence of frame-based RGB sensors and DVS (Digital Visual System). On the other hand, the computing layer is evolving from single CNN inference acceleration to a hybrid computing architecture that includes SNN (Synchronous Neural Network), event-driven computing, and traditional deep learning models working together. While current heterogeneous end-to-end intelligent systems can improve perception accuracy, response speed, and energy efficiency, they also significantly increase the complexity of the system in the three dimensions of sensing, computing, and the interconnection between sensing and computing, making system behavior highly dependent on the collaborative working methods between hardware and software.

[0022] Currently, common software and hardware verification methods can be divided into two categories. One is independent software and hardware verification. In this type, software verification techniques typically rely on instruction set simulators or virtual platforms, such as the Quick Emulator (QEMU). This type of verification method, for the hardware of end-to-end intelligent systems, usually employs an abstract hardware model (e.g., simulating only the hardware's functionality without considering its timing; all hardware is abstracted using software modeling). This allows for the rapid execution of complex software logic without relying on real hardware, offering advantages such as fast simulation speed and abundant debugging tools. However, because the hardware is all abstracted using software modeling, it cannot accurately reflect the bandwidth limitations, cache conflicts, interconnect arbitration, and physical timing characteristics of the real hardware system. This leads to situations where the software runs normally in the simulation environment but experiences performance degradation or system instability after deployment on real hardware. Hardware verification techniques typically rely on Register Transfer Level (RTL) simulation or Field-Programmable Gate Array (FPGA) prototyping platforms. RTL simulation can accurately verify hardware functions and timing at the signal level, but its simulation speed is usually only on the order of kilohertz (kHz). Its verification efficiency will be greatly reduced when facing complex system-level loads (such as large-scale data streams or long-running tasks (such as video stream processing lasting several seconds)). Although FPGA prototyping platforms can run hardware logic at frequencies close to those of real systems and have high physical realism, their debugging methods are limited, the observability of internal signals during the verification process is poor, and it is difficult to achieve deep integration with complex software environments. Its drawbacks are also quite obvious.

[0023] Therefore, it can be seen that the independent verification method for software and hardware, due to the use of independent technical paths for software verification and hardware verification, results in a significant physical isolation, leading to a clear separation in the verification process. This separation manifests specifically in the lack of real hardware interfaces, bandwidth limitations, and timing constraints during software verification, while hardware verification lacks the involvement of complex software loads and real software control logic. Consequently, cross-layer problems during software-hardware collaboration (such as software-hardware deadlock, bandwidth contention, and resource starvation) are difficult to detect before the deployment of the end-to-end intelligent system.

[0024] To bridge the gap between software and hardware verification and avoid verification fragmentation, another verification approach has been proposed: RTL-based software-hardware co-simulation. This approach achieves co-verification of software and hardware logic by jointly running an instruction set simulator (such as QEMU) and an RTL simulator. However, this approach is limited by the execution efficiency of RTL simulation itself. Since RTL simulation speeds are typically in the kHz range, there is a significant difference in speed compared to the megahertz (MHz) range of actual end-to-end intelligent systems. When faced with large-scale data flows or long-running tasks (such as processing video streams lasting several seconds), the time required for co-simulation becomes unacceptable and fails to meet engineering requirements. Therefore, in actual development, developers are often forced to abandon co-simulation solutions and revert to the isolated process of verifying software and hardware separately, further exacerbating the verification fragmentation problem.

[0025] In addition, some academic work has attempted to accelerate the whole-system simulation process by using parallel computing resources, such as leveraging the Graphics Processing Unit (GPU) to accelerate QEMU or multi-core system simulations. While these methods improve simulation efficiency to some extent, they are still limited by the slow simulation speed caused by the execution model of the simulation framework itself, making it difficult to complete the full verification of complex real-time tasks in engineering practice.

[0026] Based on the above research, this disclosure provides a software-hardware co-verification system and method according to embodiments of this disclosure. By designing a software-hardware co-verification system for an end-to-end system, the system includes a host computer for verifying the functions of the software to be verified and a hardware verification chip for verifying the functions of the hardware. A physical loopback channel matching the data transmission characteristics of the hardware verification chip is constructed between the host computer and the hardware verification chip. This not only allows the first processing result of software verification to be transmitted to the hardware in a timely manner, thereby enabling hardware verification under real complex software load and real software control logic, but also allows the second processing result of hardware verification to be transmitted back to the software for software verification in a timely manner under conditions close to real hardware interface, bandwidth limitations, and timing constraints. Thus, without sacrificing the physical authenticity of the hardware, rapid closed-loop co-verification between software and hardware is achieved. In summary, this application, by pre-constructing a physical loopback channel between the host computer and the hardware verification chip, enables closed-loop collaborative verification of the hardware and software in an end-to-end system without losing the physical authenticity of the hardware and ensuring the participation of complex software loads and real software control logic. This facilitates the timely detection of potential problems during hardware and software collaboration, thereby improving verification accuracy and efficiency.

[0027] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below are all contributions made by the inventor to this disclosure.

[0028] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0029] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0030] It should be noted that the specific terms mentioned in the embodiments of this disclosure include: Mobile Industry Processor Interface (MIPI) D-Type Physical Layer Interface: D-Type Physical Layer, abbreviated as D-PHY; Camera Serial Interface 2 (CSI2) AXI4-Stream: An interface standard in the bus protocol, specifically designed for high-speed data stream transmission; Physical layer protocol interface: PHY Protocol Interface, or PPI for short, is the standard interface between the MIPI CSI-2 protocol controller and the physical layer (PHY).

[0031] To facilitate understanding of this embodiment, a hardware-software co-verification system disclosed in this disclosure will first be described. For example... Figure 1 The diagram shows a framework of a hardware-software co-verification system provided in this embodiment of the present disclosure. It includes a host computer 101 and a hardware verification chip 102. The host computer 101 is used to verify various software functions in the end-to-end system to be verified, and to optimize and adjust parameters of the end-to-end system to be verified based on the verification results. The hardware verification chip 102 is used to verify various system hardware components in the end-to-end system to be verified. Optionally, the hardware verification chip 102 may include a field-programmable gate array (FPGA) chip. The end-to-end system to be verified can be any intelligent end-to-end system to be functionally verified. For any developed end-to-end system, verification is required after the design is completed, and it can only be put into actual engineering application after successful verification. The end-to-end system may include various software and various hardware components. The software may include various computing units and various neural networks, such as image processing units, tensor computing units, SNN networks, CNNs, and recurrent neural networks (RNNs). The hardware may include various functional hardware components, such as heterogeneous sensors, processors, GPUs, integrated circuit boards, embedded hardware, etc.

[0032] Specifically, the host computer 101 can construct a software runtime environment related to the software functions to be verified in the end-to-end system, and deploy the software functions to be verified. Here, the software functions to be verified refer to the various software functions included in the end-to-end system. The software runtime environment is the collection of all external conditions and resources required for the normal operation of the software functions to be verified, including both hardware resources and software infrastructure and configuration. Various software functions to be verified can be deployed in the host computer 101 to verify these functions within the host computer's software runtime environment. The hardware verification chip 102 deploys the hardware functions to be verified corresponding to the various system hardware components in the end-to-end system. Different system hardware components have different hardware functions, and the hardware functions to be verified are the functions possessed by these various system hardware components. In the end-to-end system, through the collaboration between various software functions and various hardware functions, real-time perception, decision-making, and control of the external environment are achieved.

[0033] A physical loopback channel 103 is pre-built between the host computer 101 and the hardware verification chip 102. The physical loopback channel 103 has data transmission characteristics that match those of the hardware verification chip 102. These data transmission characteristics may include at least data transmission bandwidth and data transmission latency. The physical loopback channel 103 is used to establish a high-bandwidth, low-latency data transmission path between the host computer 101 and the hardware verification chip 102, enabling the physical signal results generated by the hardware verification chip 102 after processing real or near-real input data to be transmitted back to the software runtime environment on the host computer 101 at near real-time speed.

[0034] Specifically, the host computer 101 can be used to receive input data, process the input data in the software running environment according to the function of the software to be verified, obtain the first processing result, and transmit the first processing result to the hardware verification chip 102.

[0035] Here, the input data can be data that matches the processing capabilities of the end-to-end system to be verified. This data can be test data actively input by the user, or processing results fed back by the hardware verification chip 102 through the physical loopback channel. Optionally, the input data can also include image information continuously acquired by heterogeneous sensors in the end-to-end system to be verified, such as single-frame image information or continuously acquired image information acquired by a frame-type RGB sensor and a DVS. In this way, by using image information continuously acquired by heterogeneous sensors as input data for the end-to-end system to be verified and verifying it with the help of the physical loopback channel, complex end-to-end system-level tasks (such as long-term video stream processing) can be made verifiable in engineering, reducing the risk of excessive hardware redundancy design and on-site trial and error.

[0036] The first processing result is the data processing result obtained by the host computer 101 after running the software function to be verified, processing the input data, and performing logic control. The first processing result can reflect whether there are any problems with the software function to be verified. The host computer 101 can transmit data to the hardware verification chip 102 through the transmission engine provided by the hardware verification chip 102. The transmission engine can be, for example, the Xilinx Direct Memory Access (XDMA) engine in the FPGA.

[0037] For example, after receiving the input data, the host computer 101 can run the software function to be verified in the software runtime environment, process the input data with software logic, obtain the first processing result, and then use the XDMA transmission engine to transmit the first processing result to the hardware verification chip 102.

[0038] Furthermore, the hardware verification chip 102 can be used to receive the first processing result, process the first processing result according to the hardware function to be verified, and obtain the second processing result; and use the physical loopback channel 103 to send the second processing result back to the host computer 101 as new input data; the first processing result and the second processing result are used to verify and debug the end-to-end system to be verified.

[0039] Here, the second processing result is the output of the hardware verification chip 102 after performing hardware logic processing on the first processing result according to the processing timing of each system hardware in the end-to-end system to be verified and the function of the hardware to be verified. This result serves as the physical signal result generated by the hardware verification chip 102 after processing the input data. The second processing result can reflect whether there are any problems with the system hardware and the function of the hardware to be verified. For example, when the input data is image data, the first processing result can be the image result after software processing, and the second processing result can be the image after hardware processing.

[0040] In specific implementation, after receiving the first processing result from the XDMA transmission engine, the hardware verification chip 102 can continue hardware-level processing and decision-making based on the first processing result, according to the processing timing of each system hardware and the function of the hardware to be verified, to obtain a second processing result. Then, the hardware verification chip 102 can use the physical loopback channel 103 to feed the second processing result back to the host computer 101 as new input data. Thus, the host computer 101 can obtain the hardware processing result transmitted according to the characteristics of real data transmission, and process it using the verification software function to be verified to obtain a new first processing result. The new first processing result can be directly output, or it can continue to be transmitted to the hardware verification chip 102 for cyclical collaborative verification until the verification end condition is met, obtaining each first processing result and second processing result. The verification end condition can be that the system logic of the end-to-end system to be verified has been completed, and / or that both the software function and the hardware function to be verified have been verified.

[0041] The first and second processing results can be used to verify and debug the end-to-end system to be verified. For example, the host computer 101 can perform collaborative analysis on the first and second processing results to identify problems in the end-to-end system to be verified and output adjustment suggestions.

[0042] In this way, by constructing a physical loopback channel between the host computer and the hardware verification chip, and feeding back the hardware processing results to the host computer through this physical loopback channel, the software system on the host computer can continue to make control decisions, adjust parameters, or schedule tasks based on the feedback of the actual hardware execution results. This achieves a closed-loop collaborative verification process between software and hardware without sacrificing the physical authenticity of the hardware, effectively eliminating the physical isolation between software and hardware in traditional verification methods. Furthermore, by using the data transmission characteristics of the physical loopback channel, which conform to real hardware, for data transmission between software and hardware, the spatiotemporal mismatch problem caused by the difference in execution speed between the software simulation platform and the hardware simulation platform in traditional simulation methods can be avoided, thus achieving the verifiability of complex system-level tasks.

[0043] In one embodiment, to address the problem of poor internal signal observability in existing FPGA prototyping platforms during hardware verification, this application also provides, as follows: Figure 1 The tracking module 104 shown is used to collect hardware operating status data during the process of the hardware verification chip 102 processing the first processing result according to the function of the hardware to be verified.

[0044] Here, the tracking module 104 can be pre-embedded within the hardware logic. It is used to capture key state signals during hardware operation in real time without interfering with the normal functioning of the hardware verification chip 102. The captured key state signals can be defined as hardware operating state data. For example, the tracking module 104 can be a lightweight module that consumes very few resources from the hardware verification chip 102 and can acquire key hardware operating states even when the hardware is running at full speed.

[0045] Optionally, the tracking module 104 can be a global tracking module, or it can include different sub-tracking modules for different system hardware. Each sub-tracking module is used to track the hardware operating status data of the corresponding system hardware.

[0046] The hardware operating status data can include the operating status data of the hardware verification chip 102 and the system hardware status data corresponding to the various system functions operated by the hardware verification chip 102. Specifically, the hardware operating status data can at least include data path occupancy status, storage and interconnect bandwidth usage, task scheduling and arbitration results, the operating status of the control state machine, and operating result information.

[0047] Specifically, the data path occupancy status indicates the occupancy status of the hardware's data transmission path, such as fully occupied, idle, or 50% occupied. Storage and interconnect bandwidth usage indicates the hardware's storage space usage and interconnect bandwidth usage. Task scheduling and arbitration results indicate the task scheduling status and arbitration results of each hardware component during data processing. The control state machine's operating status indicates the operating status of the hardware's critical control state machines. Operation result information indicates the hardware's operating results and hardware attribute information, which may include, for example, operational accuracy, latency, energy efficiency, and frame rate information from heterogeneous sensors.

[0048] In one embodiment, in order to further realize automated verification of the end-to-end system to be verified, the tracking module 104 of this application can also be used to send hardware operating status data to the host computer 101.

[0049] Here, the tracking module 104 also has the ability to communicate with the host computer 101 to feed back the collected hardware operating status data to the host computer 101 for analysis and verification.

[0050] In practice, the tracking module 104 can transmit the captured hardware operating status data to the software side of the host computer 101 through an independent data path or dedicated interface established with the host computer 101 for analysis and correlation, thereby providing software developers with observability close to the software debugging experience under the condition of full-speed hardware operation.

[0051] Furthermore, the host computer 101 is also used to verify and analyze the end-to-end system to be verified based on the second processing result, hardware operating status data, and the first processing result, so as to obtain the software and hardware verification results and software and hardware analysis results of the end-to-end system to be verified.

[0052] Here, an end-to-end collaborative analysis mechanism can also be deployed in the host computer. This mechanism is used to analyze, verify, and locate problems in the software behavior, hardware behavior, and software-hardware collaborative behavior of the end-to-end system to be verified. The software and hardware verification results can specifically include software operation problems, hardware operation problems, and collaborative operation problems of the end-to-end system to be verified. Software operation problems can specifically include issues with various software functions; hardware operation problems can specifically include issues with the hardware functions of various system hardware components; collaborative operation problems indicate collaborative problems existing during software-hardware interaction, such as cross-layer problems during software-hardware interaction, including but not limited to software-hardware resource contention, software-hardware collaborative deadlock, timing anomalies, bandwidth contention, system-level performance degradation, and resource starvation. The software and hardware analysis results can at least include problem location results and optimization suggestions related to the software and hardware verification results. Specifically, the software and hardware analysis results can include problem location results and optimization suggestions corresponding to software operation problems, hardware operation problems, and collaborative operation problems. The problem location results indicate the location and cause of the problem, and the optimization suggestions indicate the optimization methods for the corresponding problem, such as parameter optimization, configuration optimization, etc.

[0053] For example, the host computer 101 can receive hardware operation status data sent by the tracking module 104. Then, the host computer 101 can use the deployed end-to-end collaborative analysis mechanism to perform unified analysis and verification on the second processing result from the physical loopback channel, the hardware operation status data from the tracking module, and the first processing result obtained by itself, so as to obtain the software and hardware verification result and software and hardware analysis result of the end-to-end system to be verified.

[0054] In this way, on the host computer software side, by combining the hardware processing results from the physical loopback channel with the hardware operating status data from the tracking module, an end-to-end collaborative analysis mechanism can be constructed. This not only enables unified analysis and verification of software and hardware interaction behavior, but also allows for the timely discovery and resolution of cross-layer problems before deploying the end-to-end system to be verified, since the end-to-end collaborative analysis mechanism can support the location and analysis of cross-layer problems.

[0055] In one embodiment, the host computer 101 can also be used to optimize the software configuration parameters and hardware configuration parameters in the end-to-end system to be verified based on the software and hardware verification results and the software and hardware analysis results.

[0056] Here, the host computer 101 can also be configured with an optimizer agent. This optimizer agent is used to optimize the end-to-end system to be verified based on the software and hardware verification results and software and hardware analysis results. The optimization content may include, but is not limited to, software configuration parameter optimization and hardware configuration parameter optimization. Software configuration parameter optimization may include, for example, software code optimization and software parameter type optimization. Hardware configuration parameter optimization may include, for example, hardware timing optimization, hardware function optimization, hardware parameter type optimization, and hardware interconnection structure optimization.

[0057] For example, after the host computer 101 determines the software and hardware verification results and the software and hardware analysis results, the intelligent agent in the host computer 101 can be used to optimize the software configuration parameters and / or hardware configuration parameters in the end-to-end system to be verified based on the problems pointed out by the software and hardware verification results and / or the positioning and optimization suggestions pointed out by the software and hardware analysis results, thereby obtaining the optimized end-to-end system to be verified.

[0058] Thus, based on the verification architecture that constructs a physical loop channel between the host software and the hardware verification platform, and the non-intrusive tracing module embedded within the hardware logic, this application can not only achieve closed-loop collaborative verification of the end-to-end system, but also complete software and hardware collaborative debugging and analysis under full-speed hardware operation conditions.

[0059] In one embodiment, the hardware verification chip 102 includes at least a physical layer receiver module and a physical layer transmitter module. The physical layer receiver module establishes an uplink data transmission path between the hardware verification chip and a host computer to send input data collected by heterogeneous sensors and / or hardware processing results from the hardware verification chip to the host computer for processing. The physical layer transmitter module establishes a downlink data transmission path between the host computer and the hardware verification chip to send software processing results (i.e., the first processing result) output by the host computer to the hardware verification chip for hardware processing. A physical loopback channel 103 may be located between the physical layer receiver modules to establish a high-bandwidth, low-latency data transmission path between the host computer and the hardware side.

[0060] For example, the physical layer receiver module can be a MIPI D-PHY RX module (Mobile Industry Processor Interface D-Type Physical Layer Receiver); the physical layer transmitter module can be a MIPI D-PHY TX module (Mobile Industry Processor Interface D-Type Physical Layer Transmitter). By constructing a real physical loopback channel between the MIPI D-PHY RX and MIPI D-PHY TX, the physical signal results generated by the hardware after processing real or near-real input data can be transmitted back to the host computer software environment at near real-time speeds for end-to-end systems.

[0061] In summary, this application provides a hardware-software co-verification system for end-to-end intelligent systems, particularly suitable for functional verification, performance verification, and debugging analysis of edge intelligent systems containing heterogeneous sensors and computing units. Furthermore, this hardware-software co-verification system achieves a true physical closed loop between the software environment and the hardware platform through a physical loopback channel, avoiding the physical isolation problem in traditional verification methods. By collecting and uploading hardware operating status data through a tracking module, debugging visibility can be significantly improved without reducing hardware operating speed, solving the spatiotemporal mismatch problem inherent in traditional hardware-software co-verification. Based on the co-verification framework, complex end-to-end system-level tasks (such as long-duration video stream processing) become verifiable in engineering, reducing the risk of excessive hardware redundancy design and on-site trial and error, effectively narrowing the verification gap between hardware and software, and improving the overall verification efficiency and reliability of the end-to-end system.

[0062] To facilitate understanding of the hardware and software co-verification system provided in the embodiments of this application, this application also provides, as follows: Figure 2The diagram shows a specific architecture of a hardware and software co-verification system. The hardware verification chip is an FPGA, which includes multiple different hardware modules, namely: a MIPI D-PHY RX module, a MIPI Camera Serial Interface 2 Receiver (MIPI CSI2 RX) module, a MIPI to Block Random Access Memory (MIPI2BRAM) module, a Block Random Access Memory (BRAM) module, a BRAM to PCIe (BRAM2PCIe) module, an AXI-Stream Data Width Converter (AXI-Stream data width converter), an XDMA module, an AXI-Stream First In First Out (AXI-Stream first in first out) buffer (AXI-Stream first in first out), a Stream Control (Stream ctrl) module, a MIPI CSI2 to PPI parallel interface (MIPI CSI2 to PPI), a timestamp pattern module, and a MIPI D-PHY TX module.

[0063] A physical loopback channel is established between the MIPI D-PHY RX module and the MIPI D-PHY TX module. XDMA includes two communication channels: one is the Slave AXI-Stream Card to Host channel (S_AXIS_C2H), which is the uplink data channel for XDMA, used to send FPGA data to the host computer; the other is the Master AXI-Stream Host to Card channel (M_AXIS_H2C), which is the downlink data channel for XDMA, used to send host computer data to the FPGA.

[0064] Specifically, the MIPI DPHY RX module can receive high-speed differential signals from heterogeneous sensors or hardware processing results from the physical loopback channel, recover the clock and data, and send the processing results to the MIPI CSI2 RX module. The MIPI CSI2 RX module parses the CSI-2 protocol header, data type, and payload, outputting 32-bit parallel image data to the MIPI2BRAM module. The MIPI2BRAM module writes the 32-bit data into the BRAM, resolving the cross-clock domain issue between the MIPI clock domain and the PCIe clock domain, and implementing data rate buffering. Then, the BRAM2PCIe module reads the 32-bit image data from the BRAM and sends it to the axis dwidth converter module to expand it into 512-bit image data, thus adapting it to XDMA's S_AXIS_C2H channel. The S_AXIS_C2H channel is then used to send the 512-bit image data to the XDMA module, which then transfers it to the host computer's memory at high speed via the PCIe bus using DMA. The host computer processes the input image data using the software function to be verified, obtaining the first processing result. Then, the host computer writes the first processing result to the XDMA module via PCIe. The XDMA module outputs a 512-bit first processing result to the axis FIFO module for asynchronous FIFO buffering using the M_AXIS_H2C channel, isolating the host computer clock domain from the FPGA clock domain to prevent data overflow or loss. The axis dwidth converter module reduces the buffered 512-bit first processing result to 32-bit data. The Stream ctrl module receives the 32-bit data processed by the axisdwidth converter module and receives control signals from the Timpstamp pattern module, inserting timestamps or test patterns into the data stream to verify system timing synchronization and data integrity. The MIPI CSI2 toPPI module encapsulates the 32-bit data processed by the Stream ctrl module into CSI-2 protocol format, then converts it into a PPI signal and sends it to the MIPI D-PHY TX module. The MIPI D-PHY TX module converts the PPI signal into a MIPI DPHY differential signal, which is then sent to the MIPI DPHY RX module via a physical loopback channel for closed-loop verification.

[0065] Furthermore, this application also provides a software-hardware co-verification method, which can be applied to the software-hardware co-verification system described in the above embodiments to achieve end-to-end software-hardware co-verification. For example... Figure 3 The flowchart shown is a software and hardware co-verification method provided in an embodiment of this application, which may include the following steps: S301: Configure the target parameters corresponding to the end-to-end system to be verified into the software and hardware co-verification system described in any of the above embodiments; the target parameters include software configuration parameters and hardware configuration parameters.

[0066] Here, the target parameters are adjustable parameters used to configure the software and hardware functions in the software-hardware co-validation system. Specifically, the target parameters may include, but are not limited to, software configuration parameters and hardware configuration parameters. Software configuration parameters may include, for example, model type, number of network layers, and network architecture. Hardware configuration parameters may include, for example, computing power, hardware type, hardware interconnect structure, and hardware transmission bandwidth. Model types may include, for example, CNN, RNN, and SNN types.

[0067] In practice, the target parameters can be determined according to the system architecture and hardware and software functions of the end-to-end system to be verified, and the target parameters can be configured to the host computer and hardware verification chip in the hardware and software collaborative verification system described in any of the above embodiments to obtain the configured hardware and software collaborative verification system.

[0068] S302: Obtain the first and second processing results output by the hardware and software co-verification system for the input data; the first and second processing results are used to verify and debug the end-to-end system to be verified.

[0069] In practical implementation, after obtaining the configured hardware-software co-verification system, the operation process of the hardware-software co-verification system described in the above embodiments can be followed. The input data is used to verify the end-to-end system to be verified, resulting in a first processing result output by the host computer and a second processing result fed back by the hardware verification chip through the physical loopback channel. Furthermore, hardware operating status data collected by the tracking module in the configured hardware-software co-verification system can also be obtained.

[0070] Then, the intelligent agent deployed in the host computer can be used to verify and analyze the end-to-end system to be verified corresponding to the software and hardware co-verification system based on the first processing result, the second processing result and hardware operation status data, to obtain the software and hardware verification results and software and hardware analysis results of the end-to-end system to be verified. Then, based on the software and hardware verification results and software and hardware analysis results, the end-to-end system to be verified is verified and debugged, and the target parameters are optimized.

[0071] Alternatively, operations and maintenance personnel can verify and analyze the end-to-end system to be verified corresponding to the software and hardware co-verification system based on the first processing result, the second processing result, and hardware operating status data, obtaining the software and hardware verification results and software and hardware analysis results of the end-to-end system to be verified. Then, based on the software and hardware verification results and software and hardware analysis results, operations and maintenance personnel can verify and debug the end-to-end system to be verified, and optimize the target parameters.

[0072] like Figure 4 The diagram shown illustrates an implementation framework of a hardware-software co-verification method according to an embodiment of this application. Adjustable parameters (such as model type and computing power) can be configured into the host computer and hardware verification chip in any of the above embodiments of the hardware-software co-verification system. Then, the input data is used to verify the end-to-end system to be verified, and a tracking module is used to collect hardware operating status data. Figure 4 The hardware operating status data exemplifies aspects such as operating accuracy, latency, energy efficiency, and frame rate information from heterogeneous sensors. Then, using an intelligent agent deployed on the host computer, adjustable parameters are automatically optimized and updated based on the hardware operating status data, resulting in an updated end-to-end system to be verified.

[0073] Thus, by utilizing the hardware and software co-verification system and method provided in this application, hardware and software co-verification and optimization of various end-to-end systems to be verified can be achieved.

[0074] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0075] Based on the same technical concept, embodiments of this application also provide a computer device. (Refer to...) Figure 5 The diagram shown is a structural schematic of a computer device provided in an embodiment of this application, comprising: The system comprises a processor 501, a memory 502, and a bus 503. The memory 502 stores machine-readable instructions executable by the processor 501. The processor 501 executes these machine-readable instructions, and when executed, performs the following steps: S301: Configuring the target parameters corresponding to the end-to-end system to be verified into the hardware-software co-verification system described in any of the above embodiments; the target parameters include software configuration parameters and hardware configuration parameters; and S302: Obtaining the first and second processing results output by the hardware-software co-verification system for the input data; the first and second processing results are used to verify and debug the end-to-end system to be verified.

[0076] The aforementioned memory 502 includes a main memory 5021 and an external memory 5022. The main memory 5021, also known as internal memory, is used to temporarily store the computational data in the processor 501, as well as the data exchanged with external memory such as a hard disk 5022. The processor 501 exchanges data with the external memory 5022 through the main memory 5021. When the computer device is running, the processor 501 and the memory 502 communicate through the bus 503, so that the processor 501 executes the execution instructions mentioned in the above method embodiments.

[0077] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the hardware-software co-verification method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0078] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the software and hardware co-verification method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0079] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0080] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it can be understood that the disclosed system and method can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some communication interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0083] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0085] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A hardware and software co-verification system, characterized in that, The system includes a host computer and a hardware verification chip. The host computer has a software runtime environment related to the software function to be verified in the end-to-end system to be verified, and the software function to be verified is deployed thereon. The hardware verification chip has the hardware function to be verified corresponding to the system hardware in the end-to-end system to be verified. A physical loopback channel is pre-established between the host computer and the hardware verification chip. This physical loopback channel has data transmission characteristics that match those of the hardware verification chip, wherein: The host computer is used to receive input data, process the input data in the software running environment according to the function of the software to be verified, obtain a first processing result, and transmit the first processing result to the hardware verification chip. The hardware verification chip is used to receive the first processing result, process the first processing result according to the hardware function to be verified, and obtain a second processing result; and use the physical loopback channel to send the second processing result back to the host computer as new input data; the first processing result and the second processing result are used to verify and debug the end-to-end system to be verified.

2. The system according to claim 1, characterized in that, It also includes a tracking module, used to collect hardware operating status data during the process of the hardware verification chip processing the first processing result according to the function of the hardware to be verified.

3. The system according to claim 2, characterized in that, The hardware operating status data includes at least the data path occupancy status, storage and interconnect bandwidth usage, task scheduling and arbitration results, the operating status of the control state machine, and operating result information.

4. The system according to claim 2, characterized in that, The tracking module is also used to send the hardware operating status data to the host computer; The host computer is further configured to verify and analyze the end-to-end system to be verified based on the second processing result, the hardware operating status data, and the first processing result, and obtain the software and hardware verification results and software and hardware analysis results of the end-to-end system to be verified.

5. The system according to claim 4, characterized in that, The software and hardware verification results include software operation problems, hardware operation problems, and collaborative operation problems; the software and hardware analysis results include at least the problem location results and optimization suggestions related to the software and hardware verification results.

6. The system according to claim 4, characterized in that, The host computer is also used to optimize the software configuration parameters and hardware configuration parameters of the end-to-end system to be verified according to the software and hardware verification results and software and hardware analysis results.

7. The system according to claim 1, characterized in that, The hardware verification chip includes a physical layer receiver module and a physical layer transmitter module; The physical loopback channel is located between the physical layer receiver module and the physical layer receiver module.

8. The system according to claim 1, characterized in that, The hardware verification chip includes a field-programmable gate array (FPGA) chip.

9. The system according to claim 1, characterized in that, The input data also includes image information continuously acquired by heterogeneous sensors.

10. A software and hardware co-verification method, characterized in that, include: Configure the target parameters corresponding to the end-to-end system to be verified into the software and hardware collaborative verification system according to any one of claims 1 to 9; The target parameters include the software configuration parameters and hardware configuration parameters of the end-to-end system to be verified; The software and hardware collaborative verification system is obtained, and a first processing result and a second processing result are output for the input data; the first processing result and the second processing result are used to verify and debug the end-to-end system to be verified.

Images