Interface for processing analog sensor data

Abstract

The present disclosure relates to an apparatus for processing analog sensor data. The apparatus includes processing circuitry, a bus interface, and at least one output interface. The processing circuitry may be configured to process analog sensor data and output the processed analog sensor data via at least one output interface. The bus interface may be configured to connect the processing circuitry to a bus of a host system and retrieve analog sensor data generated by the host system. The at least one output interface may be configured to connect the processing circuitry to a simulated test environment. Further, a host system, a simulated test environment, and a method for processing simulated sensor data are described.

Description

Technical Field

[0001] The present disclosure relates to the processing of analog sensor data. Specifically, the present disclosure relates to methods and devices for processing analog sensor data, host systems for analog sensor data, and simulation test environments. According to a preferred embodiment, the present disclosure may relate to an interface of a drive simulator for processing simulated sender data. Background Art

[0002] Simulation test environments, such as industrial process simulators or drive simulators, are used to test dedicated software or hardware components that have been designed to operate or react to a specific environment, to name just a few. Such dedicated software or hardware components may include, for example, automotive components for autonomous driving or robotic components for industrial applications. In order to be able to test the dedicated software or hardware in various scenarios, the simulation test environment provides the dedicated software or hardware with analog sensor outputs that affect the behavior of the dedicated software or hardware components in the simulated environment.

[0003] The analog sensor outputs can be generated by a general-purpose processing unit or a dedicated processing unit, such as a graphics processing unit (GPU) or a general-purpose processor, which can generate video data, LiDAR output, or other data reflecting physical characteristics, such as acoustic data, temperature data, etc. For example, the GPU can be provided in a graphics server and be used to render video data of the simulated environment. The output rendered by the graphics server can be provided via a graphics interface, such as HDMI, DVI, DisplayPort, USB-C, or any other graphics interface or port suitable for carrying video data. The output of the graphics server can be captured and converted by a hardware component and then fed into the simulation test environment for testing purposes. Summary of the Invention

[0004] The present disclosure addresses the disadvantages of existing methods regarding processing speed, latency, and timing mismatches, which may occur, for example, when transmitting analog sensor outputs to a simulation test environment. The analog sensor outputs need to be provided preferably in real time or with reduced latency or delay to closely match the real environment. Other disadvantages of existing methods relate to the limited possibility of the simulation test environment providing feedback and adjusting the processing of the analog sensor outputs.

[0005] Therefore, there is a need in this technical field to further optimize the processing of analog sensor data, such as providing, converting, or processing analog sensor data in a way that solves the above problems.

[0006] The problems are solved by a method and a device for processing analog sensor data, a host system for analog sensor data, a simulation test environment, and at least one computer-readable medium as defined in the independent claims. Preferred embodiments are defined in the dependent claims.

[0007] A first aspect of the present disclosure defines a device for processing analog sensor data. The device includes a processing circuit, a bus interface, and at least one output interface. The processing circuit may be configured to process analog sensor data generated by a host system and output the processed analog sensor data via the at least one output interface. The bus interface is configured to connect the processing circuit to a bus of the host system and retrieve the analog sensor data (generated by the host system). The at least one output interface is configured to connect the processing circuit to an analog test environment (and output the processed analog sensor data).

[0008] The device may be constructed as an integrated device or an embedded device that can be integrated into a host system. For example, the bus interface of the device may be directly inserted into the bus of the host system, which may enable the device and / or the processing circuit of the device to directly access other devices, processing units, or memories of the host system at a low hardware level. This can achieve access in an efficient manner. It should be understood that the term "bus" refers to any kind of interconnection or switching link of the host system, which can provide the exchange of data, parameters, or commands between hardware or software components of the host system at a low system or hardware level. In particular, the bus interface is not limited to existing bus technologies. On the contrary, any kind of bus technology or interface can be used and is covered by the present disclosure.

[0009] The host system may be configured to generate analog sensor data using a general-purpose processing unit or a dedicated processing unit of the host system. For example, at least one central processing unit or at least one graphics processing unit may be used to generate the analog sensor data. The analog sensor data is retrieved by the device and / or the processing circuit via the bus and preferably cached in an internal memory or buffer. Thus, the processing circuit can directly access the analog sensor data in the format directly output by the central processing unit or the graphics processing unit. This direct access avoids any processing required to capture the analog sensor output at the output interface of the host system, which significantly increases the processing speed and reduces the latency.

[0010] The analog sensor data is processed such that the analog sensor data can be output via the at least one output interface. Preferably, the analog sensor data can be converted into a format suitable for or required by the analog test environment. The at least one output interface may be connected to the analog test environment in a direct manner, which again reduces the latency.

[0011] In a preferred embodiment, the processing circuit is configured to convert analog sensor data. The analog sensor data can be converted into a format corresponding to the output format of the real sensor that the sensor data simulates. For example, the output of a real camera can be simulated by rendered graphic data that can be converted according to the interface protocol of the real camera. The interface protocol can include one or more of GMSL, FPD-III, or MIPI. In this way, the simulated test environment can feed the analog sensor data into the dedicated software or hardware components under test as the output of a real sensor, such as a real camera frame. This further reduces the required processing steps, such as converting the output of the analog unit in the host system into an output format (such as the HDMI output of graphic data) and converting the HDMI output into the desired interface protocol. Instead, the direct output of the simulator unit (such as a GPU) is directly converted according to the target interface protocol.

[0012] According to a particularly preferred embodiment, the processing circuit is configured to output one or more of sensor parameters and status information. The sensor parameters and status information can reflect the processing parameters of the simulated real sensor that is applied to generate the analog sensor data. The sensor parameters and / or status information can be included in the output according to the target interface protocol. In this way, the dedicated software or hardware components in the simulated test environment can receive and use the processing parameters of the analog sensor as if the real sensor were connected to the simulated test environment. Even though the simulator unit of the host system knows the parameters of the simulated real sensor, the output interface protocol of the host system (such as the graphic output protocol) does not allow the inclusion of processing parameters that reflect the status and settings of real sensors (such as cameras, microphones, or LiDAR). This increases the degree of realism of the analog sensor data.

[0013] According to another embodiment, the device further includes a feedback interface that can be connected to an analog test environment and is configured to receive one or more parameters, wherein the processing circuitry is configured to pass at least some of the one or more parameters to the host system. The feedback interface enables better coupling to the analog test environment by providing a direct link or a reverse control channel for feedback data including one or more parameters. The one or more received parameters (or a portion thereof) can be passed directly to the host system via the bus interface. The parameters can be provided directly to the analog unit in the host system. This can include using one or more parameters (at least some of them) to configure the host system and / or the analog unit, thereby affecting the generation of analog sensor data in the host system. Thus, the feedback interface can enable one or more feedback channels from the analog test environment (or directly from the dedicated software or hardware components being tested in the analog test environment) to the analog unit in the host system. For example, in response to a frame request from a dedicated software or hardware component being tested in the analog test environment, an exposure request (trigger or synchronization) signal can propagate back to the host system via an interrupt triggered by serializing a camera trigger interference. This allows the host system and the analog test environment to remain synchronized with the dedicated software or hardware components being tested. This can be important for multi-sensor or multi-camera environments such as in a stereoscopic perspective environment.

[0014] In yet another embodiment, the processing circuit is configured to add a timestamp to the processed analog sensor data. This may include adding a timestamp to the analog sensor data. The timestamp may be used to further enhance the authenticity of the analog sensor data. By default, the host system does not use a standard interface such as an HDMI port to provide a timestamp (or timestamp information or timestamp data). HDMI ports only allow the output of image data; they do not contain a metadata channel for accommodating timestamp or exposure time data. For example, the auxiliary data in HDMI format only relates to aspects of image or audio quality, or only allows the synthesis of an audio sampling clock. Thus, one method may include overwriting the rendered image with a timestamp by rewriting (or otherwise combining) the actual image pixels, which may affect the quality of the analog sensor data. This may be critical in, for example, autonomous driving or industrial simulators. Preferably, the device (and / or the processing circuit of the device) may implement timestamp logic on the device such that timestamp processing can be close to the origin of the analog sensor data. Additionally, if multiple parallel sensor data streams are used, such as parallel image streams, the parallel sensor data streams may be synchronized before they are output via at least one output interface or a corresponding plurality of output interfaces (such as in batch) and transmitted to the analog test environment. This may include synchronizing the analog sensor data of multiple different (analog) sensors, including the analog sensor data. For example, if one or more other analog sensor data are provided by the host system, the video data of the analog camera may be synchronized with the one or more other analog sensor data (e.g., radar, LiDAR, or CAN signals).

[0015] According to another embodiment, the device further includes a memory and memory logic. The bus interface is configured to cache analog sensor data in the memory, and the processing circuit is configured to request access to the cached analog sensor data via the memory logic. The device may retrieve analog sensor data from a host system (e.g., via a bus) using direct memory access and preferably from the memory or cache of the host system, or directly from an analog unit in the host system. The retrieved analog sensor data may be cached in the memory of the device so that other logic units on the device (such as an output logic unit or a conversion logic unit) can access the analog sensor data. If multiple logic units are provided in the device, e.g., to process analog sensor data for multiple output interfaces, the memory logic may control access to the sensor data cached in the memory for each logic unit. The memory logic may include an arbiter and a multiplexer, which may switch the communication link between the memory (or the corresponding memory interface and / or memory controller) and the corresponding logic unit. The arbiter may control the order in which each logic unit is connected to the memory (or the corresponding memory interface and / or memory controller). The arbiter may ensure, for example, that each logic unit requesting access to the memory is served in a strict round-robin. This can ensure equal bandwidth is provided to each logic unit requesting access to the memory.

[0016] In a preferred embodiment, the analog sensor data is one or more of graphics data, radar data, and LiDAR data. Preferably, the analog sensor data is graphics data. The graphics data may be an analog video output. The analog video output may simulate the output of a real camera. The analog sensor data may be generated on a host system using one or more graphics processing units (GPUs), such as on one or more graphics cards. The GPU may render a view in a simulated environment according to one or more parameters. For example, the parameters may define one or more of the following: camera parameters, such as position, orientation, field of view, exposure time, etc.; output parameters, video parameters, such as frame rate, resolution, etc.; and other parameters that define the generation of the analog sensor data.

[0017] In another embodiment, the device is a semiconductor card. The semiconductor card can be an expansion card for a host system, which may include semiconductor circuits. The term expansion card should be understood as a card that provides functionality to a host system, where the card can be inserted into a connector, expansion slot, or bus slot of the host system. Thus, in the present disclosure, an expansion card can also be referred to as an expansion board, adapter card, peripheral card, or accessory card. For example, the device (or semiconductor card) can be an FPGA card or an ASIC card. FPGA and ASIC define programmable circuits that can be configured for various tasks and purposes and can be completed or executed in an efficient manner. FPGA is the acronym for Field Programmable Gate Array. It is an integrated circuit (IC) in which most of the electrical functions inside the device can be changed at various stages of the FPGA's life cycle. It can be changed by a design engineer during the PCB assembly process or after being shipped to a customer. Similarly, ASIC is the acronym for Application Specific Integrated Circuit. Similar to an FPGA, an ASIC represents an IC chip customized for a specific use or function. The semiconductor card can include an IC or one or more corresponding IC chips, as well as other logic or logic units that can couple the IC (chip) to a memory, bus, functional unit, one or more interfaces, etc. in any combination. For example, the semiconductor card can provide a bus interface to interconnect the IC chip and other logic (units) with the bus of a bus system.

[0018] Preferably, the bus interface is a parallel or serial expansion bus interface. This can include one or more of PCI Expansion (PCI-X), PCI Express (PCIe), MiniPCIe, M.2, Accelerated Graphics Port (AGP), PCI, Industry Standard Architecture (ISA), Micro Channel Architecture (MCA), VESA Local Bus (VLB), etc.

[0019] The present disclosure relates to any kind of software or hardware testing that requires simulating sensor data. This includes various industrial applications, such as testing robotic components or functional parts of an assembly line and other automated processes in industrial applications. Other application areas include testing various transportation vehicles such as drones or cars. In a preferred embodiment, the simulation test environment is an autonomous driving simulator. The test software or hardware for autonomous driving needs to simulate various sensor data. This can include, in any combination, one or more of simulating camera feeds, simulating LiDAR data, simulating processed data of a car, etc. The simulated camera (or video) feed can be generated using one or more GPUs in the host system, which can be configured as a GPU server, a graphics server, or a rendering server. The device according to the present disclosure enables a direct coupling of the graphics server to the simulation test environment to test dedicated software and / or hardware components for autonomous driving. The output of the graphics server is directly transmitted within the host system to the device; the simulated sensor data is processed and directly passed to an output interface that matches the requirements of the autonomous driving simulator. Thus, the device (and the corresponding system configuration) of the present invention effectively reduces the latency and delay caused by capturing the graphics output at the interface of the host system.

[0020] A second aspect of the present disclosure defines a host system for simulating sensor data, the host system including at least one central processing unit, a bus, and a system memory. The host system further includes: at least one (dedicated) simulation unit coupled to the bus, the simulation unit being configured to generate simulated sensor data; and at least one device coupled to the bus. The device can be a device according to one or more embodiments of the first aspect. Specifically, the device can include a processing circuit, a bus interface, and at least one output interface. The bus interface can be configured to connect the processing circuit (or device) to the bus of the host system and retrieve the simulated sensor data generated by the host system, preferably generated by at least one (dedicated) simulation unit. At least one output interface can be configured to connect the processing circuit (or device) to the simulation test environment. The processing circuit can be configured to process the simulated sensor data and output the processed simulated sensor data via at least one output interface.

[0021] In a preferred embodiment, the at least one device is configured to retrieve analog sensor data via the system memory or via a buffer of the at least one analog unit. The analog unit generates analog sensor data, which is typically provided via a standardized output interface (such as a graphics port) of the host system. Instead of scraping the output, the host system provides a component configured to retrieve analog sensor data via an internal memory (e.g., the system memory or a buffer of the analog unit). For example, the analog unit may cache the generated analog sensor data in an internal buffer of the analog unit, and the device may preferably directly access the buffer via a bus to retrieve and directly process the analog sensor data. For example, a GPU may cache rendered frames in an internal buffer of the GPU, and the device may directly access the buffer of the GPU to retrieve and directly process the rendered data. Similarly, the analog unit may be configured to write the generated analog sensor data to a system memory or other memory within the host system configured to receive the analog sensor data. The device may preferably directly access the system memory or other memory via a bus to retrieve and directly process the analog sensor data.

[0022] Preferably, direct memory access (DMA) is used to retrieve the analog sensor data. DMA is a configuration of a computer system such as the host system that allows a component to access memory without involving the central processing unit of the computer system. DMA can be used for system memory or other memory (units) or buffers in a computer system. Thus, the device can directly address the system memory or other memory units of the host system and retrieve the data stored in the system memory or other memory units, preferably without involving at least one central processing unit of the host system. Additionally or alternatively, the device can directly address the buffer of the analog unit in the host system and retrieve the data stored in the buffer, preferably without involving at least one central processing unit of the host system. The DMA access is preferably performed via the bus of the host system. Thus, the analog unit can function in a normal manner and write the output to the system memory, other memory, or buffer to provide the analog sensor data. However, it should be understood that the present disclosure can also implement another link between the at least one analog unit and the device, which can be used to retrieve the analog sensor data from the analog unit.

[0023] In one embodiment, the host system is a graphics server. Preferably, the at least one dedicated analog unit includes a graphics processing unit. A graphics server may also be referred to as a GPU server or a rendering server.

[0024] According to a preferred embodiment, the at least one analog unit includes a LiDAR analog unit. The present disclosure also encompasses other analog units for other types of sensor data.

[0025] Preferably, the host system can simulate a virtual environment with virtual objects. The virtual objects can be arranged in a scene graph. The virtual environment can be based on map data of an industrial environment or a construction plan or building plan, or any other data suitable for describing the (real) environment to be constructed as a virtual environment.

[0026] The at least one central processing unit can be configured to use one or more simulation engines (such as a physics engine, a graphics engine, other simulation engines, etc.) in any combination to simulate the virtual objects of the virtual environment. For example, the host system can implement a real-time 3D engine to achieve flexible simulation of the virtual environment. The host system can utilize one or more engines to render the virtual environment according to one or more modalities (including the visual modality or the acoustic modality). For example, a (visual) renderer can provide video data, which preferably includes an image sequence or video stream representing the virtual environment. The image sequence or video stream can include stereoscopic image data to achieve a stereoscopic representation of the simulated virtual environment. The at least one central processing unit can also utilize the renderer or an (acoustic) renderer to generate audio data representing the virtual environment. The audio data can include acoustic information that realizes a spatial representation of sound. The virtual environment can be further simulated and rendered according to any other modalities (such as the tactile, gustatory, or olfactory modalities) in any combination. For each modality, the host system can include a simulation unit, which can be configured to execute the corresponding renderer or rendering engine. The renderer or rendering engine can render the simulated virtual environment based on the parameters of the simulated sensors (such as virtual cameras, virtual LiDARs, microphones, etc.) in the virtual environment (which can include one or more of the position and orientation in the virtual environment) and further processing parameters of the simulated sensors (such as for a virtual camera, field of view, exposure time, etc.).

[0027] Preferably, one or more simulation engines can include a real-time graphics engine or rendering engine configured to generate a rendering output in response to at least one application. As used throughout this specification, a rendering engine can be a software and / or hardware framework that is capable of receiving an abstract representation of a scene (such as a scene graph, including virtual objects having one or more of graphical elements, acoustic elements, and elements of any other modality) and a definition of the conditions in the scene (such as lighting conditions). The rendering engine is capable of processing the information and delivering a realistic representation of the scene. The rendering engine can be implemented on a simulation unit and utilize dedicated rendering hardware that is capable of calculating the realistic appearance of the virtual objects in the virtual environment considering surface characteristics, reflections, multiple light sources, ambient light, and other conditions. Preferably, the simulation engine can be implemented by at least one central processing unit of the host system to simulate the virtual environment. Preferably, the rendering engine (or other engines for other modalities or simulated sensors) can be implemented on the simulation unit independently of any application executed on the host system. Therefore, the rendering engine can be provided as hardware, software, or a combination thereof.

[0028] A third aspect of the present disclosure defines a simulation test environment, the simulation test environment including: one or more processing units, each processing unit configured to simulate a physical environment using one or more simulated sensor data, wherein at least one of the one or more simulated sensor data is received from a host system according to one or more embodiments of the third aspect of the present disclosure. In particular, the host system may be configured to simulate sensor data and includes at least one central processing unit, a bus, and a system memory. The host system further includes at least one (dedicated) simulation unit coupled to the bus and at least one device coupled to the bus, the at least one simulation unit being configured to generate simulated sensor data. The device may be a device according to one or more embodiments of the first aspect. In particular, the device may include a processing circuit, a bus interface, and at least one output interface. The bus interface may be configured to connect the processing circuit (or device) to the bus of the host system and retrieve the simulated sensor data generated by the host system, preferably generated by at least one (dedicated) simulation unit. The at least one output interface may be configured to connect the processing circuit (or device) to the simulation test environment. The processing circuit may be configured to process the simulated sensor data and output the processed simulated sensor data via the at least one output interface.

[0029] A fourth aspect of the present invention defines a method for processing simulated sensor data, which includes: providing a device having a bus interface, at least one output interface, and a processing circuit, the bus interface being coupled to the bus of a host system and the at least one output interface being connected to a simulation test environment; retrieving, via the bus interface, the simulated sensor data generated by the host system; processing, by the processing circuit, the simulated sensor data; and outputting, by the processing circuit via the at least one output interface, the processed simulated sensor data.

[0030] The method may be a computer-implemented method. Specifically, the method may be a method executed by a processing circuit according to one or more embodiments of the first aspect. The method may also be executed in a system including a simulation test environment according to the third aspect and a host system according to the second aspect, wherein the host system may be equipped with a device according to the first aspect. In one or more embodiments, the method may be a method for testing software and / or hardware in a simulation test environment based on simulated sensor data generated by a host system, wherein the simulated sensor data is processed by a device located in or embedded in the host system.

[0031] According to one embodiment, the method further includes converting the simulated sensor data.

[0032] In a preferred embodiment, the method further includes outputting one or more of the analog sensor parameters and the analog status information.

[0033] In another embodiment, the method further includes the device receiving one or more parameters from an analog test environment and the processing circuit passing at least some of the one or more parameters to the host system.

[0034] According to a preferred embodiment, the method further includes the processing circuit adding a timestamp to the processed analog sensor data.

[0035] In one embodiment, the method further includes the bus interface caching the analog sensor data in the device's memory and the processing circuit requesting access to the cached analog sensor data via the device's memory access unit.

[0036] A fifth aspect of the present disclosure defines a computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a computing device, cause the computing device to perform the method according to one or more embodiments of the fourth aspect of the present disclosure. Specifically, the method can be used to process analog sensor data, including: providing a device having a bus interface, at least one output interface, and a processing circuit, the bus interface being coupled to the bus of a host system and the at least one output interface being connected to an analog test environment; retrieving analog sensor data generated by the host system via the bus interface; processing the analog sensor data by the processing circuit; and outputting the processed analog sensor data via the at least one output interface by the processing circuit. In one or more embodiments, the method can be a method for testing software and / or hardware in a test analog test environment based on analog sensor data generated by a host system, wherein the analog sensor data is processed by a device located in or embedded in the host system.

[0037] It should be understood that the embodiments according to the first, second, or third aspect can configure the logic, processor, or functional units with the features of the embodiments according to the fourth aspect of the present disclosure in any combination. Preferably, the device, the host system, and the analog test environment can be configured in any combination according to the embodiments of the fourth aspect. Similarly, the method according to the embodiments of the fourth aspect can include processing steps that reflect the structural features of the device, the host system, and the analog test environment in any combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The specific features, aspects, and advantages of the present disclosure will be better understood with reference to the following description and drawings, wherein:

[0039] Figure 1 Illustrates the conventional generation of analog sensor data for an analog test environment;

[0040] Figure 2 Shows a schematic layout of a host system according to an embodiment of the present disclosure;

[0041] Figure 3 Shows a flowchart of a method according to an embodiment of the present disclosure;

[0042] Figure 4 Are details of a host system of an embedded device according to an embodiment of the present disclosure;

[0043] Figure 5 Are details of a host system of an embedded device according to an embodiment of the present disclosure;

[0044] Figure 6 Shows a schematic layout of a device for processing analog sensor data according to an embodiment of the present disclosure;

[0045] Figure 7 Provides details of a register simulator component of a device according to an embodiment of the present disclosure;

[0046] Figure 8 Are details of a video output component of a device according to an embodiment of the present disclosure; and

[0047] Figure 9 Shows details of an implementation of a device according to an embodiment of the present disclosure. Detailed Description

[0048] In the following description, reference is made to the accompanying drawings, which illustrate various embodiments in an illustrative manner. In addition, various embodiments will be described by reference to several examples. It should be understood that embodiments may include design and structural changes without departing from the scope of the claimed subject matter.

[0049] The techniques described herein can be implemented in a variety of computing systems, examples of which are described in more detail below. Such systems generally involve the use of a suitably configured computing device that implements multiple modules, each of which provides one or more operations required to perform the execution of such techniques. Each module can be implemented in its own way; all of them do not need to be implemented in the same way. As used herein, a module can be a structural component of a system that performs an operational role, which can be part of or the entire software element (e.g., a procedure, a discrete process, or the functionality of any other suitable embodiment). A module can include computer-executable instructions and can be encoded on a computer storage medium. Modules can execute in parallel or serially as appropriate and can communicate with each other using shared memory on the computer on which they are executing, using a messaging protocol, or in any other suitable way. Exemplary modules that perform one or more tasks are described below, but it should be understood that the division of the described modules and tasks can be merely illustrative of the types of modules by which the exemplary techniques described herein can be implemented, and the present invention is not limited to being implemented with any particular number, division, or type of modules. In some embodiments, all functions can be implemented in a single module. Additionally, for clarity, the modules are discussed below as all executing on a single computing device, but it should be understood that in some embodiments, the modules can be implemented on separate computing devices adapted to communicate with each other. In particular, according to one or more embodiments of the present disclosure, the modules can be implemented in any combination on a device for processing analog sensor data, on a host system, or in an analog test environment.

[0050] In the present disclosure, abbreviations can be used to refer to known technologies. For example, AD refers to automatic drive or autonomous drive; ASIC refers to application-specific integrated circuit; DMA refers to direct memory access; FPD-Link refers to flat panel display link; FPGA refers to field-programmable gate array; GMSL refers to gigabit multimedia serial link; HDMI refers to high-definition multimedia interface; HIL refers to hardware-in-the-loop; I2C refers to inter-integrated circuit; ISP refers to image signal processor; PCIe refers to peripheral component interconnect express. Other known abbreviations can be used with reference to technologies known to those skilled in the art.

[0051] Figure 1 Illustrates the conventional generation of analog sensor data for an analog test environment. Figure 1 Illustrates a system 100 having a host system 102 and an analog test environment 104. The analog test environment 104 can be an industrial process simulator or an (automatic) drive simulator.

[0052] The simulation test environment 104 can use simulated sensor data to test dedicated software or hardware components that are designed to operate in or react to a specific environment, such as automotive components for autonomous driving, autonomous driving (AD) software, robotic components for industrial applications, and the like.

[0053] Regardless of the type of hardware or software to be tested, it is important to integrate the final hardware platform (such as a robotic hardware platform or an automotive hardware platform) into the simulation pipeline as quickly as possible to save valuable time and mileage in testing robots or testing vehicles in an industrial environment and to reveal extreme cases (corner cases) that are difficult to induce in real-world testing.

[0054] One way of this "hardware-in-the-loop" (HIL) testing is to feed simulated sensor data into the simulation test environment 104 for the target, such as an embedded automotive hardware platform running AD software.

[0055] The simulated sensor data can be generated on the host system 102 using one or more general-purpose processing units 106 or dedicated processing units, such as one or more graphics processing units (GPUs) 108, which can generate the required simulated sensor data. For example, the GPU 108 can render video data 110 of the simulated environment, and the output rendered by the GPU 108 can be provided via a graphics interface, such as HDMI, DVI, DisplayPort, USB-C, or any other graphics interface or port suitable for carrying video data.

[0056] Conventional HIL testing that relies on port output requires specially constructed external hardware components 112 that capture the simulated sensor data and convert it into a serial interface-compatible format. Generally, designing and manufacturing the external hardware components 112 can be complex and time-consuming, and such a solution even adds further latency in the propagation of the simulated sensor data. The hardware component 112 can feed the converted data into the simulation test environment 104 for the simulation and testing of dedicated software or hardware components. Therefore, current HIL testing involves transmitting the simulated video output, which relies on and uses the conventional display or HDMI output port of the GPU 108. Here, the propagation time from the rendering buffer of the GPU 108 and the external timestamp logic of the hardware component 112 may result in observable latency.

[0057] The propagation latency can reach 30 to 100 milliseconds, which can significantly affect the simulation and testing.

[0058] Software or hardware components dedicated to simulation and testing, such as AD software, rely heavily on large-scale simulation testing. This requires an improvement in the rendering capabilities of the host system 102 and generally requires a large infrastructure of GPU server clusters and tight scheduling and maximization of the efficiency and utilization of the available hardware infrastructure so that thousands of simulation test cases can be run within a short time frame. Therefore, timing considerations are highly relevant, and every millisecond matters when executing simulation test scenarios.

[0059] It should be understood that although the examples and embodiments may relate to HDMI ports, HDMI is only one widely used port type, and other output ports that can carry video or other sensor data (e.g., DVI, DisplayPort, USB-C, Thunderbolt, NDI, SDI, etc.) can be applied to the same or similar constraints as HDMI.

[0060] By the host system according to an embodiment of the present disclosure, as Figure 2 shown, the above-mentioned latency problem is solved, and timing considerations are addressed. Figure 2 FIG. 200 shows a schematic layout of a system 200 including a host system 202 according to an embodiment of the present disclosure. The host system 202 may be configured to simulate sensor data and includes at least one central processing unit 204, a bus 206, and a system memory. The host system 202 further includes at least one dedicated simulation unit 208, and at least one dedicated simulation unit 208 is coupled to the bus 206 and is configured to generate simulated sensor data. At least one dedicated simulation unit 208 may be one or more GPUs configured to render video data. The host system 202 further includes at least one device 210 coupled to the bus 206. At least one device 210 may be embedded within the host system 202. The host system 202 may treat at least one device 210 as an internal component, such that at least one device 210 can access the internal data, protocols, other hardware components, etc. of the host system 202 in an efficient manner.

[0061] The device 210 may be configured to process simulated sensor data and may include a bus interface to connect the device to the bus 206 and retrieve the simulated sensor data generated by at least one dedicated simulation unit 208. The simulated sensor data may be retrieved via the bus 206 from the dedicated simulation unit 208, a buffer of the dedicated simulation unit 208 (e.g., a rendering buffer), or a buffer or memory external to the dedicated simulation unit 208 of the host system 202.

[0062] The device 210 may also include one or more output interfaces 212, which may be used to connect the host system 202 to the simulation test environment 214. The processing circuitry of the device 210 may be configured to process the analog sensor data and output the processed analog sensor data to the simulation test environment 214 via at least one output interface.

[0063] Embodiments of the present disclosure may implement the device 210 as a custom hardware device. For example, it may be an (FPGA) expansion card inserted into a (PCIe) bus slot of the host system 202 itself. The host system 202 may implement simulator software. Thus, the device 210 may directly read the memory or cache of the host system 202 or directly access each dedicated analog unit in at least one dedicated analog unit 208 to retrieve the analog sensor data and directly output the processed data to the output interface, for example, using a serial interface format (such as GMSL), to output to the simulation test environment 214.

[0064] It should be understood that although the examples and embodiments may refer to FPGA as a technology for implementing the device 210 as a circuit, logic, or expansion card, it is feasible for the present disclosure to develop and apply other semiconductor technologies (such as ASIC, etc.).

[0065] Additionally or alternatively, other sensor data (such as data from LiDAR or other sensors) may be simulated, processed, and transmitted, and the use of video data is an example of analog sensor data.

[0066] In one or more embodiments, the device 210 may have a DMA controller, a memory, and a camera output interface. Preferably, under the control of a software driver, the device 210 may transfer the rendered simulator image from the GPU render cache to the memory of the device 210.

[0067] For example, this may be accomplished by directly transferring the rendered image, frame, or tile or any other type or portion of the rendered data from the render cache to the memory in the device 210 via direct memory access (DMA), or by first transferring the rendered image, frame, or tile or any other type or portion of the rendered data from the render cache to the system memory and then from the system memory to the memory in the device 210.

[0068] This may be done by the DMA controller of the device 210. The transfer method used may depend on the capabilities of the dedicated analog unit 208 (in this case, the rendering device or GPU).

[0069] Direct Memory Access allows saving CPU overhead for data transfer. With DMA, the Central Processing Unit 204 does not need to actively participate in the data transfer operation; it only needs to initiate the data transfer, and then it can perform other operations while the transfer is in progress, and it preferably finally receives an interruption from the DMA Controller (DMAC) when the operation is completed.

[0070] Device 210 converts and outputs any number of retrieved images, frames or tiles or any other type or portion of rendered data into the target format of the analog test environment 214. This may include, for example, GMSL / GMSL2 used by typical automotive-grade cameras to feed automotive HW platforms such as NVIDIA Drive PX or Drive AGX (Xavier, Pegasus).

[0071] Embedding device 210 into the host system 202 can save tens or hundreds of milliseconds for each scenario (test case), which is crucial when running a large number of simulation test cases (such as thousands of server-based simulation test cases).

[0072] Latency may also affect the decision-making algorithms of the hardware or software under test (such as AD software) for each frame. For example, the AD software can process images faster, and it can react faster to the overall traffic scenario. For example, when testing the behavior of AD software in a highway environment with a speed of 100 - 130 km / h (28 - 36 m / s), saving 100 ms in the vehicle control cycle can save valuable meters on the simulated road and can relieve the burden on the vehicle control algorithm of the AD software in compensating for the impact of data transfer latency.

[0073] Compared to Figure 1 the method shown, which includes a propagation delay of 30 to 100 milliseconds, Figure 2 the embodiment shown has a propagation delay far lower than 1 millisecond.

[0074] Similarly, the AD software is an example of a dedicated hardware or software component tested in the analog test environment 214, and the present disclosure is not limited to specific use cases. Instead, other dedicated hardware or software components in industrial applications are also covered, such as automated robots in assembly lines, etc.

[0075] Figure 3 A flowchart of a method according to an embodiment of the present disclosure is shown. Method 300 may be a computer-implemented method that may be executed on a computing device or multiple computing devices. As Figure 2 shown, the multiple computing devices may include the host system 202 and / or the analog test environment 214.

[0076] Method 300 may start in item 302. Method 300 may be used to process analog sensor data.

[0077] Method 300 may include providing 304 a device. The device may be Figure 2 device 210. The device may include a bus interface, at least one output interface, and processing circuitry, the bus interface being coupled to a bus of a host system and the at least one output interface being connected to an analog test environment. Thus, the host system may be host system 202, and the analog test environment may be analog test environment 214, as Figure 2 shown.

[0078] Method 300 may continue in item 306 by retrieving analog sensor data generated by the host system by the bus interface or the device. The analog sensor data may be retrieved using DMA from a buffer or memory of the host system or directly from one or more dedicated analog units of the host system.

[0079] In item 308, the processing circuitry or the device may process the analog sensor data. In item 310, the processing circuitry or the device may output the processed analog sensor data via the at least one output interface.

[0080] Method 300 may iterate by returning to item 306 and retrieving further data. Method 300 may end in item 312.

[0081] Figure 4 and Figure 5 FIGS. and Figure 2 detail a host system of an embedded device according to an embodiment of the present disclosure. Host systems 400 and 500 may depict similar configurations. Thus, like components are denoted with like reference numerals. Host systems 400 and 500 may include a CPU 402 and a GPU 404. Host system 400 may include a HIL circuit 406. Host system 500 may include a HIL circuit 506. CPU 402 may correspond to a central processing unit 204 as Figure 2 shown, and may have an associated CPU memory 408. GPU 404 may correspond to a dedicated analog unit 208 as Figure 2 shown, and may have an associated GPU memory 410. Each of the HIL circuits 406, 506 may correspond to a device 210 as

[0082] Figure 4 and Figure 5Embodiments of HIL circuits 406 and 506 are shown respectively, showing possible paths for retrieving the analog sensor data generated by GPU 404 through HIL circuits 406 and 506. HIL circuit 406 preferably retrieves the analog sensor data from CPU memory 408 using DMA. GPU 404 can render the CPU memory 408, or the system can apply texture copies to move the rendered data from GPU memory 410 into CPU memory 408. Additionally or alternatively, HIL circuit 506 can preferably use DMA to directly access GPU memory 410 to retrieve the rendered data.

[0083] DMA is a preferred technique for accessing memory or other hardware within the host without involving interaction with CPU 402. It should be understood that other techniques for directly accessing memory, caches, or hardware components can be implemented in the host system, and these other techniques are covered by this disclosure.

[0084] Figure 6 A schematic layout of a device for processing analog sensor data according to an embodiment of the present disclosure is shown. Device 600 can correspond to Figure 2 device 210 of Figure 4 HIL circuit 406 of Figure 5 HIL circuit 506 of

[0085] Device 600 can include a bus interface 602 configured to connect device 600 to the bus 604 of the host system and retrieve the analog sensor data generated by the host system. The device also includes a plurality of output interfaces 606a, 606b, and 606c configured to connect device 600 to an analog test environment. It should be understood that although Figure 6 three output interfaces 606a, 606b, and 606c are shown in

[0086] device 600 can be extended from a single output interface to any number of output interfaces in any combination, such as two output interfaces, four output interfaces, five output interfaces, six output interfaces, seven output interfaces, eight output interfaces, or any other number of output interfaces. For illustrative purposes only, three output interfaces 606a, 606b, and 606c are shown.

[0087] The block or unit may be implemented as a hardware block or unit, e.g., a semiconductor block or unit. Thus, in the present disclosure, the plurality of blocks or units may be designated as processing circuitry 607. At least a portion of the block or unit may also represent software instructions executed by the block or unit to perform a task. Each block or unit may be implemented in its own way; all blocks or units need not be implemented in the same way. As used herein, a block or unit may be a structural component of a semiconductor device that performs an operational function. A block or unit may include hardware elements and / or computer-executable instructions. The blocks or units execute in parallel or serially depending on the circumstances and may communicate with each other using a communication link and / or a bus and / or a shared memory unit or buffer, using a messaging protocol, or in any other suitable way. Exemplary blocks and units that perform one or more tasks are described below, but it should be understood that the described blocks or units and the division of tasks may be illustrative only of the types of blocks or units that may implement the exemplary techniques described herein, and the present invention is not limited to implementing with any particular number, division, or type of blocks or units. In some embodiments, all functions may be implemented in a single block or unit. Additionally, for clarity, the blocks or units are discussed below as all executing on device 600, but it should be understood that in some embodiments, the blocks or units may be implemented on separate semiconductor devices adapted to communicate with each other, e.g., via bus 604 or another direct communication link.

[0088] According to one embodiment, device 600 may be implemented as an FPGA circuit, where the lines surrounding processing circuitry 607 may represent the FPGA boundaries. Device 600 may be an expansion card for bus 604 of a host system. However, it should be understood that other semiconductor technologies may be applied to implement device 600. Figure 6 An overview of the internal hardware architecture of device 600 is provided.

[0089] Device 600 may be described in terms of video data from analog sensor data. However, it should be understood that other analog sensor data may be processed by device 600 in any combination. In this example, video data may flow into and through device 600 such that video frames may be moved to the memory 608 of device 600 via DMA from the GPU memory or the system memory of the host system.

[0090] When analog sensor data arrives via bus interface 602, which can be a PCIe interface, bus interface 602 can request access to memory 608 from the memory arbiter of memory logic 610. When the access is authorized, the memory multiplexer of memory logic 610 can be switched to connect bus interface 602 to memory controller 612. The analog sensor data arriving via bus interface 602 can then flow through the memory multiplexer and memory controller 612 of memory logic 610 and be written into memory 608.

[0091] Under the control of the video timing control blocks of video output blocks 614a, 614b, and 614c, the analog sensor data can be read from memory 608 by video output block 614 and output to the corresponding video interface 606. This can include, for example, GMSL, FPD-III, or MIPI, etc. Similar to bus interface 602, each video output block 614 must request access to memory 608 from the memory arbiter of memory logic 610 in order to be authorized to access and read the analog sensor data from memory 608 via the memory multiplexer and memory controller 612.

[0092] Device 600 may also include a DMA controller 616. The DMA controller 616 can request data transfer from the memory, cache, or hardware of the host system (such as main memory or GPU memory) via the bus interface. The requested data stream can be passed through the DMA controller 616 and memory logic 610 to memory controller 612, and memory controller 612 can write the data into memory 608.

[0093] Memory logic 610 can implement a memory arbiter and a memory multiplexer. Memory logic 610 can implement the connection between memory 608 (for the sensor frame buffer), DMA controller 616, and output interface 606. Memory 608 can be, for example, an external DDR4 memory on an FPGA board. Memory logic 610 can implement the scheduling of memory access, such as DDR4 memory access. The memory multiplexer can switch the bus interface of memory controller 612 between the memory interfaces of each video output block 614 and the memory interface of DMA controller 616. The order in which each client connects to the memory interface is controlled by the memory arbiter. The memory arbiter ensures that each client requesting access to memory 608 is preferably served in strict rotation. Doing so can ensure equal bandwidth is provided to all clients requesting access to memory 608.

[0094] Device 600 may also include a plurality of I2C register simulator blocks 618, which are further described below with respect to Figure 7 the plurality of I2C register simulator blocks 618.

[0095] Device 600 may also include a control register block 620, an interrupt controller 622, and serializer hardware 624, which may be provided for each output interface 606.

[0096] Device 600 also implements timestamping and synchronization of analog sensor data. Timestamping and synchronization of analog sensor data can be addressed in a variety of ways. By default, timestamp data is not provided by the simulator over the HDMI port. The HDMI ports only allow the output of image data; they do not contain metadata channels for accommodating timestamp or exposure time data (the only auxiliary data allowed relates to aspects of image or audio quality; the defined audio timestamp information frames only allow the synthesis of audio sampling clocks). Thus, the timestamping of an image can only be addressed by burning the timestamp into the actual image pixels, which is a demanding and unacceptable solution for industrial or automotive solutions.

[0097] A viable way of timestamping could be to stop the hardware elements built for this task or any intermittent purpose for the host system or the emulation test environment, marking the moment when analog sensor data is read out from the HDMI port. However, this can result in a measurable delay and non-uniform time distribution of the video data.

[0098] Embodiments of the present disclosure may enable the implementation of timestamp logic on a custom HIL device, such as device 600. The timestamping is implemented close to the origin of the analog sensor data. Thus, parallel streams of analog sensor data, such as parallel image or video streams, can be synchronized before they are bulk transferred to the emulation test environment. This solution can also meet the synchronization of analog sensor data from different sensors, such as video data with other sensor data (radar, LiDAR, or CAN signals), if they are also provided by the host system.

[0099] Embodiments of the present disclosure may also implement one or more feedback channels for sensor triggering (such as camera triggering). For example, a feedback channel may be provided from an analog test environment to a host system by device 600, which may host one or more GPUs (simulator GPU servers). For example, in response to a frame request from a hardware or software component under test in the analog test environment, the feedback channel enables the propagation of an exposure request (trigger or synchronization) signal from the analog test environment back to the host system via device 600 using an interrupt triggered by a serialized camera trigger interference. Device 600 may notify the simulation engine, simulation unit, or any other simulator software executing in the host system. For example, device 600 may notify the simulator software via one or more interrupts. For example, in the case of PCIe connectivity, device 600 may notify the simulator software via MSI interrupts. One or more interrupts may be processed by the operating system of the host system and passed to the simulation engine, simulation unit, or simulator software. This allows the analog test environment and the host system to remain synchronized with the hardware or software component under test. This may be applied to, for example, a multi-camera environment, such as a stereoscopic view. Since there is no dedicated signal for triggering, it would be significantly more difficult to implement this functionality via an HDMI port.

[0100] Embodiments of the present disclosure may also implement flexible image sizes. The HDMI port allows the output of standard, easily scalable image sizes that apply standard ratios (16:9, 4:3). Typical examples of image sizes are 1280x720 pixels (approximately 1Mpx). However, various sensors (such as automotive cameras) typically apply non-standard image sizes, such as 1928x1208 pixels. These images are difficult to relay through an HDMI port without distortion or cropping. This is solved by device 600 and embodiments of the present disclosure, which preferably allow the output of any image size in any ratio, resulting in lossless video data transmission.

[0101] Embodiments of the present disclosure may also simulate camera parameters and / or provide a reverse control channel. Device 600 may allow the transmission of metadata to a hardware or software component under test in the analog test environment. This may also be used to relay camera parameters or conditions. Since the internal registers of the camera can also be simulated by device 600, the simulator may also modify the simulated camera registers to present camera-related metadata to the hardware or software component under test in the analog test environment. For example, the temperature sensor register of the camera may be set to simulate a high temperature in the sensor. This may be used, for example, to simulate an over-temperature alarm in the hardware or software component under test in the analog test environment. This functionality would be significantly more difficult to implement via an HDMI port and would require additional devices to provide a configuration data path.

[0102] One preferred use case is to use a device 600 with an Image Signal Processor (ISP). The ISP typically processes tone mapping, automatic exposure, and color balance control, especially tuned for a given sensor. For each video frame, the ISP can assume the use of the previous set of exposure times, global, and color channel gains, and based on that assumption and the received frame, the ISP can calculate the next set of sensor parameters to use. In this case, latency can have a significant impact - while I2C communication, which can be used to set sensor parameters, is typically slow (with bitrates in the range of 100 - 1000 kHz). Using a 100 kHz I2C clock frequency and assuming 16 - bit sensor register addresses and values, one register write (with I2C communication overhead) takes approximately 0.5 - 1.0 milliseconds, while at 30 FPS, the frame time is 33.333 milliseconds.

[0103] If the transmission of sensor settings has a large latency, the analog software in the host system cannot prepare the next video frame with the configured exposure and thus sends a frame with an earlier configuration. This can lead to oscillations in the exposure parameters when there are transients in the image data (e.g., when a bright frame follows a darker frame, such as when an analog vehicle exits a tunnel). A small latency allows the simulator software more time to render the image frame and reduce oscillations.

[0104] In the case where the image is transmitted via HDMI, configuring the data path requires additional devices. This significantly increases the overhead. For example, a common FTDI FT4222 chip can be used to capture sensor I2C writes from an Electronic Control Unit (ECU) and transmit them via USB and the USB driver stack. This increases the jitter and latency of the I2C data path that can cause ISP oscillations.

[0105] Embodiments of the present disclosure can directly simulate sensor gain in the device 600 - eliminating the additional latency after controlling the transmission through the output link channel, or can provide fast access to these registers via the bus 604. In the latter case, the simulator software can directly use these register values in the fragment shader code of the GPU that generates the image to be emitted.

[0106] Embodiments of the present disclosure support additional sensors. Device 600 allows the transmission of other analog sensor data, such as LiDAR point clouds, through DMA access. Simulator software in the host system or any suitable simulation unit in the host system can generate point clouds from a 3D world model and can send the resulting sensor data from the processor or GPU to device 600. Similar to the rendered data, any other analog sensor data can preferably be exchanged through device 600 using DMA via the memory or buffer of the host system or the simulation unit on the host system. Depending on the virtual sensors supported by the simulator software and / or the simulation unit in the host system, other sensor data types can be provided similarly.

[0107] Embodiments of the present disclosure apply a custom hardware device. In one or more preferred embodiments, the custom hardware device is a specially programmed FPGA card with a custom software driver that can be inserted into the PCIe slot of a GPU server (host system) hosting the simulator software. It can preferably directly read the memory or buffer of the GPU server through the direct memory access (DMA) method to access, retrieve, and process analog sensor data.

[0108] Embodiments bypass the standard ports, significantly reducing latency. Embodiments provide the video output of the host system to the simulation test environment (for AD software) through a real camera interface protocol (e.g., GMSL, FPD-Link, etc.). In this way, the target AD software can detect the input as a real camera frame. It is also able to simulate a real camera by relaying, for example, analog camera parameters to the AD software (e.g., temperature). Embodiments also allow for flexible image sizes and lossless transmission. Embodiments also allow communication in both directions through DMA, i.e., from the GPU server (host system) to the AD software host computer (simulation test environment) and vice versa. Embodiments are also able to trigger the virtual camera sensor through a feedback channel (reverse communication). Embodiments also allow the transmission of additional analog sensor data, such as LiDAR point clouds, through DMA access. Embodiments also allow for early timestamping and synchronization of analog sensor data.

[0109] In particular, embodiments of the present disclosure can be applied to a GPU server farm that includes multiple servers and incorporates hundreds of GPUs. This enables thousands of simulation test cases to be run daily on the GPU server farm. Providing the embedded device 600 in one or more servers allows for a complete HIL test of the automotive HW platform while executing thousands of test cases in a shorter period of time. A complete HIL test that relies on standard sensor interface inputs saves hundreds of valuable public road test hours, which always requires at least two participants (a safety driver and an operator). Additionally, a typical large-scale regression test suite includes n thousand test cases (scenarios). Saving, for example, 100 milliseconds per execution can save 2n minutes per test suite run, thereby allowing for earlier detection of potential faults and better utilization of the GPU servers. This behavior also more realistically simulates the real in-vehicle sensor data stream and can reveal defects earlier in the AD software, the configuration or optimization of the HW platform, or the simulation software.

[0110] Figure 7 Details of a register simulator component of a device according to an embodiment of the present disclosure are provided. The register simulator component 700 may correspond to the I2C register simulator block 618 of the device 600, as Figure 6 shown. I2C is known in the art as an interface standard for chip-to-chip communication. The component 700 may read and feed sensor information to an AD host computer via I2C messages, which may correspond to, for example, the host system Figure 6 discussed with respect to. The registers of the component 700 may also be programmed from the simulator side, which may correspond to, for example, the simulation test environment Figure 6 discussed with respect to. Figure 7 Embodiments of may implement I2C register simulation and / or an embedded control channel.

[0111] The register simulator component 700 may include a dual-port memory 702, a control register memory controller 704, an I2C-to-memory bus interface 706, and an I2C block 708. Data from and to the control registers may be received via the link 710 by the control register memory controller 704, which may be implemented by the control register block 620 of the Figure 6 device 600. The I2C component 708 may interface to serializer hardware via the link 712, such as the serializer hardware 624 of the Figure 6 device 600.

[0112] Both the GMSL and FPD protocols provide an embedded control channel that carries an I2C or UART communication channel, as well as multiple GPIO bits over the same physical link medium as the video protocol. In the case of MIPI CSI-2, similar I2C and GPIO bits are available. Device 600 may present an I2C interface to each video output link 606. The I2C interface provides access to a dual-port memory 702, which may also be accessed by the host system via a bus interface. The dual-port memory 702 appears to the device at the other end of the video link (in an analog test environment) as the control register of the camera (or other sensor) that device 600 is emulating. The control register may be provided in the control register block 620 of Figure 6 This setup allows the host environment to access and write the configuration values to the emulated sensor (camera) and adjust the sensor emulation accordingly. The host system may also write the analog state information to the dual-port memory (702), and the device at the other end of the video link (in an analog test environment) may read this as if it were the state information from a real sensor (camera).

[0113] The GPIO bits can be largely dedicated and user-defined. This can be implemented in device 600. One use case can be camera synchronization. In this use case, one of the GPIO bits is used by the device controlling the camera to synchronize image capture. Such synchronization of all the cameras connected to the device is important in surround view, for example, where lack of synchronization would result in inconsistencies between the images. Device 600 may use one of the GPIO bits as a trigger to send a video frame.

[0114] Figure 8 is the detail of the video output component of the device according to an embodiment of the present invention. The video output component 800 may correspond to the video output blocks 614a, 614b, and 614c of device 600, as Figure 6As shown. The video output component 800 implements the actual physical interface of the camera. The video output component 800 may represent a structure including a video timing control block 802. The video timing control block 802 may synthesize all video synchronization signals, H sync, V sync, and data enable. The signals may be output at 804. The video timing control block 802 may also generate a signal that indicates when the burst read memory controller 806 needs to schedule reading pixel data from the memory 608 of the device 600 via the interface 808 in order to maintain sufficient pixel data in the dual-port memory 810 such that it can be read when needed at a certain (correct) time. Since the dual-port memory 810 may be smaller than the entire video frame, the video timing control block 802 controls the rate of reading data to prevent too much data from being read and to prevent the dual-port memory 810 from being flooded with data. On the other hand, the video timing control block 802 ensures that there is always sufficient data in the dual-port memory 810 and that the dual-port memory 810 does not become empty.

[0115] The video timing control block 802 is also configured to control the operations of the dual-port pixel read block 812 and the pixel output buffer 814 to ensure timely output of pixel data via the link 816.

[0116] The video timing control block 802 may also receive frame synchronization via the input link 818 and communicate with a control register, which may be implemented by the control register block 620 of the device 600 via the link 820. Figure 6 of the device 600 via the link 820.

[0117] The geometry of the output video frame (number of valid pixels per line, number of valid lines per frame, and blanking period) may be controlled by the video timing control block 802. These parameters may be configured via the control register of the device 600 via the link 820.

[0118] When instructed by the video timing control block 802, the burst read memory controller 806 may read a block of data from the memory 608 in a burst manner and write the data to the dual-port memory 810. When signaled by the video timing control block 802 that a new frame is about to start, the burst read memory controller 806 may set its read pointer to point to the start of the frame video in the memory 608 of the device 600. Then, the burst read memory controller 806 may read pixel data and write it to the dual-port memory 810, preferably, in a burst manner at a certain time when instructed by the video timing control block 802, preferably, incrementing its read pointer as the process progresses until the entire frame has been read.

[0119] Pixel data is preferably transferred from memory 608 in a high-speed burst fashion because multiple devices may need to access memory 608 and access and bandwidth are preferably shared in a time-sliced manner. It would not be feasible for multiple devices to read data at the pixel rate or when they desire because this would result in contention for memory 608 and it would also make memory access very inefficient as transferring data bursts / blocks is much more efficient than individual transactions.

[0120] Dual-port memory 810 is preferably a special type of memory with two interfaces to a single memory array. Data can be written to or read from both interfaces simultaneously. In this system, the burst read memory controller 806 can read pixel data from memory 608 in a high-speed burst and write it to dual-port memory 810 such that the dual-port pixel read block 812 can read out pixel data stably and continuously at the pixel clock rate. In effect, the content of dual-port memory 810 can form a buffer pool of pixel data to be output.

[0121] Dual-port memory 810 can contain much less data than is required for an entire image frame; in fact, in the preferred embodiment, it can only hold a few lines. Thus, the burst read memory controller 806 and the dual-port pixel read block 812 can use dual-port memory 810 as a circular buffer.

[0122] Under the control of the video timing control block 802, the dual-port pixel read block 812 preferably reads pixel data from dual-port memory 802 at the output pixel clock rate and preferably within the H-sync, V-sync, and data enable framework. From the dual-port pixel read block 812, the video data has the timing characteristics of video data rather than the high-speed block burst transfer seen earlier in the pipeline.

[0123] The pixel output buffer 814 can be a functional block whose task can be to output pixel data and H-sync, V-sync, and data enable at the desired time under the control of the video timing control block 802.

[0124] Figure 9 Details of an implementation of a device according to an embodiment of the present disclosure are shown. Device 900 may correspond to Figure 2 device 210 of Figure 6 or device 600 of Figure 6 . Device 900 can be an FPGA board having an FPGA circuit 902 and multiple interfaces. The FPGA circuit 902 may correspond to

[0125] Device 900 may include a bus interface 904 that connects the FPGA circuit 902 to system memory, GPU memory, and any other hardware components via a system bus (such as PCIe or any other bus technology / protocol) to read analog sensor data and provide a feedback channel to an analog unit (such as a GPU).

[0126] Device 900 may also include a LIN interface 906. The LIN interface 906 may be configured to simulate vehicle sensor information, such as temperature, gyroscope, etc.

[0127] Device 900 may also include a CAN interface 908. The CAN interface 908 may be configured to allow simulation of sensor data using the vehicle's CAN interface (radar, vehicle wheel speed, etc.).

[0128] Device 900 may also include one or more serial interfaces 910 and 912. Examples may include any number and combination of RS232, UART, PPS, etc. Each of the one or more serial interfaces 910 and 912 may be configured to simulate, for example, GPS data or inertial measurement unit data, etc.

[0129] Device 900 may also include one or more camera interfaces 914. Examples may include GMSL interfaces. Each of the one or more camera interfaces 914 may represent a typical camera interface to relay a camera stream to a target AD hardware platform or any other target platform in an industrial environment.

[0130] Device 900 may also include an Ethernet gigabit interface 916. The Ethernet gigabit interface 916 may be configured to simulate, for example, LiDAR sensor data or radar data, etc.

[0131] Even though device 900 has been shown as including multiple interfaces 904...916, it should be understood that embodiments of the present disclosure may include a subset of interfaces 904...916 in any combination. For example, one embodiment may include a bus interface 904 for communicating with a host system and one or more serial interfaces 914 for communicating with an analog test environment. Other embodiments may be adapted to reflect an analog or test scenario based on dedicated hardware or software components for analog sensors and testing.

[0132] It should be understood that Figures 1 to 9 the implementation details provided represent preferred examples. Other implementations may be employed using different components, modules, blocks, units, circuits, connections, and links, and the present disclosure is not limited to a particular implementation in silicon.

[0133] Although some embodiments have been described in detail, it should be understood that aspects of the present disclosure can take many forms. Specifically, the claimed subject matter can be practiced or implemented differently from the examples described, and the described features and characteristics can be practiced or implemented in any combination. The embodiments shown herein are intended to illustrate rather than limit the invention defined by the claims.

Claims

1. An apparatus for processing analog sensor data, comprising: a processing circuit (607) configured to process analog sensor data generated by a host system; a bus interface (602) configured to connect the processing circuit (607) to a bus (604) of the host system and retrieve the analog sensor data; and at least one output interface (606a, 606b, 606c) configured to connect the processing circuit (607) to an analog test environment and output the processed analog sensor data.

2. The apparatus according to claim 1. Wherein, the processing circuit (607) is configured to convert the analog sensor data.

3. The apparatus according to claim 1 or 2. Wherein, the processing circuit (607) is configured to output one or more of sensor parameters and status information.

4. The apparatus according to any one of the preceding claims, further comprising: a feedback interface capable of connecting to the analog test environment and configured to receive one or more parameters, wherein the processing circuit (607) is configured to transfer at least some of the one or more parameters to the host system.

5. The apparatus according to any one of the preceding claims, wherein, the processing circuit (607) is configured to timestamp the processed analog sensor data.

6. The apparatus according to any one of the preceding claims, further comprising a memory (608) and memory logic (610), wherein, the bus interface (602) is configured to cache the analog sensor data in the memory (608), and the processing circuit (607) is configured to request access to the cached analog sensor data via the memory logic (610).

7. The apparatus according to any one of the preceding claims, wherein, the analog sensor data is one or more of graphic data, radar data, and LiDAR data.

8. The apparatus according to any one of the preceding claims, wherein, the apparatus is a semiconductor card.

9. The apparatus according to any one of the preceding claims, wherein, the bus interface (602) is a parallel or serial expansion bus interface.

10. The apparatus according to any one of the preceding claims, wherein, the analog test environment is an autonomous driving simulator.

11. A host system for analog sensor data, comprising: at least one central processing unit (204), a bus (206), and a system memory; at least one analog unit (208) coupled to the bus (206) and configured to generate analog sensor data; and at least one apparatus (210) according to any one of the preceding claims, coupled to the bus (206).

12. An analog test environment, comprising: one or more processing units, each processing unit being configured to simulate a physical environment using one or more analog sensor data, wherein at least one of the one or more analog sensor data is received from the host system according to claim 11.

13. A method for processing analog sensor data, comprising: providing (304) a device having a bus interface, at least one output interface, and processing circuitry, the bus interface being coupled to a bus of a host system, and the at least one output interface being connected to an analog test environment; retrieving (306) analog sensor data generated by the host system via the bus interface; processing (308) the analog sensor data by the processing circuitry; and outputting (310) the processed analog sensor data by the processing circuitry via the at least one output interface.

14. The method according to claim 13, wherein the method is performed on a device as defined in any one of claims 1 to 10.

15. At least one computer-readable medium having instructions stored thereon that, when executed on a computing device, configure the computing device to perform the method according to claim 13 or 14.

Images