A vehicle-mounted ranging system, a position information determination method and related equipment

Abstract

The embodiment of the application provides a vehicle-mounted ranging system, a position information determination method and related equipment, and belongs to the technical field of automobiles. The system combines an on-chip direct time-of-flight sensor, an embedded dynamic random access macro memory and a central computing unit, effectively improves the calculation efficiency of the system under the load of a hybrid CNN and a Transformer under the premise of ensuring low power consumption and low delay, reduces the off-chip storage access energy consumption, and improves the real-time performance and reliability of overall three-dimensional environment perception.

Description

Technical Field

[0001] This application relates to the field of automotive technology, and in particular to an on-board ranging system, a method for determining location information, and related equipment. Background Technology

[0002] With the rapid development of industries such as the Internet of Things, electric vehicles, drones, and intelligent service robots, autonomous driving and autonomous navigation have become key technologies. Distance perception, 3D point cloud construction, and 3D target recognition are the core requirements of these technologies. 3D point cloud perception can provide spatial geometry and depth information, playing an irreplaceable role in 3D object recognition, scene understanding, and path planning. However, existing traditional 3D perception technologies and hardware architectures still have significant shortcomings in terms of power consumption, latency, storage efficiency, and system cost.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose an on-board ranging system, a location information determination method, and related equipment, so as to effectively improve the system's computing efficiency, reduce the energy consumption of off-chip memory access, and improve the real-time performance and reliability of the overall three-dimensional environment perception while ensuring low power consumption and low latency.

[0005] To achieve the above objectives, one aspect of this application provides a vehicle-mounted ranging system, the system comprising: An on-chip direct time-of-flight sensor is used to acquire image data in the direction of a target, the image data including two-dimensional brightness information and three-dimensional point position information; An embedded dynamic random access macro memory is directly connected to the central computing unit. The embedded dynamic random access macro memory supports multiple working modes, which are used to perform convolutional neural network calculations for 2D target detection on the two-dimensional brightness information output by the on-chip direct time-of-flight sensor, or to perform 3D Transformer calculations on the three-dimensional point information after voxelization to generate the position information of the target direction. The central computing unit is connected to the embedded dynamic random access macro memory and the on-chip direct time-of-flight sensor, respectively. It is used to control the embedded dynamic random access macro memory and the on-chip direct time-of-flight sensor, perform voxelization processing on the three-dimensional point information, and send the voxelized three-dimensional point information to the embedded dynamic random access macro memory to obtain the position information output from the embedded dynamic random access macro memory.

[0006] In some embodiments, the system further includes: a preprocessing module, a reduced instruction set computing processor, and the direct time-of-flight sensor including a readout circuit; The preprocessing module is connected to the readout circuit and is used to read the two-dimensional brightness information or the three-dimensional point information from the direct time-of-flight sensor, and to construct point cloud information based on the three-dimensional point information. The preprocessing module includes: a first-in-first-out buffer, a point cloud construction unit, and a noise reduction unit; The reduced instruction set computing processor is connected to the readout circuit and the noise reduction unit respectively, and is used to control the preprocessing module and the direct time-of-flight sensor.

[0007] In some embodiments, the embedded dynamic random access macro memory integrates multiple single-transistor-single-capacitor memory cells and dual-transistor-single-capacitor computing cells.

[0008] In some embodiments, the system further includes a phase-locked loop (PLL); the PLL is connected to the on-chip direct time-of-flight sensor, and the PLL is used to provide a high-speed clock signal to the on-chip direct time-of-flight sensor.

[0009] One aspect of this application proposes a method for determining the location information of a vehicle-mounted ranging system, which operates on an embedded dynamic random access macro memory. The method includes: Two-dimensional brightness information of the target to be identified is obtained through an on-chip direct time-of-flight sensor; Based on the two-dimensional brightness information, target detection based on a 2D convolutional neural network is performed to determine the region of interest where the target to be identified is located. The on-chip direct time-of-flight sensor acquires the three-dimensional point information corresponding to the region of interest, and uses the three-dimensional point information to generate a three-dimensional point cloud. Perform 3D Transformer calculations on the voxelized 3D point cloud to determine the location information of the target to be identified.

[0010] In some embodiments, the step of performing target detection based on the two-dimensional brightness information using a 2D convolutional neural network to determine the region of interest where the target to be identified is located includes: Based on the two-dimensional brightness information, coarse sector detection based on a 2D convolutional neural network is performed on the target to be identified to determine the coarse region of interest. The coarse sector is divided based on the rotation angle. The step of performing 3D Transformer calculations on the voxelized 3D point cloud to determine the location information of the target to be identified includes: In the coarse region of interest, fine sectors are divided based on logarithmic distance. For each fine sector containing the target to be identified, 3D detection based on the 3D Transformer model is performed to determine the location information of the target to be identified.

[0011] In some embodiments, the embedded dynamic random access macro memory can be reconfigurably switched between a storage mode, an in-memory computation mode, and an XNOR mode. In the in-memory computation mode, the embedded dynamic random access macro memory is used to perform 3D Transformer computations on the three-dimensional point information or to perform 2D convolutional neural network operations on the two-dimensional brightness information.

[0012] In some embodiments, in the storage mode, the model of the 3D Transformer or the weights or activation values ​​of the 2D convolutional neural network are stored in the single-transistor-single-capacitor storage unit of the embedded dynamic random access macro memory. 3D Transformer calculations are performed on the voxelized 3D point cloud to determine the position information of the target to be identified, including: Control the embedded dynamic random access macro memory to be in the in-memory computing mode; The model weights or activation values ​​in the single transistor-single capacitor memory cell are read by enabling the word line, and the model weights or activation values ​​are multiplied and added using the dual transistor-single capacitor computing unit and the local sensing amplifier. By performing charge sharing operations between local cell arrays within and / or between columns, the multiply-accumulate operations of multiple input channels are completed. Finally, the 3D point cloud is used to perform 3D Transformer calculations to determine the position information of the target to be identified.

[0013] To achieve the above objectives, another aspect of this application provides a location information determination device for a vehicle-mounted ranging system, the device comprising: The acquisition module is used to acquire two-dimensional brightness information of the target to be identified through the on-chip direct time-of-flight sensor; The determination module is used to perform target detection based on the two-dimensional brightness information and determine the region of interest where the target to be identified is located; The generation module is used to acquire the three-dimensional point information corresponding to the region of interest through the on-chip direct time-of-flight sensor, and to generate a three-dimensional point cloud using the three-dimensional point information; The calculation module is used to perform 3D Transformer calculations on the voxelized 3D point cloud to determine the position information of the target to be identified.

[0014] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0016] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0017] The embodiments of this application include at least the following beneficial effects: This application provides a vehicle-mounted ranging system, a position information determination method, and related equipment. The system includes an on-chip direct time-of-flight sensor for acquiring image data in the target direction. The image data includes two-dimensional brightness information and three-dimensional point position information. An embedded dynamic random access macro memory is directly connected to a central computing unit. The embedded dynamic random access macro memory supports multiple working modes for performing 2D convolutional neural network calculations on the two-dimensional brightness information or performing 3D Transformer calculations on the voxelized three-dimensional point position information to generate position information in the target direction. The central computing unit is connected to both the embedded dynamic random access macro memory and the on-chip direct time-of-flight sensor, and is used to control the embedded dynamic random access macro memory and the on-chip direct time-of-flight sensor, perform voxelization processing on the three-dimensional point position information, and send the voxelized three-dimensional point position information to the embedded dynamic random access macro memory to obtain position information from the embedded dynamic random access macro memory. This system combines an on-chip direct time-of-flight sensor, an embedded dynamic random access macro memory, and a central computing unit. While ensuring low power consumption and low latency, it effectively improves the computational efficiency of the system under mixed CNN and Transformer workloads, reduces the energy consumption of off-chip memory access, and improves the real-time performance and reliability of the overall 3D environment perception. Attached Figure Description

[0018] Figure 1 This is a structural diagram of the vehicle-mounted ranging system provided in the embodiments of this application; Figure 2 This is a structural diagram of the embedded dynamic random access macro memory provided in the embodiments of this application; Figure 3 This is a schematic diagram of the in-memory mode of the embedded dynamic random access memory provided in the embodiments of this application; Figure 4This is a schematic diagram of the storage mode of the embedded dynamic random access memory provided in the embodiments of this application; Figure 5 This is a schematic diagram of the XOR mode of the embedded dynamic random access memory provided in the embodiments of this application; Figure 6 This is a schematic diagram of the point information preprocessing of the on-chip direct time-of-flight sensor provided in the embodiments of this application; Figure 7 This is a flowchart of a method for determining the location information of a vehicle-mounted ranging system provided in an embodiment of this application; Figure 8 This is a voxelized schematic diagram provided in an embodiment of this application; Figure 9 This is a schematic diagram of the sector division method provided in the embodiments of this application; Figure 10 This is a schematic diagram illustrating the optimization effect of an embodiment of this application; Figure 11 This is a schematic diagram of the structure of the location information determination device of the vehicle-mounted ranging system provided in the embodiments of this application; Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0022] 1) Voxelization Method: In recent years, Transformer models have been introduced into 3D point cloud processing to capture long-range dependencies and sparse spatial structures. A typical example is the dynamic sparse voxel transformer (DSVT), which outperforms convolutional neural networks in accuracy, but its computational cost and parameter size are much higher than convolutional methods, requiring extremely high hardware resources, resulting in very low utilization of traditional convolutional neural network (CNN) accelerators.

[0023] Among related technologies, 3D point cloud perception differs from 2D image processing, providing spatial geometry and depth information, which plays an irreplaceable role in 3D object recognition, scene understanding, and path planning. However, existing traditional 3D perception technologies and hardware architectures still have significant shortcomings in terms of power consumption, latency, storage efficiency, and system cost.

[0024] Currently, the main distance sensors include lidar time-of-flight sensors, millimeter-wave radar, or ultrasonic sensors.

[0025] Millimeter-wave radar utilizes millimeter-wave electromagnetic signals (typically in the 24GHz, 60GHz, or 77GHz bands) to estimate the target's range, velocity, and azimuth by measuring the flight time, Doppler shift, and phase information of the reflected wave. Its advantages include long detection range (tens to hundreds of meters), insensitivity to adverse weather and lighting conditions, the ability to jointly measure velocity and range, and relatively low hardware cost. It is widely used in advanced driver assistance systems and automotive autonomous driving perception systems. However, its spatial resolution and point cloud density are relatively low, making it difficult to provide detailed object contour information; it is prone to multipath interference in complex scenes and struggles to achieve high-precision 3D object recognition.

[0026] Ultrasonic sensors, on the other hand, calculate target distance by emitting ultrasonic pulses and receiving echoes, utilizing the propagation time of sound waves in the air. Their core advantages lie in low cost, small size, and low power consumption, making them suitable for short-range detection (typically within a few centimeters to several meters), and they are widely used in scenarios such as automotive reversing radar and robot obstacle avoidance. However, their detection range is short, angular resolution is low, and they are significantly affected by environmental factors such as temperature and humidity, and they cannot generate high-precision 3D point clouds for object detection.

[0027] LiDAR time-of-flight sensors estimate depth by modulating laser signals and detecting their phase difference or time of flight. Single-photon avalanche diode (SPAD) arrays are one of the mainstream implementation paths for time-of-flight sensors, capable of achieving high-sensitivity measurements under low-light conditions. They can also acquire large-scale, high-precision point cloud data using scanning modes. Therefore, SPAD-based sensors are gradually becoming the mainstream ranging sensors. However, most existing ToF systems transmit the acquired distance data to an external preprocessor via a high-speed interface for point cloud construction, resulting in excessively high data transmission power consumption and latency.

[0028] 3D point cloud perception tasks, such as 3D object recognition, detection, and tracking, typically require preprocessing and feature extraction of point cloud data. Current mainstream technologies for point cloud AI processing include: Point neural network (PNN) methods directly extract features from point cloud clusters through methods such as sampling the furthest distance and clustering, and then use convolutional neural networks to extract these features. However, these methods involve large amounts of data, high computational complexity, and require significant storage space for the point clouds, leading to their gradual obsolescence.

[0029] Voxelization 2D / 3D convolutional network methods: This approach divides the space into a regular 3D mesh, maps the point cloud onto voxel units, and then performs sparse convolution or pooling operations on the voxels. This method helps to unify the data structure and reduce computational complexity, but it requires a large amount of storage space, resulting in numerous external memory accesses.

[0030] Bird's-eye view approach: This method projects point clouds onto a ground plane to form a 2D feature map, which is then processed using mature 2D convolutional networks. This type of method is widely used in object detection tasks in autonomous driving, but it suffers from spatial information loss.

[0031] Existing algorithms present significant trade-offs between accuracy, computational efficiency, and storage overhead, posing new challenges to hardware architecture. To address the high computational demands and bandwidth requirements of 3D point cloud processing on general-purpose hardware such as GPUs and CPUs, several dedicated chip designs for 3D point cloud processing have emerged in recent years, such as accelerators for voxelized 2D / 3D convolutional neural network structures. However, these accelerators employ a sensor-computation separation architecture, where sensors are treated as independent devices. Ranging data is transmitted to an independent accelerator chip via UDP or serial port, where the independent accelerator chip performs point cloud construction and AI recognition. This approach cannot avoid high latency and high power consumption due to off-chip transmission. According to previous literature, although the overall system can support up to 7 frames per second (FPS), limitations in sensor data transmission and off-chip memory access ultimately resulted in a final demonstrated output of only 1 frame per second (FPS). At the same time, because its optimization is directed at point cloud convolution, it is difficult to support the core attention mechanism in transformer. Furthermore, the large scale of point clouds makes it difficult to store voxels completely on-chip, while storing voxels off-chip leads to a large number of external memory access operations, thus limiting the overall system's energy efficiency and throughput.

[0032] Therefore, to address this issue, some near-sensor SoCs have emerged. These SoCs introduce convolution operations at the pixel level of the image sensor, avoiding high off-chip transmission latency and power consumption, ultimately achieving energy efficiency of tens of trillions of operations per second per watt (TOPS / W). However, their design is only suitable for 2D image region of interest detection and only supports some simple 2D AI tasks such as face recognition detection. These chips cannot directly support 3D point cloud processing. Therefore, some works have proposed SPAD vision chips, which implement event-driven vision and direct impulse computation based on SPAD arrays, possessing strong sparse event perception capabilities. However, they can only support a limited scale of point clouds, and their fully digital design results in low energy efficiency and computational throughput, making them unsuitable for large-scale scenarios such as autonomous driving. 3D stacking solutions, on the other hand, integrate image sensors and processor chips in a 3D manner, shortening the data path and improving energy efficiency and overall data throughput. However, the manufacturing complexity and high cost of 3D packaging limit its large-scale adoption in consumer and automotive applications.

[0033] It can be seen that existing chips either focus on two-dimensional tasks or rely on high-cost processes, and have not yet been able to simultaneously support point cloud construction, voxelization and CNN / Transformer hybrid AI computing in a low-power, small-area, and scalable SoC architecture.

[0034] Therefore, there is an urgent need for a SoC architecture that can achieve on-chip integrated acceleration of sensing, point cloud construction, voxelization, and in-memory computing to simultaneously meet the multi-dimensional requirements of energy efficiency, area, and real-time performance.

[0035] In view of this, this application provides a vehicle-mounted ranging system, a position information determination method, and related equipment. The system includes an on-chip direct time-of-flight sensor for acquiring image data in the target direction. The image data includes two-dimensional brightness information and three-dimensional point position information. An embedded dynamic random access macro memory is directly connected to a central computing unit. The embedded dynamic random access macro memory supports multiple operating modes for performing 2D convolutional neural network calculations on the two-dimensional brightness information or performing 3D Transformer calculations on the voxelized three-dimensional point position information to generate position information in the target direction. The central computing unit is connected to both the embedded dynamic random access macro memory and the on-chip direct time-of-flight sensor. It controls the embedded dynamic random access macro memory and the on-chip direct time-of-flight sensor, performs voxelization processing on the three-dimensional point position information, and sends the voxelized three-dimensional point position information to the embedded dynamic random access macro memory to obtain position information output from the embedded dynamic random access macro memory. This system combines an on-chip direct time-of-flight sensor, an embedded dynamic random access macro memory, and a central computing unit. While ensuring low power consumption and low latency, it effectively improves the computational efficiency of the system under mixed CNN and Transformer workloads, reduces the energy consumption of off-chip memory access, and improves the real-time performance and reliability of the overall 3D environment perception.

[0036] This application provides an in-vehicle ranging system, a location information determination method, and related equipment, relating to the field of automotive technology. The in-vehicle ranging system, location information determination method, and related equipment provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application implementing the location information determination method, but is not limited to the above forms.

[0037] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0038] Figure 1 This is an optional structural diagram of the vehicle-mounted ranging system provided in the embodiments of this application. Figure 1 The vehicle-mounted ranging system may include: On-chip direct time-of-flight sensor, such as Figure 1 As shown in P0~P127, image data is used to acquire image data in the target direction. The image data includes two-dimensional brightness information and three-dimensional point position information.

[0039] The sensor employs a 128×1 SPAD-based pixel linear array, with each pixel unit integrating analog front-end circuitry and upper / lower counters. It can output 9-bit depth resolution in 3D mode, with a maximum range of approximately 30 meters. Through an external scanning galvanometer, it simultaneously supports high-resolution dual-mode imaging: in 2D mode, it obtains a 128×256 intensity image through photon counting for low-power 2D target detection; in 3D mode, it uses a coarse-to-fine encoding time-to-digital converter for time-of-flight decoding, achieving a maximum depth map of 128×512.

[0040] The enhanced dynamic random access memory (eDRAM), as shown in eDRAM macrocells #0 to #7 in the figure, is connected to the on-chip direct time-of-flight sensor. The eDRAM supports multiple computing modes, which can be used to perform 2D convolutional neural network calculations on two-dimensional brightness information or 3D Transformer calculations on three-dimensional point information.

[0041] The embedded dynamic random access macro memory in this embodiment can be a high-density tri-mode eDRAM CIM macro, and its structure is as follows: Figure 2As shown, it supports both storage and compute modes. Each eDRAM macro cell consists of a single-transistor-single-capacitor (1T1C) memory cell and a dual-transistor-single-capacitor (2T1C) compute cell, along with local sense amplifiers, global sense amplifiers, word lines (WL), bit lines (BL), and drivers for the word and bit lines. It also includes four differential successive approximation analog-to-digital converters. It contains a total of 256x258 1T1C cells with a capacity of 516K bits. It is divided into 16x128 local cell arrays, each containing 16x2 1T1C memory cells, one local sense amplifier, and one 2T1C compute cell. The overall macro supports storage mode, in-memory compute mode, and same / or similarity detection mode.

[0042] The embedded dynamic random access macro memory proposed in this invention supports multi-bit matrix multiplication operations through charge sharing operations between intra-column / inter-column local cell arrays (LCAs). This operation greatly improves the overall area efficiency of the macro, achieving a 30-fold improvement in area efficiency compared to the previous traditional eDRAM macro.

[0043] The eDRAM (embedded dynamic random access memory) macro simultaneously supports tri-mode operation, flexibly switching between memory mode, in-memory computation, and XNOR mode. Through flexible macro configuration and full utilization of the sparsity of point cloud data, its energy efficiency can be improved by 3.5 to 6.85 times compared to point cloud processors in previous literature. The in-memory mode of eDRAM is as follows... Figure 3 As shown, the weights of the model are pre-stored in the 1T1C storage cell. During in-memory computation, the charge is read out by opening the word line, and the voltage is amplified by a local sensing amplifier. The activation of the circuit is controlled by the dual transistors of the 2T1C, and the charging and discharging of the coupled capacitors characterizes the multiplication operation. The truth table is shown in the figure. Then, multi-bit charge sharing is achieved through intra-column / inter-column local unit arrays, and finally, a successive approximation analog-to-digital converter completes the efficient quantization result readout. This mode can achieve a 9.2-fold increase in storage density and a 34.2-fold increase in area efficiency compared to the embedded dynamic random access memory in previous literature.

[0044] The storage modes of embedded dynamic random access memory are as follows: Figure 4 As shown, its storage mode is used as a high-density tensor cache, supporting temporary storage of activation values ​​such as Q / K / V tensors. At the same time, when it is executed in the next attention layer, it is reconstructed into a CIM operation to avoid additional memory access overhead and avoid a decrease in the overall system utilization.

[0045] The XOR mode of embedded dynamic random access memory, such as Figure 5 As shown, it can be used for low-power similarity detection. Ultimately, it supports maintaining a always-on state under low power consumption. In always-on mode, the system operates at 1 FPS, using XOR operations to detect differences in intensity maps between adjacent frames. When the similarity exceeds a threshold, it maintains low-power operation; otherwise, it wakes up the full 3D point cloud processing pipeline.

[0046] The central control unit (CCU), connected to the embedded dynamic random access macro memory, is used to control the direct time-of-flight sensor and the embedded dynamic random access macro memory, and to acquire distance information output by the embedded dynamic random access macro memory.

[0047] In this embodiment, the on-chip direct time-of-flight sensor can acquire two-dimensional brightness information and three-dimensional point information, and can support the acquisition of different forms of information in the target direction. The on-chip direct time-of-flight sensor establishes an on-chip connection with the embedded dynamic random access macro memory, which can realize the rapid transfer of information. The central computing unit is used to control the embedded dynamic random access macro memory, so that the embedded dynamic random access macro memory can process the image data acquired by the on-chip direct time-of-flight sensor, realize the recognition of different types of image data, and determine the distance information of the object in the target direction.

[0048] In some embodiments, the system further includes: a preprocessing module, a reduced instruction set computing processor, and a direct time-of-flight sensor including a readout circuit; the preprocessing module is connected to the readout circuit and is used to read two-dimensional brightness information or three-dimensional distance data and two-dimensional brightness information from the direct time-of-flight sensor. The preprocessing module includes: a first-in-first-out buffer, a point cloud construction unit, and a denoising unit; the reduced instruction set computing processor is connected to the readout circuit and the denoising unit respectively and is used to control the preprocessing module and the direct time-of-flight sensor.

[0049] In some embodiments, such as Figure 6As shown, the preprocessing module performs point-to-point processing on the 2D brightness information and 3D point information acquired by the on-chip direct time-of-flight sensor. The preprocessing module combines the distance data measured by dToF with the system's rotation and vertical angles to construct 3D point cloud data. The noise suppression part uses the ROR algorithm to remove isolated points and environmental noise, improving point cloud quality. Simultaneously, the point cloud data is transmitted to the central computing unit in parallel stream form, avoiding large-scale off-chip storage access and repeated data movement as in traditional architectures. Through 2D region-of-interest guided point cloud construction and point cloud filtering mechanisms, redundant point cloud data is reduced by up to 82%.

[0050] like Figure 1 As shown, the on-chip direct time-of-flight sensor is directly connected to the on-chip first-in-first-out (FIFO) buffer, providing low-power and high-bandwidth sensor data transmission. The FIFO buffer is also directly connected to the point cloud construction unit and the noise reduction unit. The top-level reduced instruction set computing (RISC) core is connected via a bus to the on-chip time-of-flight sensor's readout control circuitry and the central computing unit, and is also connected to the noise reduction unit and the point cloud construction unit.

[0051] Combination Figure 7 As shown, this application embodiment also provides a method for determining the location information of a vehicle-mounted ranging system, which is applied to the vehicle-mounted ranging system in the above embodiment, including but not limited to steps S701 to S705.

[0052] Step S701: Obtain the two-dimensional brightness information of the target to be identified through the on-chip direct time-of-flight sensor.

[0053] The two-dimensional brightness information captured by the on-chip direct time-of-flight sensor is a brightness image, which is used to reflect the reflection intensity of objects in the direction of the target.

[0054] Step S702: Target detection based on 2D convolutional neural network is performed based on 2D brightness information to determine the region of interest where the target to be identified is located.

[0055] Using traditional image detection algorithms (such as YOLO, SSD, etc.) or custom brightness analysis, target objects are located in 2D images, and their regions of interest (ROIs) are delineated. Although targets can be identified from both 2D brightness information and 3D point position information, determining targets from 2D images is computationally efficient and suitable for rapid preliminary localization. By identifying the target, its position within the 2D brightness information can be determined, thereby identifying the ROI where the target is located. This avoids the inefficiency caused by directly processing 3D data.

[0056] In step S703, the on-chip direct time-of-flight sensor acquires the three-dimensional point information corresponding to the region of interest, and uses the three-dimensional point information to generate a three-dimensional point cloud.

[0057] The 3D point information is a depth image, which contains distance information of objects in each target direction. The 3D space is divided into multiple sector regions according to azimuth. The 3D point cloud data within the sector where the target to be identified is located is converted into a regular 3D mesh, and statistical features (such as density and average depth) of points within voxels are extracted.

[0058] After determining the sector where the target to be identified is located using two-dimensional brightness information, the three-dimensional point information within that sector is measured. The three-dimensional point cloud is then generated using the measured three-dimensional point information, providing information input for subsequent geometric processing.

[0059] Step S704: Perform 3D Transformer calculations on the voxelized 3D point cloud to determine the distance information of the target to be identified.

[0060] Voxelization process as follows Figure 8 As shown, the on-chip voxelization operation implemented through the Translation Lookaside Buffer (TLB) begins with data mapping and compression. The point cloud first passes through an index generation unit, mapping it to voxel indices based on the voxel size. The TLB then constructs a voxel table, recording whether each voxel contains a point cloud and the number of point clouds within each voxel. Subsequently, the point cloud undergoes address translation by the TLB, storing each point cloud according to its voxel distribution, thus completing the voxelization of the entire point cloud. All stored point clouds are then processed by the Single Instruction Multiple Data (SIMD) module: each central computing unit has a built-in programmable SIMD unit that performs mean calculation and feature extraction on non-empty voxels, providing structured input for subsequent CNN or Transformer tasks. Finally, the eDRAM CIM macro performs high-dimensional feature extraction on the non-empty voxels.

[0061] Transformers are used to capture global contextual relationships, such as the relationships between different parts of the target to be identified, through a self-attention mechanism. Voxelized data is input into a 3D Transformer network, which ultimately outputs the target's precise distance and location information.

[0062] The method in this embodiment first divides the detected 2D image into multiple coarse sectors according to the scanning angle. Then, it performs target detection on the 2D brightness information data, takes the detected coarse sectors as regions of interest, and outputs the distance data and angle data of the coarse sectors of interest to generate point cloud data. By reducing the invalid point cloud constructed through the regions of interest, the on-chip storage requirements and memory access operations are reduced.

[0063] In some embodiments, target detection based on a 2D convolutional neural network is performed based on two-dimensional brightness information to determine the region of interest where the target to be identified is located, including: Based on two-dimensional brightness information, coarse sector detection is performed on the target to be identified to determine the coarse region of interest. The coarse sector is divided based on the rotation angle. In some embodiments, 3D Transformer computation is performed on the voxelized 3D point cloud to determine the location information of the target to be identified, including: In the coarse region of interest, fine sector division is performed based on logarithmic distance. In each fine sector where the target to be identified is located, 3D detection based on the 3D Transformer model is performed sector by sector to determine the location information of the target to be identified.

[0064] Based on the sectors of interest detected by a 2D convolutional neural network, the target to be identified is divided into fine sectors and 3D detection is performed sector by sector. Finally, the position and distance information of the 3D target are obtained, thereby avoiding a large amount of on-chip point cloud storage. The entire sensor field of view (FOV) is uniformly divided into multiple coarse sector regions according to azimuth angle (or rotation angle), for example, one sector every 10°. Then, the location of the coarse sector of interest is determined by target detection using a 2D convolutional neural network.

[0065] Output: Coarse interest region = one or several angular intervals, which greatly reduces the search space for subsequent processing.

[0066] Within the angular range defined by the coarse region of interest, a finer division is made along the radial direction (distance direction). This division is not linear, but rather based on logarithmic distance. Because the point cloud distribution, due to occlusion and other issues, would result in an overly dense point cloud distribution in sectors near the distance sensor and an overly dense point cloud distribution further away if linearly distributed, thus requiring more on-chip storage space, this invention re-divides each ray (or pixel column at each angle) originating from the sensor according to the logarithmic distance.

[0067] Within each logarithmically spaced sector, the point cloud distribution is also analyzed. This distribution is more uniform, thus requiring more uniform on-chip storage and reducing the need for storage resources.

[0068] By avoiding the calculation of the entire point cloud, the detection of coarse and fine sectors captures the coarse sectors where the target may exist, and performs 3D target detection of fine sectors one by one. This provides high-quality, low-noise input data for subsequent point cloud generation, voxelization and neural network processing, and reduces the large demand for on-chip storage. It is the cornerstone of the whole solution to achieve high efficiency and high accuracy.

[0069] This invention employs a sector division method with M=8 and N=6, first dividing the coarse sectors by rotation angle, and then dividing the fine sectors by distance binning. Because point cloud ranging cannot bypass obstacles, therefore... Figure 9 As shown, the point cloud in the near field is often dense, while the point cloud in the far field is sparser. Therefore, to balance the dense points in the near field and the sparse points in the far field, a logarithmic function is used for distance binning, as shown in the formula: (1) Where Rmin is the closest distance supported by the sensor, Rmax is the maximum distance measured by the sensor, i is the subdivision sector for this distance, and N is the maximum number of subdivision sectors for this distance. This method ensures a balanced distribution of point cloud data within different distance ranges and reduces the risk of local storage overflow.

[0070] In some embodiments, the embedded dynamic random access macro memory can be reconfigurably switched between a storage mode, an in-memory computation mode, and an XOR mode. When the embedded dynamic random access macro memory is in in-memory computation mode, it performs 3D Transformer computation on the voxelized 3D point cloud or 2D convolutional neural network computation on the 2D brightness signal.

[0071] In some embodiments, in storage mode, the model weights or activation values ​​of the convolutional neural network and the Transformer are stored in a single-transistor-single-capacitor storage unit of an embedded dynamic random access macro memory. 3D Transformer calculations are performed on the voxelized 3D point cloud, or 2D convolutional neural network calculations are performed on the 2D brightness information to determine the location information of the target to be identified, including: The embedded dynamic random access macro memory is controlled to be in in-memory computing mode; The model weights or activation values ​​in the single transistor-single capacitor memory cell are read out by opening the word line, and the model activation weight values ​​are multiplied and added using the dual transistor-single capacitor computing unit and the local sense amplifier. By performing charge sharing operations between local cell arrays within and / or between columns, the multi-channel accumulation operation is completed after voxelization, and 3D Transformer calculations are performed using 3D point clouds to determine the position information of the target to be identified.

[0072] like Figure 10As shown, compared to the basic reference scheme model, the vehicle ranging system and location information determination method provided in this embodiment reduce overall energy consumption by 97.5% and system latency by 91.2%.

[0073] Please see Figure 11 This application also provides a location information determination device for a vehicle-mounted ranging system, which can implement the above-described method. The device includes: The acquisition module 1101 is used to acquire two-dimensional brightness information of the target to be identified through an on-chip direct time-of-flight sensor; The determination module 1102 is used to perform target detection based on a 2D convolutional neural network based on the two-dimensional brightness information, and determine the region of interest where the target to be identified is located. The generation module 1103 is used to acquire the three-dimensional point information corresponding to the region of interest through the on-chip direct time-of-flight sensor, and to generate a three-dimensional point cloud using the three-dimensional point information. The calculation module 1104 is used to perform 3D Transformer calculations on the voxelized three-dimensional point cloud to determine the distance information of the target to be identified.

[0074] In some embodiments, the determining module 1102 is configured to: Based on the two-dimensional brightness information, coarse sector detection is performed on the target to be identified to determine the coarse region of interest. The coarse sector is divided based on the rotation angle. The calculation module 1104 is used to divide the coarse region of interest into fine sectors based on logarithmic distance, perform 3D detection based on the 3D Transformer model on a sector-by-sector basis for the fine sector in which the target to be identified is located, and determine the position information of the target to be identified.

[0075] In some embodiments, the embedded dynamic random access macro memory can be reconfigurably switched between a storage mode, an in-memory computation mode, and an XNOR mode. When the embedded dynamic random access macro memory is in the in-memory computation mode, the computation module 1105 is used to run 3D Transformer computation on the three-dimensional point information or run 2D convolutional neural network operations on the two-dimensional brightness information.

[0076] In some embodiments, the computing module 1105 is configured to: Control the embedded dynamic random access macro memory to be in the in-memory computing mode; The model weights or activation values ​​in the single transistor-single capacitor memory cell are read by opening the word line, and the model weights or activation values ​​are multiplied by the dual transistor-single capacitor computing unit and the local sensing amplifier. By performing charge sharing operations between local cell arrays within and / or between columns, the multi-channel accumulation operation after voxelization is completed, and the 3D point cloud is used to perform 3D Transformer calculations to determine the position information of the target to be identified.

[0077] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0078] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0079] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0080] Please see Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1202 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1202 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 using the methods described in the embodiments of this application. The input / output interface 1203 is used to implement information input and output; The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204); The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0081] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0082] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0083] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0084] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0085] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0086] This application provides an on-board ranging system, a location information determination method, and related equipment. The system includes an on-chip direct time-of-flight sensor (DTFS) for acquiring two-dimensional brightness information of a target to be identified; performing target detection based on a 2D convolutional neural network (2DN) to determine the region of interest where the target is located; acquiring three-dimensional point information corresponding to the region of interest using the DTFS, and generating a three-dimensional point cloud using the three-dimensional point information; and performing 3D Transformer calculations on the voxelized three-dimensional point cloud to determine the location information of the target. This system combines an on-chip DTFS, an embedded dynamic random access macro memory, and a central computing unit, effectively improving the computational efficiency of the system under mixed CNN and Transformer loads while ensuring low power consumption and low latency, reducing off-chip memory access power consumption, and improving the real-time performance and reliability of overall three-dimensional environment perception.

[0087] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0088] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0089] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0091] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0092] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0094] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A vehicle-mounted ranging system, characterized in that, include: An on-chip direct time-of-flight sensor is used to acquire image data in the direction of a target, the image data including two-dimensional brightness information and three-dimensional point position information; An embedded dynamic random access macro memory is directly connected to the central computing unit. The embedded dynamic random access macro memory supports multiple working modes, which are used to perform convolutional neural network calculations for 2D target detection on the two-dimensional brightness information output by the on-chip direct time-of-flight sensor, or to perform 3D Transformer calculations on the voxelized three-dimensional point information through a dynamic sparse voxel Transformer to generate the position information of the target direction. The central computing unit is connected to the embedded dynamic random access macro memory and the on-chip direct time-of-flight sensor, respectively. It is used to control the embedded dynamic random access macro memory and the on-chip direct time-of-flight sensor, perform voxelization processing on the three-dimensional point information, and send the voxelized three-dimensional point information to the embedded dynamic random access macro memory to obtain the position information output from the embedded dynamic random access macro memory.

2. The system according to claim 1, characterized in that, The system also includes: a preprocessing module, a reduced instruction set computing processor, and the direct time-of-flight sensor including a readout circuit; The preprocessing module is connected to the readout circuit and is used to read the two-dimensional brightness information or the three-dimensional point information from the direct time-of-flight sensor, and construct point cloud information based on the two-dimensional brightness information or the three-dimensional point information. The preprocessing module includes: a first-in-first-out buffer, a point cloud construction unit, and a noise reduction unit; The reduced instruction set computing processor is connected to the readout circuit and the noise reduction unit respectively, and is used to control the preprocessing module and the direct time-of-flight sensor.

3. The system according to claim 1, characterized in that, The embedded dynamic random access macro memory integrates multiple single-transistor-single-capacitor memory units and dual-transistor-single-capacitor computing units.

4. The system according to claim 1, characterized in that, The system further includes a phase-locked loop (PLL); the PLL is connected to the on-chip direct time-of-flight sensor, and the PLL is used to provide a high-speed clock signal to the on-chip direct time-of-flight sensor.

5. A method for determining the location information of a vehicle-mounted ranging system, characterized in that, The method comprises: an embedded dynamic random access macro memory operating in the system as described in any one of claims 1 to 4; Two-dimensional brightness information of the target to be identified is obtained through an on-chip direct time-of-flight sensor; Based on the two-dimensional brightness information, target detection based on a 2D convolutional neural network is performed to determine the region of interest where the target to be identified is located. The on-chip direct time-of-flight sensor acquires the three-dimensional point information corresponding to the region of interest, and uses the three-dimensional point information to generate a three-dimensional point cloud. The location information of the target to be identified is determined by performing 3D Transformer calculations on the voxelized 3D point cloud using a dynamic sparse voxel Transformer.

6. The method according to claim 5, characterized in that, The step of performing target detection based on the two-dimensional brightness information using a 2D convolutional neural network to determine the region of interest where the target to be identified is located includes: Based on the two-dimensional brightness information, coarse sector detection based on a 2D convolutional neural network is performed on the target to be identified to determine the coarse region of interest. The coarse sector is divided based on the rotation angle. The step of performing 3D Transformer calculations on the voxelized 3D point cloud to determine the location information of the target to be identified includes: In the coarse region of interest, fine sector division is performed based on logarithmic distance. For each fine sector in which the target to be identified is located, 3D detection based on the 3D Transformer model is performed sector by sector to determine the location information of the target to be identified.

7. The method according to claim 5, characterized in that, The embedded dynamic random access macro memory can be reconfigured and switched between storage mode, in-memory computation mode and XNOR mode. In the in-memory computation mode, the embedded dynamic random access macro memory is used to run 3D Transformer calculations on the three-dimensional point information or to run 2D convolutional neural network operations on the two-dimensional brightness information.

8. The method according to claim 7, characterized in that, In the storage mode, the model of the 3DTransformer or the weights or activation values ​​of the 2D convolutional neural network are stored in the single-transistor-single-capacitor storage unit of the embedded dynamic random access macro memory. The step of performing 3DTransformer calculations on the voxelized 3D point cloud to determine the position information of the target to be identified includes: Control the embedded dynamic random access macro memory to be in the in-memory computing mode; The model weights or activation values ​​in the single transistor-single capacitor memory cell are read by opening the word line, and the model weights or activation values ​​are multiplied by the dual transistor-single capacitor computing unit and the local sensing amplifier. By performing charge sharing operations between local cell arrays within and / or between columns, the multi-channel accumulation operation after voxelization is completed, and the 3D point cloud is used to perform 3D Transformer calculations to determine the position information of the target to be identified.

9. A location information determination device for a vehicle-mounted ranging system, characterized in that, The apparatus is used to implement the method as described in any one of claims 5 to 8, comprising: The acquisition module is used to acquire two-dimensional brightness information of the target to be identified through the on-chip direct time-of-flight sensor; The determination module is used to perform target detection based on a 2D convolutional neural network based on the two-dimensional brightness information, and determine the region of interest where the target to be identified is located. The generation module is used to acquire the three-dimensional point information corresponding to the region of interest through the on-chip direct time-of-flight sensor, and to generate a three-dimensional point cloud using the three-dimensional point information; The calculation module is used to perform 3DTransformer calculations on the voxelized 3D point cloud using a dynamic sparse voxel Transformer to determine the location information of the target to be identified.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 5 to 8.

Images