Integrated sensing system, module structure, generation method and related equipment

Abstract

The invention relates to the technical field of vehicle control, in particular to an integrated sensing system, a module structure, a generation method and related equipment wherein the integrated sensing system comprises: a sensing module (51) comprising at least two sensing assemblies (501), the sensing assemblies (501) being used for monitoring and acquiring initial information of a detection object; and a processing device (52) configured to receive the initial information and obtain point cloud information of the detection object based on the initial information corresponding to the at least two sensing assemblies (501). The integrated sensing system provided by the invention is used for optimizing the sensing system required by intelligent driving.

Description

Technical Field

[0001] This application relates to the technical field of vehicle control, and particularly to an integrated sensing system, a module structure, a generation method, and related devices. Background Art

[0002] The sensor system of an autonomous vehicle generally includes four types of radars: Ultrasonic Radar, mm Wave Radar, Lidar, and Infrared Radar. Among them, the ultrasonic radar is a basic and necessary in-vehicle sensor device, and the lidar and mm wave radar are key sensor devices for high-level intelligent driving vehicles.

[0003] For cameras, ultrasonic radars, mm wave radars, lidars, etc., different types of sensors have their own advantages and disadvantages. A single sensor is difficult to meet various complex application scenarios of intelligent driving; while the technical route of multi-sensor fusion can perform multi-level and multi-space combination processing on the information and data of multiple sensors or multiple sources, which can improve the perception accuracy of intelligent driving in extreme environments, avoid the performance limitations of a single sensor and the safety risks brought by single-system failures, and thus has become the mainstream technical route for the perception layer of high-level intelligent driving at home and abroad.

[0004] The lidar system includes two parts: a front-end data sensing structure and a back-end data processing structure. Among them, the core processor in the front-end data sensing structure is expensive, accounting for a large proportion in the cost composition of the lidar. When the lidar is damaged, the cost of replacing the entire system is relatively high.

[0005] In addition, the core processor of the front-end data sensing structure transmits the original data to the back-end data processing structure after a certain processing, which is not only a waste of computing power but also results in a loss of information dimension. Summary of the Invention

[0006] This application provides an integrated sensing system, a module structure, a generation method, and related devices to optimize the sensing system required for intelligent driving.

[0007] To achieve the above object, in the first aspect of this application, an integrated sensing system (50) is provided, including: a sensing module (51), including at least two sensing components (501), where the sensing components (501) are used to monitor and obtain the initial information of the detection object; a processing device (52), configured to receive the initial information and obtain the point cloud information of the detection object based on the initial information corresponding to the at least two sensing components (501).

[0008] In some embodiments, the processing device (52) is further configured to provide a clock synchronization signal to the sensing module (51), and the at least two sensing components (501) in the sensing module (51) perform clock calibration in response to the clock synchronization signal.

[0009] In some embodiments, the processing device (52) includes: a clock module (504) connected to the sensing module (51), and the clock module (504) is configured to provide the clock synchronization signal.

[0010] In some embodiments, the sensing module (51) includes at least one passive sensing component (520) and at least one active sensing component (510).

[0011] In some embodiments, the active sensing component (510) is further configured to provide a synchronization trigger signal, and the passive sensing component (520) monitors and acquires the initial information in response to the synchronization trigger signal.

[0012] In some embodiments, the active sensing component (510) provides the synchronization trigger signal based on the emission time of the detection signal and a preset delay.

[0013] In some embodiments, the synchronization trigger signal is provided once in each cycle of the detection signal.

[0014] In some embodiments, the synchronization trigger signal is provided once in multiple cycles of the detection signal.

[0015] In some embodiments, in the cycle of the detection signal for providing the synchronization trigger signal, the delay time of the preset delay is the same.

[0016] In some embodiments, the initial information provided by the active sensing component (510) includes timestamp information.

[0017] In some embodiments, the sensing module (51) further includes a serializer (530), and the processing device (52) further includes a deserializer (540). The serializer (530) is connected to the at least two sensing components (501), and the deserializer (540) is connected to the serializer (530).

[0018] In some embodiments, the sensing module (51) includes a plurality of serializers (530), and the at least two sensing components (501) form a plurality of serial sensing links based on the plurality of serializers (530).

[0019] In some embodiments, the initial information provided by the at least two sensing components (501) is serialized based on the serializer (530) to generate serial information, and the serial information is sent to the processing device (52).

[0020] In some embodiments, the processing device (52) deserializes the serial information based on the deserializer (540) to obtain the initial information.

[0021] In some embodiments, the processing device (52) includes: a data fusion module (503) configured to perform information fusion on the initial information obtained by the at least two sensing components (501) to obtain the point cloud information.

[0022] In some embodiments, the data fusion module (503) outputs the point cloud information of the detection object based on the spatial cubic model matrix.

[0023] In some embodiments, the data fusion module (503) includes: a first processing unit configured to establish the spatial cubic model matrix and fill the initial information corresponding to each sensing component into the spatial cubic model matrix; a second processing unit configured to output the point cloud information of the detection object based on the spatial cubic model matrix.

[0024] In some embodiments, the data fusion module (503) further includes: a third processing unit configured to convert the initial information corresponding to each sensing component (501) to the same coordinate system based on preset information; the first processing unit establishes the spatial cubic model matrix based on the coordinate system.

[0025] In some embodiments, the preset information includes: the internal parameters of the sensing component (501) and the corresponding relationship of the sensing component (501) in physical space.

[0026] In some embodiments, the first processing unit establishes the spatial cubic model matrix based on the maximum field of view angle of the passive sensing component (520) and / or the spatial resolution of the active sensing component (510).

[0027] A second aspect of the present application provides a module structure (600) applied to the sensing module (51) provided in the first aspect above, including: a carrier bottom plate (601), at least two sensing components (501), and a processing module (602); the sensing components (501) and the processing module (602) are arranged on the carrier bottom plate (601); the processing module (602) is configured to control the sensing components (501) to detect and obtain the initial information of the detection object and transmit the initial information to the processing device (52).

[0028] In some embodiments, the module structure (600) further includes: a receiving cavity (603) disposed on the carrier bottom plate (601) and forming a closed space with the carrier bottom plate (601), and the closed space is at least used to receive the sensing component (501).

[0029] In some embodiments, the surface of the receiving cavity (603) has a window (604), and the window (604) is used for the sensing component (501) to obtain the initial information.

[0030] In some embodiments, heat dissipation patterns (605) are further provided on the surface of the receiving cavity (603).

[0031] In some embodiments, the module structure (600) further includes: a positioning component (606) disposed at corresponding positions of the receiving cavity (603) and the carrier bottom plate (601) for fixing the receiving cavity (603) and the carrier bottom plate (601).

[0032] In some embodiments, the positioning component (606) is further used to fix the carrier bottom plate (601) on the application carrier of the module structure (600).

[0033] In some embodiments, the sensing component (501) includes an active sensing component (510) and a passive sensing component (520).

[0034] A third aspect of the present application provides a method for generating an intelligent driving strategy, which is applied to the integrated sensing system (50) provided in the above embodiments, and includes: obtaining initial information obtained by at least two sensing components (501) monitoring a detection object; performing information fusion processing on the initial information to obtain point cloud information of the detection object.

[0035] In some embodiments, the method for generating an intelligent driving strategy further includes: obtaining an intelligent driving strategy based on the point cloud information corresponding to multiple target objects.

[0036] A fourth aspect of the present application provides a computer-readable storage medium, and the computer-readable storage medium stores instructions, and the instructions are suitable for being loaded by a processor to execute the method for generating an intelligent driving strategy provided in the third aspect above.

[0037] A fifth aspect of the present application provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the method for generating an intelligent driving strategy provided in the third aspect above is implemented.

[0038] The sixth aspect of the present application provides a controller, which includes a processor and a memory. The processor is connected to the memory. The memory is used to store a computer program, and the processor is used to execute the computer program to implement the integrated sensing system (50) provided in the first aspect above.

[0039] The seventh aspect of the present application provides a vehicle, which includes the integrated sensing system (50) provided in the first aspect above, or includes the module structure (600) provided in the second aspect above, or includes the computer-readable storage medium provided in the fourth aspect above, or includes the computer program product provided in the fifth aspect above, or includes the controller provided in the sixth aspect above.

[0040] For the integrated sensing system provided by the present application, the point cloud processing function of the sensing component is transferred to the processing device, so that the sensing component only needs to complete signal acquisition, and the data processing process is executed on the backend processing device. The product standardization degree is further improved, the scenario algorithm can be flexibly customized according to application requirements, and the environmental adaptability is stronger.

[0041] In some embodiments, integrating different types of sensors at the physical space level can effectively improve the integration degree of the entire system, avoid the impact on vehicle styling design caused by installing each sensor at different positions on the vehicle body, and is easy to install. Description of the Drawings

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for description in the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 It is a schematic structural diagram of a discrete architecture lidar system;

[0044] Figure 2 It is a schematic diagram of the setting of multiple sensors;

[0045] Figure 3 It is a schematic structural diagram of the integrated sensing system provided by the embodiment of the present application;

[0046] Figure 4 It is a schematic structural diagram of the integrated sensing system provided by the embodiment of the present application with an active sensing component and a passive sensing component;

[0047] Figure 5 It is a specific structural example of the integrated sensing system provided by the embodiment of the present application;

[0048] Figure 6Schematic diagram of the method steps corresponding to data fusion by the data fusion module provided in the embodiments of the present application;

[0049] Figure 7 Schematic diagram of the field of view angle ranges of the respective sensing components provided in the embodiments of the present application;

[0050] Figure 8 Schematic diagram of the structure of the spatial cube model matrix established in the embodiments of the present application;

[0051] Figure 9 Schematic diagram of the structure of the sensing processing system under the specific sensing device structure provided in the embodiments of the present application;

[0052] Figure 10 Provided in the embodiments of the present application Figure 9 Schematic diagram of the working timing of the sensing processing system corresponding to the example;

[0053] Figure 11 Schematic diagram of the structure of the rotary mirror and the grating code disk provided in the embodiments of the present application;

[0054] Figure 12 Provided in the embodiments of the present application Figure 9 Schematic diagram of the working process of the sensing processing system corresponding to the example;

[0055] Figure 13 Provided in the embodiments of the present application Figure 10 Schematic diagram of the working timing of the active sensing component and the passive sensing component corresponding to the example;

[0056] Figure 14 Provided in the embodiments of the present application Figure 12 Schematic diagram of the working process of the corresponding sensing processing system in combination with the passive sensing component;

[0057] Figure 15 Schematic diagram of the external structure of the module structure provided in the embodiments of the present application;

[0058] Figure 16 Schematic diagram of the internal structure of the module structure provided in the embodiments of the present application. Detailed implementation manners

[0059] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present application.

[0060] In the description of the present application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present application, "a plurality" means two or more unless otherwise specifically defined.

[0061] In the present application, the word "exemplary" is used to mean "serving as an example, instance, or illustration". Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or superior to other embodiments. The following description is provided to enable any person skilled in the art to make and use the present application. In the following description, details are set forth for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the present application can be practiced without these specific details. In other instances, well-known structures and processes are not described in detail to avoid unnecessary details from obscuring the description of the present application. Therefore, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0062] Meanwhile, the present application uses specific words to describe the embodiments of the present application. Such as "an embodiment", "one embodiment", and / or "some embodiments" mean a certain feature, structure, or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that the "an embodiment" or "one embodiment" or "an alternative embodiment" mentioned twice or more at different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the present application can be appropriately combined.

[0063] Similarly, it should be noted that, in order to simplify the presentation of the disclosure of the present application and thus assist in the understanding of one or more embodiments of the application, in the foregoing description of the embodiments of the present application, various features are sometimes grouped together into one embodiment, drawing, or description thereof. However, this method of disclosure does not mean that the features required by the subject matter of the present application are more than those recited in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0064] In some embodiments, numbers are used to describe components and the quantity of attributes. It should be understood that such numbers used in the description of embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise stated, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may vary according to the characteristics required by individual embodiments. In some embodiments, numerical parameters should consider the specified significant digits and adopt the method of retaining the general number of digits. Although the numerical ranges and parameters used in some embodiments of the present application to confirm the breadth of their scope are approximate values, in specific embodiments, such numerical settings are made as precise as possible within the feasible range.

[0065] For each patent, patent application, patent application publication, and other materials cited in the present application, such as articles, books, specifications, publications, documents, etc., their entire contents are hereby incorporated into the present application by reference, except for application history documents that are inconsistent with or conflict with the content of the present application, and also except for documents that limit the broadest scope of the claims of the present application (currently or subsequently appended to the present application). It should be noted that if there are inconsistencies or conflicts between the descriptions, definitions, and / or uses of terms in the attached materials of the present application and the example content of the present application, the descriptions, definitions, and / or uses of terms in the present application shall prevail.

[0066] With the development of intelligent driving technology, the requirements for the perception accuracy and richness of environmental information by automobiles are also constantly increasing. Automotive sensors mainly include cameras, ultrasonic radars, millimeter-wave radars, lidars, etc. Different types of sensors have their own advantages and disadvantages, and a single sensor is difficult to meet various complex application scenarios of intelligent driving; while the technical route of multi-sensor fusion can perform multi-level and multi-space combined processing on the information and data of multiple sensors or multiple sources, which can improve the perception accuracy of intelligent driving in extreme environments and avoid the safety risks brought by the performance limitations of a single sensor and single-system failures. Therefore, it has become the mainstream technical route for the perception layer of high-level intelligent driving at home and abroad.

[0067] Reference Figure 1 , Figure 1 FIG. is a schematic structural diagram of a lidar system with a discrete architecture. The lidar system 1 includes a front-end data sensing structure and a rear-end data processing structure. Among them, the front-end data sensing structure is the lidar sensor 2, and the rear-end data processing structure is the information processing system 3.

[0068] In some embodiments, the lidar sensor 2 includes a transmitting module 23, an optical system 22, a scanning module 21, a receiving module 24, and a core processor 25. The information processing system 3 includes a domain control SoC (System on a Chip) 31.

[0069] For lidar system 1, the transmitting module 23, optical system 22, scanning module 21, and receiving module 24 cooperate together to transmit and receive laser signals to complete the perception of the target.

[0070] In one example, the receiving module 24 mainly consists of an APD (avalanche photodiode), SiPM (silicon photomultiplier), TIA (transimpedance amplifier), AMP (operational amplifier), and ADC (analog-to-digital converter) to complete the processes of photoelectric conversion, signal amplification, and analog-to-digital conversion.

[0071] The core processor 25 acts as the brain of the lidar for calculation, undertaking functions such as system logic control, ranging calculation, point cloud data processing, packet assembly and transmission, and complex scenario algorithm processing to obtain the point cloud information of the targets in the detection area of the lidar system 1. The core processor 25 is generally composed of a high-computing-power field programmable gate array (FPGA, which can be attached with an ARM core processor or an independent MCU) or an application specific integrated circuit (ASIC) chip.

[0072] After the point cloud information obtained and processed by the lidar sensor 2 is transmitted to the domain control SoC 31, the domain control SoC 31 performs data fusion on the point cloud information obtained by the lidar sensor 2 and other sensors, and makes intelligent driving decisions based on the fusion information, enabling the lidar system 1 to achieve the precise measurement and perception functions of the targets in the detection area and improving the application experience of the intelligent driving system.

[0073] However, for the lidar system 1 adopting a discrete component architecture, the structural complexity is high and the integration level is low, which means that a relatively large space needs to be reserved inside the lidar system 1 to accommodate each module, and it is very difficult to further reduce the volume.

[0074] In addition, the lidar system 1 includes a high-computing-power core processor 25. The high-computing-power core processor 25 is expensive and accounts for a large proportion in the total cost of the lidar system 1. When the lidar system 1 is damaged, the cost of replacing the entire lidar system 1 is relatively high. At the same time, the core processor 25 inside the lidar system 1 processes the original data and then transmits it to the domain control SoC 31, which may cause waste of computing power and may also result in loss of information dimensions.

[0075] In addition, referring to Figure 2 , in some applications, automotive sensors 401 such as cameras, ultrasonic radars, millimeter-wave radars, and lidars are distributed at different positions on the vehicle body, obtaining real-time external environment information and converting it into electrical signals for transmission to the central processing unit of the vehicle to assist intelligent driving decision-making behaviors.

[0076] Since different automotive sensors 401 are composed of different modules and are distributed at different positions inside and outside the vehicle body, their layout and installation positions need to be considered during the vehicle model design stage. At the same time, in order to ensure that the data obtained by the sensors can be accurately fused and processed, external parameter joint calibration is also required - to determine the position and orientation information of each sensor relative to the vehicle body coordinate system, as well as their internal parameters (such as field of view, scanning range, etc.). Through these calibrations, the data obtained by different automotive sensors 401 can be converted to a unified reference coordinate system, enabling the central processing unit to accurately understand and integrate this data. In addition, since the automotive sensors 401 are distributed at different positions inside and outside the vehicle body, the physical distance will cause a certain time delay, and there are certain difficulties in time synchronization.

[0077] In some applications, a carrier is used to integrate multiple cameras and one or more lidars in a physical space, which greatly improves the spatio-temporal alignment accuracy of each sensor during the external parameter joint calibration stage. However, each sensor is still an independent individual. In particular, the lidar still retains a complex point cloud information processing unit inside, and there is a loss of information dimension in the processed point cloud data, resulting in a decrease in the quality of data fusion (fusion with the image data of the camera).

[0078] In summary, the separate multi-sensor fusion scheme is technically complex, requires multiple calibration processes, and the different position distributions of the sensors will bring errors in time and space synchronization. In addition, although the physical-level integration of multi-sensors brings an improvement in spatio-temporal synchronization accuracy, the loss of its information dimension will lead to a decrease in the quality of data fusion.

[0079] The embodiment of the present application provides an integrated sensing system to optimize the sensing system required for intelligent driving.

[0080] Refer to Figure 3 , Figure 3 which is a schematic structural diagram of the integrated sensing system provided in this embodiment. The integrated sensing system 50 includes a sensing module 51 and a processing device 52. Among them, the sensing module 51 includes at least two sensing components 501, and the sensing component 501 is used to detect and obtain the initial information of the detection object; the processing device 52 is configured to receive the initial information and obtain the point cloud information of the detection object based on the initial information corresponding to at least two sensing components 501.

[0081] For the sensing component 501, each sensing component 501 is used to implement Figure 2 the sensing function of the automotive sensor 401 shown.

[0082] Among them, the detection object is the target object of the sensing device corresponding to the integrated sensing system 50 in the detection area. For example, if the integrated sensing system 50 is installed on a vehicle, the detection objects are environmental objects such as the road conditions and obstacles around the current vehicle.

[0083] The initial information is the raw data obtained by the sensing component 501 through monitoring the detection object. For example, if the sensing component 501 is a lidar, the initial information is the raw data such as the laser emission time, the laser reception time, the laser intensity, and the laser reception angle; if the sensing component 501 is an image sensor, the initial information is the image data obtained by monitoring the detection object.

[0084] The point cloud information is obtained after processing the initial information. In one example, the point cloud information includes a dataset of points in space and can represent a three-dimensional shape or object. Among them, the position of each point in the point cloud information is described by a set of Cartesian coordinates, and some points may contain color information (R, G, B), distance information, intensity information, etc.

[0085] For the integrated sensing system provided in this embodiment, the point cloud processing function of the sensing component 501 is transferred to the processing device 52, so that the sensing component 501 only needs to complete signal acquisition, and the data processing process is executed by the processing device 52 at the back end. The product standardization degree is further improved, the scenario algorithm can be flexibly customized according to application requirements, and the environmental adaptability is stronger. In addition, integrating different types of sensors at the physical space level can effectively improve the integration degree of the entire system, avoid the impact on the vehicle styling design caused by installing each sensor at different positions on the vehicle body, and is easy to install.

[0086] In addition, transferring the information processing part to the processing device 52 is more convenient for online upgrading of the processing algorithm. The product standardization degree is further improved, the scenario algorithm can be flexibly customized according to application requirements, and the environmental adaptability is stronger. At the same time, the sensing component 501 is freed from the dependence on high-complexity proprietary chips, the cost of the system is reduced, and when the product is damaged, only the sensing component 501 without a core processor needs to be replaced.

[0087] Reference Figure 4 and Figure 5 , Figure 4 is a schematic structural diagram of the integrated sensing system provided in this embodiment, which has an active sensing component and a passive sensing component. Figure 5 is a specific structural example of the integrated sensing system provided in this embodiment.

[0088] In some embodiments, the sensing module 51 includes at least one passive sensing component 520 and at least one active sensing component 510.

[0089] The passive sensing component 520 mainly refers to a sensor that does not generate energy by itself and makes corresponding actions or outputs corresponding physical quantity changes by acquiring signals generated or reflected by the detected object. It can include a monocular camera, a binocular camera, a trinocular camera, a telephoto camera, a wide-angle camera, or a far-infrared camera, etc. Additionally, the passive sensing component 520 mentioned in this embodiment is a sensing front end, that is, the passive sensing component 520 does not include an ISP (Image Signal Processing) processing module.

[0090] The active sensing component 510 refers to a sensor that emits energy to the object to be measured and measures the change in the energy reflected back or projected out. It can include a lidar, a millimeter-wave radar, or an ultrasonic radar, etc. Additionally, the active sensing component 510 mentioned in this embodiment is a sensing front end, that is, the active sensing component 510 does not include the part of a large computing power dedicated processing chip.

[0091] The sensing module 51 includes an active sensing component 510 and a passive sensing component 520 to achieve the acquisition of different types of initial information, improve the acquisition of different types of information of the integrated sensing system 50, and also improve the information richness or redundancy.

[0092] It should be noted that this embodiment does not limit the number of sensing components 501, active sensing components 510, or passive sensing components 520 in the sensing module 51. The number of sensing components 501 can include one or more according to the actual application requirements.

[0093] In some embodiments, the active sensing component 510 is further configured to provide a synchronous trigger signal, and the passive sensing component 520 monitors and acquires the initial information in response to the synchronous trigger signal.

[0094] In an example, since the time for a lidar and a camera to output a frame of data is inconsistent (for example, it usually takes 100 ms for a lidar to output a frame of point cloud, while a camera usually has an instantaneous exposure imaging), triggering the passive sensing component 520 by the active sensing component 510 can ensure the alignment accuracy in time between the active sensing component 510 and the passive sensing component 520.

[0095] In some embodiments, the active sensing component 510 provides a synchronous trigger signal based on the emission moment of the detection signal and a preset delay. To achieve that at a specific moment when a lidar completes a frame of data scanning or detects a specific target, the camera is immediately triggered to take a picture. This can ensure that the image captured by the camera is as close as possible to the detection time of the target by the lidar, thereby greatly improving the alignment accuracy in time between the active sensing component 510 and the passive sensing component 520 and providing more accurate spatio-temporal information for subsequent target recognition and analysis.

[0096] In some embodiments, the synchronization trigger signal is provided once in each period of the detection signal.

[0097] In some embodiments, the synchronization trigger signal is provided once in multiple periods of the detection signal.

[0098] It should be noted that in the description of this embodiment, the active sensing component 510 is a lidar, and at this time, the detection signal corresponding to the active sensing component 510 is a laser signal.

[0099] In some embodiments, in the period of the detection signal that provides the synchronization trigger signal, the delay time of the preset delay is the same. By triggering the camera to take pictures at the same moment when the lidar completes a frame of data scanning, the alignment accuracy of the active sensing component 510 and the passive sensing component 520 in time is further ensured.

[0100] In some embodiments, the processing device 52 is further configured to provide a clock synchronization signal to the sensing module 51, and at least two sensing components 501 in the sensing module 51 perform clock calibration in response to the clock synchronization signal.

[0101] In some embodiments, the processing device 52 includes a clock module 504. The clock module 504 is connected to the sensing module 51, and the clock module 504 is configured to provide a clock synchronization signal.

[0102] In practical applications, different sensing components 501 of the sensing module 51 need to perform clock synchronization to ensure the validity of the initial information sent to the processing device 52, thereby ensuring the accuracy of multimodal fusion of the data fusion module 503. The processing device 52 provides a clock synchronization signal through the clock module 504 for different sensing components 501 to perform clock calibration, and then adds timestamp information to the collected data according to the calibrated time to achieve timestamp synchronization of different sensing components 501.

[0103] Correspondingly, in some embodiments, the initial information provided by the active sensing component 510 includes timestamp information.

[0104] In some embodiments, the sensing module 51 further includes a serializer 530, and the processing device 52 further includes a deserializer 540. The serializer 530 is connected to at least two sensing components 501, and the deserializer 540 is connected to the serializer 530.

[0105] In some embodiments, the initial information provided by at least two sensing components 501 is serially processed based on the serializer 530 to generate serial information, and the serial information is sent to the processing device 52.

[0106] In some embodiments, the deserializer 540 performs deserialization processing on the serial information to obtain the initial information.

[0107] Specifically, the deserialiser 540 is connected to the serializer 530. The main function of the serializer 530 is to distribute the power-on / off and clock synchronization signals transmitted from the deserialiser 540 to each sensing component 501. At the same time, the initial information obtained by each sensing component 501 is serially processed to generate serial information, and the serial information is transmitted to the deserialiser 540 through a physical interface. The deserialiser 540 performs deserialization processing on the serial information to obtain the initial information generated by the sensing module 51, so as to improve the efficiency and reliability of data transmission.

[0108] In one example, the deserialiser 540 internally includes an I2C interface to transmit control signals through I2C, such as power-on / off signals, time synchronization trigger signals, etc.

[0109] In some embodiments, the sensing module 51 includes multiple serializers 530, and at least two sensing components 501 form multiple serial sensing links based on the multiple serializers 530.

[0110] Specifically, the deserialiser 540 can be connected to multiple serializers 530 to form multiple links. The number of sensing components 501 on each link can be the same or different. Additionally, the types of sensing components 501 on each link can be the same or different.

[0111] In some embodiments, the processing device 52 includes a data fusion module 503, and the data fusion module 503 is configured to perform information fusion on the initial information obtained by at least two sensing components 501 to obtain point cloud information.

[0112] In one example, the initial information at least includes first initial information corresponding to the passive sensing component 520 and second initial information corresponding to the active sensing component 510.

[0113] Specifically, the data fusion module 503 mainly performs multi-modal fusion on the initial information uploaded by different types of sensing components 501.

[0114] Reference Figure 6 , Figure 6 is a schematic diagram of the method steps corresponding to the data fusion performed by the data fusion module provided in this embodiment. In some embodiments, the data fusion module 503 performs multi-modal data fusion including: outputting the point cloud information of the detection object based on the spatial cubic model matrix.

[0115] In some embodiments, the data fusion module 503 includes a first processing unit and a second processing unit. Among them, the first processing unit is configured to establish a spatial cubic model matrix and fill the initial information corresponding to each sensing component into the spatial cubic model matrix; the second processing unit is configured to output the point cloud information of the detection object based on the spatial cubic model matrix.

[0116] In some embodiments, the data fusion module 503 converts the initial information corresponding to each sensing component into the same coordinate system based on preset information; establishes a spatial cubic model matrix, and fills the initial information corresponding to each sensing component into the spatial cubic model matrix; outputs the point cloud information of the detection object based on the spatial cubic model matrix.

[0117] Specifically, the data fusion module 503 further includes a third processing unit, a first processing unit and a second processing unit. Among them, the third processing unit is configured to convert the initial information corresponding to each sensing component into the same coordinate system based on preset information; the first processing unit establishes a spatial cubic model matrix based on the coordinate system.

[0118] Among them, based on preset information, the initial information corresponding to each sensing component 501 is converted into the same coordinate system, where the preset information includes the internal parameters of the sensing component 501 and the corresponding relationship of the sensing component 501 in physical space.

[0119] In some embodiments, the preset information is the internal parameters of the sensing component 501 and the corresponding relationship of the sensing component 501 in physical space obtained during the module calibration process.

[0120] Reference Figure 7 and Figure 8 , Figure 7 is a schematic diagram of the field of view angle range of each sensing component provided in this embodiment, Figure 8 is a schematic diagram of the structure of the spatial cubic model matrix established in this embodiment.

[0121] Establish a spatial cubic model matrix, and fill the initial information corresponding to each sensing component into the spatial cubic model matrix, including: establishing a spatial cubic model matrix m*w*h with the maximum field of view angle of each sensing component 501 and / or the spatial resolution of the lidar, and each small cube contains information such as RGBLIH (RGB is color channel information, L is distance information, I is intensity information, H is thermal radiation intensity information). Fill the distance and intensity information of the lidar into the matrix, and fill the corresponding items of the points without distance and intensity with 0. Fill the data of each camera into the matrix according to a certain order and rule.

[0122] In one example, reference Figure 7, the field of view (FoV) and resolution of each sensor are inconsistent. Assume that the FoV of the binocular camera is 120° * 64°, and the resolution is 3840×2160 (0.03° * 0.029°); the FoV of the telephoto camera is 30° * 15°, and the resolution is 3840×2160 (0.03° * 0.009°); the FoV of the far-infrared camera is 48° * 38°, and the resolution is 640×512 (0.07° * 0.07°); the FoV of the lidar is 120° * 20°, the resolution is 0.1° * 0.1°, the maximum detection distance is 200m, and the distance angle resolution is 0.03m. In order to provide comprehensive data as much as possible, the size of the spatial matrix model is determined by the maximum FoV of each sensor and the maximum detection distance of the lidar, and the angular resolution and distance resolution of the lidar are used as the size standard of the cube; the finally established matrix model is as shown in Figure 8 : m = 200 / 0.03 = 6667; w = 120° / 0.1° = 1200; h = 64° / 0.1° = 640.

[0123] In one example, filling the data of each camera into the matrix according to a certain order and rules includes: (1) supplementing the RGBLIH data one by one from left to right and top to bottom according to the established matrix model. (2) Processing in the order of the binocular camera, the long-focus camera, and the far-infrared camera. (3) When processing the data of the binocular camera, first divide according to the distance: taking the working area of the long-focus camera as the boundary, if the long-focus camera mainly detects scenes beyond 150m, then the matrix data within or less than 150m uses the data of the binocular camera. Since the pixel resolution is much higher than that of the lidar, a single cube of the model matrix will cover multiple pixels of the binocular camera. If the RGB data of each pixel is relatively close, the average value is taken; if the difference is large, the centroid method or the weighted method can be used for processing and then taking the value; at the same time, the distance corresponding to the current cubic voxel can be obtained using the binocular camera parallax principle for comparison with the ranging value of the lidar, and the distribution of the surrounding lidar point cloud is used to confirm whether this point is a noise point. If it is a noise point, correct L to 0. (4) When processing the data of the long-focus camera, also taking the working area of the long-focus camera as the boundary, if the long-focus camera mainly detects scenes beyond 150m, then the matrix data beyond 150m uses the RGB data of the long-focus camera; similarly, since the pixel resolution of the long-focus camera is much higher than that of the lidar, a single cube of the model matrix will cover multiple pixels of the long-focus camera. If the RGB data of each pixel is relatively close, the average value is taken; if the difference is large, the centroid method or the weighted method can be used for processing and then taking the value; the matrix data outside the detection area is not processed. (5) When processing the data of the far-infrared camera, the same method as above is used to fuse and process the pixels, and the matrix data outside the detection area is not processed.

[0124] Based on the spatial cubic model matrix, the point cloud information of the detection object is output, that is, the data fusion module 503 outputs the processed multi-modal fusion data.

[0125] The data fusion module 503 performs multi-modal data fusion on the initial information corresponding to different types of sensing components 501 and outputs it in the format of multi-dimensional pixels to obtain the complete information of the homologous target, so that the integrated sensing system 50 can maintain efficient and reliable operation in complex and changing environments and give full play to its advantages in target detection, environmental perception, data collection, etc.

[0126] In some embodiments, the processing device 52 also generates an intelligent driving strategy based on the point cloud information. Combining the above description, the data processing of the processing device 52 includes two processes. One is to generate point cloud information based on the initial information corresponding to different sensing components, and the other is to generate an intelligent driving strategy based on the point cloud information.

[0127] For the interaction between a single sensing component 501 and a processing device 52 (which can be regarded as a sensing and processing system 100), as well as the interaction between multiple sensing components 501 and a processing device 52, the following will be described in detail with reference to the accompanying drawings.

[0128] Reference Figure 9 , Figure 9 FIG. is a schematic structural diagram of a sensing and processing system under the specific sensing device structure provided in this embodiment. In this embodiment, the sensing component 501 is a lidar to describe the sensing and processing system 100 provided in this embodiment.

[0129] In some embodiments, the sensing component 501 includes a sensing module and a receiving and processing module 122. The sensing module generates a laser signal based on the control timing issued by the receiving and processing module 122; the receiving and processing module 122 obtains initial information based on the received laser signal and transmits the initial information to the processing device 101 for the processing device 101 to perform point cloud calculation based on the initial information to obtain point cloud information.

[0130] In some embodiments, the receiving and processing module 122 includes a sensing control unit connected to the sensing module, and the sensing control unit is configured to issue control timing to the sensing module.

[0131] In some embodiments, the receiving and processing module 122 includes a data processing unit configured to receive a laser signal, convert the laser signal into a feedback electrical signal, and preprocess the feedback electrical signal to obtain initial information.

[0132] In some embodiments, the receiving and processing module 122 includes a data sending unit 124 connected to the data processing unit, and the data sending unit 124 is configured to obtain initial information and transmit the initial information to the processing device 101.

[0133] Specifically, the sensing module includes a transmitting component 201, a lens component 202, and a scanning component 203.

[0134] In some embodiments, the control timing includes a first control signal, and the transmitting component 201 is configured to emit a laser signal based on the first control signal.

[0135] In some embodiments, the transmitting component 201 includes a laser emitter 303, and the laser emitter 303 includes one of an edge-emitting laser and a vertical-cavity surface-emitting laser.

[0136] Specifically, the laser emitter 303 can convert an electrical signal into a laser signal and is one of the core components of the lidar system. Its design and technical selection directly affect the performance of the lidar, such as ranging accuracy, resolution, detection range, and system cost.

[0137] In some embodiments, the laser emitter 303 can be an Edge-Emitting Laser (EEL) or a Vertical-Cavity Surface-Emitting Laser (VCSEL) to generate and emit near-infrared laser pulses (laser signals).

[0138] In some embodiments, the data processing unit includes a Time-to-Digital Converter (TDC) histogram accumulation unit 313. The TDC histogram accumulation unit 313 is configured to measure the echo electrical signal between the emission time and the reception time of the laser signal and record the measurement results in the relative intervals of the histogram.

[0139] The TDC histogram accumulation unit 313 can measure the time between the emission time and the reception time and record each measurement result in the corresponding time bin of the histogram. By counting the number of events in different time bins, the distribution of the time intervals can be obtained.

[0140] In some embodiments, the time-to-digital conversion histogram accumulation unit 313 is based on a first control signal.

[0141] In some embodiments, the transmitting component 201 further includes a driving unit 301. The driving unit 301 is connected to the laser emitter 303 and is configured to perform digital-to-analog conversion amplification on the first control signal.

[0142] Among them, the driving unit 301 is composed of a dedicated driving circuit, which performs digital-to-analog conversion and amplification on the first control signal issued by the receiving and processing module 122 to meet the voltage requirements for the laser emitter 303 to emit infrared laser pulses.

[0143] In some embodiments, the transmitting component 201 further includes a switching unit. The switching unit is connected to the laser emitter 303 and is configured to control the switching operation of the laser emitter 303.

[0144] In some embodiments, the switching unit includes a gallium nitride (GaN) switching transistor 302. One end of the GaN switching transistor 302 is connected to the driving unit 301, and the other end is connected to the laser emitter 303.

[0145] The GaN switching transistor 302 is a semiconductor device with low on-resistance and low switching losses, capable of operating at high frequencies to control the switching operation of the laser emitter 303, enabling the laser emitter 303 to generate laser pulses with high repetition rate and narrow pulses.

[0146] In some embodiments, the lens assembly 202 collimates, shapes, and diverges the laser signal emitted by the transmitting assembly 201, and converges the received laser signal.

[0147] In one example, the lens assembly 202 is configured based on an optical system.

[0148] In some embodiments, the lens assembly 202 includes a transmitting lens module 304, a rotating mirror 305, and a receiving lens module 306. Among them, the transmitting lens module 304 is configured to collimate and shape the laser signal emitted by the transmitting assembly 201; the rotating mirror 305 is configured to direct the laser signal adjusted by the transmitting lens module 304 to different directions and reflect the returned laser signal to the receiving lens module 306; the receiving lens module 306 is configured to converge the received laser signal.

[0149] The transmitting lens module 304 mainly collimates and shapes the laser emitted by the transmitting assembly 201 to make it collimated into parallel laser pulses; the rotating mirror 305 usually consists of a set of reflecting mirrors, and its function is to direct the laser pulses from the transmitting lens module 304 to different directions, thereby scanning the entire field of view, and then reflecting the returned laser signal to the receiving lens module 306; the receiving lens module 306 mainly converges the laser reflected by the target to focus it on the receiving processing module 122.

[0150] In some embodiments, the lens assembly 202 should ensure that the transmitting optical path and the receiving optical path always remain parallel.

[0151] In some embodiments, the transmitting lens module 304 and the receiving lens module 306 include one or more groups of lenses and one or more groups of reflecting mirrors for collimating or focusing the transmitted or received light beams.

[0152] In some embodiments, the transmitting lens module may further include a beam splitter, a polarizer, etc.

[0153] In some embodiments, the receiving lens module 306 should also include a narrowband filter, which may also be a narrowband filter with a grating, to filter out the reflection signals of other light beams except for the laser pulses of the preset wavelength, and enhance the contrast of the reflection signals of the laser pulses among all signals.

[0154] In some embodiments, the scanning assembly 203 is configured to perform one-dimensional scanning on the laser signal to obtain some initial information and transmit the initial information back to the receiving processing module 122. It should be noted that the initial information obtained by the scanning assembly 203 is the angular information of the laser signal.

[0155] In one example, the scanning assembly is configured based on a scanning system.

[0156] In some embodiments, the scanning assembly 203 includes a scanning motor 308 and a grating disk 309; the scanning motor 308 is connected to the grating disk 309 and a scanning mirror, and the scanning motor 308 is configured to drive the grating disk 309 and the scanning mirror to rotate.

[0157] In some embodiments, the scanning mirror can be the mirror 305 in the lens assembly 202.

[0158] In some embodiments, the scanning assembly 203 includes an optical encoder 310, and the grating disk 309 is connected to the optical encoder 310. The grating disk 309 is configured to obtain a square wave signal based on the rotation angle of the grating disk 309, and the square wave signal characterizes the reception angle information of the laser signal.

[0159] Specifically, the scanning assembly 203 uses a scanning mirror. The scanning motor 308 is the core component for realizing the scanning of the laser beam. It is physically connected to the scanning mirror and the grating disk 309 through a mechanical structure to drive the scanning mirror and the grating disk 309 to move, so that the laser beam can cover the required field of view. The grating disk 309 consists of a disk with a precision grating, and a series of evenly spaced light and dark stripes are engraved on the grating. Cooperating with the optical encoder 310, it can provide high-resolution position and angle data and is widely used as a feedback device for the scanning motor 308 in a lidar system (LiDAR).

[0160] In some embodiments, the control timing includes a second control signal. The sensing control unit is connected to the scanning motor 308 and is configured to control the rotation speed and direction of the scanning motor 308 based on the second control signal.

[0161] In some embodiments, the sensing control unit includes a transmission control unit 311 and a scanning control unit 315. Among them, the scanning control unit 315 controls the scanning motor 308 based on the second control signal, and the transmission control unit 311 controls the laser emitter 303 based on the first control signal.

[0162] In one example, if the sensing module includes the transmission assembly 201, the sensing control unit includes the transmission control unit 311, and the transmission control unit 311 is used to provide a first control signal to the transmission assembly 201. In another example, if the sensing module includes the scanning assembly 203, the sensing control unit includes the scanning control unit 315, and the scanning control unit 315 is used to provide a second control signal to the scanning assembly 203. In yet another example, if the sensing module includes the transmission assembly 201 and the scanning assembly 203, the sensing control unit includes the transmission control unit 311 and the scanning control unit 315.

[0163] In some embodiments, the data processing unit includes an angle reading unit 316, which is connected to an optoelectronic encoder 310. The angle reading unit 316 is configured to obtain initial information based on the square wave signal.

[0164] In some embodiments, the sensing control unit is further configured to adjust the second control signal based on the initial information to perform feedback control on the scanning motor 308.

[0165] Specifically, the scanning control unit 315 generates a control timing sequence (second control signal) according to the requirements preset by the system, and controls the rotation speed and direction of the scanning motor 308 based on the second control signal. During the adjustment process of the scanning motor 308, the angle reading unit 316 obtains the initial information in real time based on the optoelectronic encoder 310, and adjusts the second control signal in real time based on the initial information to achieve feedback control on the scanning motor 308.

[0166] In some embodiments, the scanning assembly 203 further includes a power amplification unit 307. The power amplification unit 307 is connected to the scanning motor 308 and the sensing control unit, and is configured to amplify and shape the second control signal.

[0167] The power amplification unit 307 can amplify and shape the second control signal generated by the scanning control unit 315 to meet the high-power requirements of the scanning motor 308.

[0168] In some embodiments, the relative position of the rotating mirror 305 and the grating code disk 309 is fixedly set.

[0169] In some embodiments, the data processing unit is further configured to obtain the rotation angle of the grating code disk 309 based on the optoelectronic encoder 310 to obtain the angle of the rotating mirror 305.

[0170] In some embodiments, the receiving and processing module 122 outputs a control timing sequence based on the frame start signal. Specifically, the lidar has a certain frame rate and horizontal field of view angle. Therefore, the laser signal can be controlled to work within a limited angle range through the frame start signal to avoid energy waste.

[0171] In some embodiments, the frame start signal is obtained based on the position of the rotating mirror 305 and the starting angle of the frame.

[0172] Based on the foregoing, the correspondence between the grating code disk 309 and the rotating mirror 305 is fixed. The position of the rotating mirror 305 can be determined by the angle of the grating code disk 309. Whether to trigger the frame start signal is judged based on whether the position of the rotating mirror 305 conforms to the starting angle of the frame. Among them, if the angle value of the rotating mirror 305 read is close to the starting angle of the frame, the frame start signal is synchronously triggered to trigger the control timing sequence to drive the sensing control unit.

[0173] Reference Figure 10 , Figure 10 is the working timing diagram of the sensing and processing system 100 corresponding to the example provided in this embodiment. Figure 9

[0174] Specifically, F_SYNC represents the synchronization signal of the frame period. The rising edge of the synchronization signal of the frame period is the frame start signal. n distance measurements (Slots) need to be completed within each frame period; S_SYNC represents the measurement synchronization signal, that is, the control timing; TRG_I represents the laser emission timing control signal, and the period (T), pulse width, and number (C) of the pulse can be set according to application requirements, that is, the first control signal; Fin_fig represents the measurement completion flag signal and is used to control the angle reading unit 316 to obtain angle data; Angle represents the square wave signal converted by the optical encoder 310 reading the grating code disk 309 and can be used for angle representation; MIPI_OUT represents the time when the data sending unit 124 transfers the initial information to the processing device 101.

[0175] Reference Figure 11 , Figure 11 is the structural schematic diagram of the rotating mirror and the grating code disk provided in this embodiment. Assuming that the disk angles angle 0, angle 1, angle 2, and angle 3 are the frame start angles, the disk rotates driven by the motor. After the angle reading unit 316 reads angle 0, angle1, angle 2, and angle 3, it triggers the synchronization signal F_SYNC of the frame period through the scan control unit 315 to start the measurement of the next frame period. F_SYNC being high represents the measurement process in the frame period, and F_SYNC being low represents the end of the frame measurement; secondly, n distance measurements (Slots) are started within the time when F_SYNC is high. S_SYNC being high represents the measurement process, and TRG_I can be generated according to application requirements. S_SYNC being low represents the end of the measurement, and TRG_I should not appear at this time; Fin_fig represents the flag signal for the completion of the Slot measurement. At this time, the angle data of each measurement (Slot) can be collected, or the angle collection time can be set according to requirements; finally, the data sending unit 124 packs and transmits the histogram information and the angle data together, and its time should be later than the measurement completion flag signal Fin_fig.

[0176] In some embodiments, the data processing unit includes a sensing array for receiving a laser signal and generating a return electrical signal based on the laser signal.

[0177] ​In some embodiments, the sensing array includes a single-photon avalanche diode (SPAD) array 318, which is configured to receive a laser signal and generate a return electrical signal based on the laser signal.

[0178] Specifically, the laser signal passes through the transmitting lens module 304 and the rotating mirror 305 and then irradiates the target surface in the detection area. The optical signal reflected by the target passes through the rotating mirror 305 and the receiving lens module 306 and then converges on the SPAD array 318. The SPAD array 318 converts the optical signal into an electrical signal to obtain the return electrical signal.

[0179] In some embodiments, the data processing unit includes a front-end analog circuit 312, which is configured to preprocess and amplify the return electrical signal.

[0180] In some embodiments, the data processing unit includes a high-precision clock sampling unit 319, which is configured to perform digital processing on the emission time and reception time of the laser signal.

[0181] In some embodiments, the data processing unit includes a finite impulse response filter 320, which is configured to perform interference filtering on the return electrical signal.

[0182] In some embodiments, the data processing unit includes a peak detection unit 314, which is configured to obtain the return time and intensity of the laser signal based on the peak of the return electrical signal.

[0183] In one example, refer to Figure 9, the emission control unit 311 enters the working state according to the frame synchronization signal of the scan control unit 315, provides the first control signal to the emission component 201 to generate laser pulses, and simultaneously synchronizes the laser emission time signal to the high-precision clock sampling unit 319 and the TDC histogram accumulation unit 313 for time measurement; the SPAD array 318 is composed of multiple single-photon avalanche diode (Single-Photon Avalanche Diode, SPAD) units, which can detect single photons, has extremely high sensitivity and time resolution, and the front-end analog circuit 312 is responsible for preprocessing and amplifying the photon signals detected by the SPAD, and amplifies the weak electrical pulse signals output by the SPAD to an amplitude that can be processed; the high-precision clock sampling unit 319 performs digital processing on the amplified analog signal, the TDC is used for high-precision time measurement, and the TDC histogram accumulation unit 313 statistically analyzes these measurement results and can provide detailed time distribution information. The finite impulse response (FIR) filter 320 is a digital filter, and its output is the finite weighted sum of the input signal, characterized by having a stable frequency response and linear phase characteristics, which can improve the signal quality, reduce noise and interference; the peak detection unit 314 is used to detect and extract the signal peak from the processed radar echo signal, and the two work together to enable the radar system to detect and identify targets more accurately.

[0184] Specifically, the receiving and processing module 122 integrates the functions of distance measurement and partial filtering internally, simplifies the amount of data output by the radar front end while retaining the original data as much as possible, avoids the loss of information dimensions, and is conducive to data standardization.

[0185] In some embodiments, the receiving and processing module 122 is set based on the SPAD SoC chip.

[0186] In this embodiment, an integrated SPAD SOC chip is used to replace most of the discrete components of the previous photodetector → signal amplification → analog-to-digital conversion → digital signal processing. The core key devices have been greatly optimized / simplified, improving the integration of the system and being conducive to the thin and light design of the product.

[0187] In some embodiments, since the SPAD has the sensitivity of single-photon detection and is very susceptible to interference, the high-precision clock sampling unit 319 and the TDC histogram accumulation unit 313 adopt the technical route of TCSPC (time-correlated technology), which can reduce the noise and interference in the system to improve the measurement accuracy and data reliability.

[0188] In addition, the overall system design was highly reconstructed with a highly integrated SPAD SoC chip as the main component, effectively utilizing the powerful computing resources of the vehicle-mounted domain controller. The system logic control function and the point cloud information processing function in the original high-computing core processor were divided and transferred, making the lidar system design more platformized and standardized. This not only reduces the cost of the lidar system but also allows for flexible customization of point cloud processing algorithms according to the application scenario, with stronger environmental adaptability. Moreover, the output of the original data can avoid information loss caused by front-end processing and can further improve the quality of data perception.

[0189] In some embodiments, the initial information includes at least one of distance information and angle information, where the distance information is obtained based on the SPAD array 318 and the angle information is obtained based on the scanning component 203.

[0190] In one example, the lens assembly 202 converges the returned laser signal to the SPAD array 318, and the receiving and processing module 122 obtains the distance information based on the control timing of the control unit and the laser signal received by the SPAD array 318.

[0191] In one example, the receiving and processing module 122 obtains the angle information of the laser signal based on the scanning component 203. If the sensing module includes the scanning component 203, the receiving and processing module 122 correspondingly includes an angle reading unit 316, and the angle reading unit 316 is connected to the optoelectronic encoder 310 to read the square wave signal recorded on the grating code disk 309 through the optoelectronic encoder 310.

[0192] In some embodiments, the initial information obtained based on the data processing unit further includes intensity information.

[0193] In some embodiments, the data sending unit 124 transmits the initial information to the processing device 101 after packing it according to a preset data format.

[0194] In some embodiments, the data sending unit 124 encodes and serializes the distance signal, intensity information, and angle information of the laser signal to obtain the initial information.

[0195] In some embodiments, the data sending unit 124 packs the distance information, intensity information, and angle information according to a certain data format and then transmits the initial information to the processing device 52.

[0196] Combined with the above description of Figure 9 the sensing and processing system 100, referring to Figure 12 , Figure 12 for this embodiment provided Figure 9 is a schematic diagram of the working process of the sensing and processing system corresponding to the example.

[0197] Step 1: Power on and start the system.

[0198] The sensing and processing system 100 is powered on and starts to enter the working state.

[0199] Step 2: The scanning control unit generates a motor drive signal according to the system preset requirements, controls the rotation speed and direction of the motor, and adjusts the drive signal according to the feedback signal at the same time.

[0200] The scanning control unit 315 integrated in the receiving and processing module 122 starts first, and generates a second control signal corresponding to the scanning motor 308 according to the system requirements, controlling the rotation speed and direction of the scanning motor 308; in this process, it is also necessary to adjust the drive signal according to the read angle feedback information to make the motor rotation smoother.

[0201] Specifically, the scanning control unit 315 generates a control timing sequence (second control signal) according to the system preset requirements, and controls the rotation speed and direction of the scanning motor 308 based on the second control signal; during the adjustment process of the scanning motor 308, the angle reading unit 316 obtains the initial information in real time based on the optical encoder 310, and adjusts the second control signal in real time based on the initial information to achieve the feedback control of the scanning motor 308.

[0202] Step 3: The power amplification unit amplifies and shapes the drive signal to meet the high-power requirements for driving the motor.

[0203] Specifically, the power amplification unit 307 amplifies and shapes the drive signal generated by the scanning control unit 315 to meet the high-power requirements for driving the motor.

[0204] Step 4: The scanning motor drives the rotating mirror and the grating code disk to rotate.

[0205] Specifically, the scanning motor 308 is physically connected to the rotating mirror 305 and the grating code disk 309, and can drive the rotating mirror 305 and the grating code disk 309 to rotate, thereby scanning the targets in the detection area; at the same time, the positional relationship between the rotating mirror 305 and the grating code disk 309 is determined, that is, the angle range of the grating code disk 309 corresponding to each mirror surface can be determined.

[0206] Step 5: The optical encoder converts the rotation angle on the grating code disk into a continuous square wave signal through photoelectric conversion.

[0207] Specifically, the optical encoder 310 converts the rotation angle on the grating code disk 309 into a continuous square wave signal through photoelectric conversion.

[0208] Step 6: The angle reading unit reads the angle, and at the same time, the rotation speed of the motor can be calculated according to the read angle, and then the motor is feedback-controlled.

[0209] Specifically, the angle reading unit 316 reads the angle of the rotary mirror 305 according to the square wave signal, and then improves the resolution of the optical encoder 310 through interpolation subdivision technology; at the same time, the rotation speed of the motor can be calculated based on the read angle, and then the scanning motor 308 is feedback controlled.

[0210] Step 7: Determine the position of the rotary mirror through the code disk angle.

[0211] Specifically, the lidar has a certain frame rate and horizontal field of view angle. Therefore, the laser pulse can be controlled to work within a limited angle range through the frame start signal to avoid energy waste. Specifically in this embodiment, the corresponding relationship between the rotary mirror 305 and the grating code disk 309 is fixed, and the position of the rotary mirror 305 (the start angle of the frame) can be determined through the angle of the grating code disk 309. If the read angle value of the rotary mirror 305 is close to the start angle of the frame, a first control signal is sent to the emission control unit 311.

[0212] Step 8: The emission timing control unit generates a first control signal according to the frame start signal and synchronously sends a time signal for TDC measurement.

[0213] The emission control unit 311 generates a first control signal that meets the system requirements for frequency, pulse width, and number according to the frame start signal, and synchronously sends a time signal for TDC measurement.

[0214] Step 9: The emission module generates laser pulses that meet the preset parameter conditions according to the first control signal.

[0215] Specifically, the driving unit 301, gallium nitride switching tube 302, and laser emitter 303 of the emission component 201 cooperate with each other to generate a laser signal that meets the preset parameter conditions according to the first control signal, and convert the electrical signal into an optical signal.

[0216] Step 10: Obtain the emission signal of the laser pulse, convert the optical signal into an electrical signal, and the SPAD preprocesses and amplifies the detected photon signal.

[0217] Specifically, the laser pulse irradiates the target surface in the detection area after passing through the emission lens module 304 and the rotary mirror 305; the optical signal reflected by the target is focused on the SPAD array 318 after passing through the rotary mirror 305 and the receiving lens module 306, converting the optical signal into an electrical signal, and the front-end analog circuit 312 preprocesses and amplifies the photon signal detected by the SPAD array 318.

[0218] Step 11: High-precision clock sampling processing + TDC processing.

[0219] Specifically, the high-precision clock sampling unit 319 can ensure the digital processing of the emission and reception times of laser pulses. The TDC histogram accumulation unit 313 can measure the time between the emission time and the reception time and record each measurement result in the corresponding time interval (bin) of the histogram. By statistically analyzing the number of events in different time intervals, the distribution of the time intervals can be obtained.

[0220] Step 12: FIR filtering process + peak detection process.

[0221] Specifically, the FIR filter 320 can effectively remove the interference of ambient light noise and electronic noise through weighted summation, improving the signal-to-noise ratio of the signal. The peak detection unit 314 can quickly and accurately find the peak of the received signal, thereby determining the return time (i.e., distance) and intensity of the laser pulse.

[0222] Step 13: The data sending unit sends the initial information to the processing device.

[0223] Specifically, the data sending unit 124 encodes and serializes the distance information, intensity information, and angle information, and sends them to the processing device 101 through the bus to improve the efficiency and reliability of data transmission.

[0224] Step 14: The processing device is responsible for the point cloud post-processing under complex conditions.

[0225] Specifically, the processing device 101 is responsible for the point cloud post-processing under complex scenarios to obtain the point cloud information based on the initial information.

[0226] Combined with the above description of Figure 10 refer to Figure 13 , Figure 13 This is Figure 10 a schematic diagram of the working timing of the active sensing component and the passive sensing component corresponding to the example provided by this application. Among them, Cam_TRG represents the flag signal indicating the completion of passive sensor data acquisition.

[0227] In some embodiments, the passive sensing component 520 obtains the corresponding initial information based on the moment when the active sensing component 510 obtains the corresponding information.

[0228] In Figure 13 the example, Cam_TGR represents the time when the passive sensing component 520 obtains the image information. At this time, the pulses of Cam_TGR and Fin_fig are consistent, that is, the moment when the active sensing component 510 obtains the first initial information is the same as the moment when the passive sensing component 520 obtains the second initial information.

[0229] In some embodiments, the frame rate of the active sensing component 510 is the same as that of the passive sensing component 520. In one example, the frame rates of the active sensing component 510 and the passive sensing component 520 are 10 Hz.

[0230] In some embodiments, obtain the time when the corresponding initial information is acquired and the preset magnification, and acquire the corresponding initial information.

[0231] In one example, assume that the preset magnification is 2. At this time, the pulse interval of Cam_TGR is twice the pulse interval of Fin_fig, that is, the active sensing component 510 acquires the first initial information twice, and the passive sensing component 520 acquires the second initial information once. If the frame rate of the active sensing component 510 is 10 Hz, the frame rate of the passive sensing component 520 is 20 Hz at this time.

[0232] In one example, assume that the preset magnification is 3. At this time, the pulse interval of Cam_TGR is three times the pulse interval of Fin_fig, that is, the active sensing component 510 acquires the first initial information three times, and the passive sensing component 520 acquires the second initial information once. If the frame rate of the active sensing component 510 is 10 Hz, the frame rate of the passive sensing component 520 is 30 Hz at this time.

[0233] It should be noted that for the setting of the preset magnification, as long as the frame rate of the passive sensing component 520 does not exceed the allowable frame rate range of the device, no numerical limitation is made in this embodiment.

[0234] Combined with the above description of Figure 12 reference Figure 14 Figure 14 is the schematic diagram of the working process of the sensing processing system provided by the embodiment of the present application Figure 12 corresponding to the passive sensing component.

[0235] Figure 14 Compared with Figure 12 , the difference is that steps 1.5, 15 and 17 are added, additional actions are added in step 8, step 13 is replaced by step 16, and step 14 is replaced by step 17.

[0236] Among them, step 1.5 is located between step 1 and step 2, and step 1.5 includes: time synchronization of the active sensing component and the passive sensing component.

[0237] Specifically, the clock module 504 of the processing device 52 calibrates the clocks of all sensing components 501 (including the active sensing component 510 and the passive sensing component 520), and synchronizes the clock calibration information to the data sending unit 124, so as to realize the time synchronization of the active sensing component 510 and the passive sensing component 520.​

[0238] New action in step 8: Provide a synchronization trigger signal at a specific time in the emission timing according to requirements.

[0239] Specifically, the active sensing component 510 provides a synchronization trigger signal based on the emission time of the detection signal and a preset delay. To achieve immediate triggering of the camera for shooting at a specific moment when the lidar completes a frame of data scanning or detects a specific target, this can ensure that the image captured by the camera is as close as possible to the detection time of the lidar for that target, thereby greatly improving the alignment accuracy in time between the active sensing component 510 and the passive sensing component 520, and providing more accurate spatio-temporal information for subsequent target recognition and analysis.

[0240] In some embodiments, the synchronization trigger signal is provided once in each cycle of the detection signal.

[0241] In some embodiments, the synchronization trigger signal is provided once in multiple cycles of the detection signal.

[0242] After step 8 is completed, step 9 and step 15 are executed simultaneously. Among them, in step 15, the camera sensing front end performs image acquisition based on the synchronization trigger signal.

[0243] Specifically, step 9 is the execution step for the active sensing component 510 to obtain the first initial information, and step 15 is the execution step for the passive sensing component 520 to obtain the second initial information. That is, the active sensing component 510 and the passive sensing component 520 after clock synchronization perform initial information acquisition simultaneously.

[0244] In step 16, the data obtained by each sensor is serialized by a serializer and then sent to the processing device.

[0245] Specifically, the serializer serially processes the first initial information obtained by the active sensing component 510 and the second initial information obtained by the passive sensing component 520 to generate serial information, and transfers the serial information to the deserialzer 540 through a physical interface to improve the efficiency and reliability of data transmission.

[0246] In step 17, the deserialzer performs a deserialization operation according to a pending protocol and data format to achieve the conversion from high-speed serial data to parallel data.

[0247] Specifically, the deserialzer 540 performs deserialization processing on the serial information to obtain the initial information generated by the sensing module 51 to improve the efficiency and reliability of data transmission.

[0248] In step 18, the processing device performs data fusion on the initial information obtained by the active sensing component and the passive sensing component.

[0249] For the integrated sensing system provided in this embodiment, the point cloud processing function of the sensing component 501 is transferred to the processing device 52, so that the sensing component 501 only needs to complete signal acquisition, and the data processing process is executed by the processing device 52 at the back end. The product standardization degree is further improved, the scene algorithm can be flexibly customized according to application requirements, and the environmental adaptability is stronger. In addition, integrating different types of sensors at the physical space level can effectively improve the integration degree of the entire system, avoid the impact on the vehicle styling design caused by installing each sensor at different positions on the vehicle body, and is easy to install.

[0250] In addition, transferring the information processing part to the processing device 52 is more convenient for online upgrading of the processing algorithm. The product standardization degree is further improved, the scene algorithm can be flexibly customized according to application requirements, and the environmental adaptability is stronger. At the same time, the sensing component 501 is freed from the dependence on high-complexity proprietary chips, reducing the cost of the system. When the product is damaged, only the sensing component 501 without a core processor needs to be replaced.

[0251] It should be noted that, without conflict, the features claimed in the integrated sensing system 50 provided in the above embodiments can be randomly combined to obtain new embodiments of the integrated sensing system 50.

[0252] The second aspect of this application also provides a module structure, which is applied to the sensing module provided in the above embodiment, enabling different types of sensors to be connected through physical structure design, which can simplify the process of joint external parameter calibration and improve the accuracy of data space alignment of different types of sensors; at the same time, the physical distance between different sensors is shortened, and the passive sensor is controlled to sample by providing a trigger synchronization trigger signal through the active sensor, and the synchronization scheme is more flexible, improving the time alignment accuracy of the sensors.

[0253] Reference Figure 15 and Figure 16 , Figure 15 is the external structure schematic diagram of the module structure provided in this embodiment, Figure 16 is the internal structure schematic diagram of the module structure provided in this embodiment. The module structure 600 provided in this embodiment will be described in detail below with reference to the drawings. The module structure 600 includes a carrier bottom plate 601, at least two sensing components 501, and a processing module 602. Among them, the sensing component 501 and the processing module 602 are arranged on the carrier bottom plate 601, and the processing module 602 is used to control the sensing component 501 to detect and obtain the initial information of the detection object and transmit the initial information to the processing device 52.

[0254] In some embodiments, the sensing component 501 includes an active sensing component 510 and a passive sensing component 520.

[0255] The passive sensing component 520 mainly refers to a sensor that does not generate energy by itself and makes corresponding actions or outputs corresponding physical quantity changes by obtaining signals generated or reflected by the detected object. It can include a monocular camera, a binocular camera, a trinocular camera, a telephoto camera, a wide-angle camera, or a far-infrared camera, etc. Additionally, the passive sensing component 520 mentioned in this embodiment is a sensing front end, that is, the passive sensing component 520 does not include an ISP (Image Signal Processing) processing module.

[0256] The active sensing component 510 refers to a sensor that emits energy to the object to be measured and measures the change in the energy reflected back or projected out. It can include a lidar, a millimeter-wave radar, or an ultrasonic radar, etc. Additionally, the active sensing component 510 mentioned in this embodiment is a sensing front end, that is, the active sensing component 510 does not include a part of a large computing power dedicated processing chip.

[0257] The sensing component 501 includes an active sensing component 510 and a passive sensing component 520 to achieve the acquisition of different types of initial information and improve the sensing accuracy of the integrated sensing system 50.

[0258] In some embodiments, the module structure 600 further includes: a receiving cavity 603, which is arranged on the carrier base plate 601 and forms a closed space with the carrier base plate 601. The closed space is at least used to accommodate the sensing component 501.

[0259] In some embodiments, the surface of the receiving cavity 603 has a window 604, and the window 604 is used for the sensing component 501 to obtain initial information.

[0260] In some embodiments, heat dissipation lines 605 are further arranged on the surface of the receiving cavity 603.

[0261] In an example, the module structure 600 includes a receiving cavity 603, a carrier base plate 601, different types of passive sensing components 520, and different types of active sensing components 510.

[0262] Among them, the passive sensing component 520 includes binocular cameras 701 and 702, a telephoto camera 703, and a far-infrared camera 704, and a coaxial cable 705. The active sensing component 510 includes a lidar sensing front end, including a transmitting component 201, a lens component 202, a scanning component 203, and a receiving and processing module 122.

[0263] The binocular cameras 701 and 702 refer to two visual sensors that can independently acquire images. They usually have a high pixel resolution and can calculate the depth information of the scene through the principle of parallax. However, a certain distance (baseline) needs to be maintained between the optoelectronic fronts of the two cameras. Therefore, in order to save space and ensure the detection effect, the above-mentioned binocular cameras 701 and 702 can be symmetrically distributed on both sides of the accommodation cavity 603, and one or more other types of sensors can be configured in the middle.

[0264] The long - focal - length camera 703 has a relatively large focal length and can provide a wider field of view than ordinary cameras. It is mainly used to capture images and videos at a long distance, such as identifying obstacles in the distance, traffic signs, or detecting the driving conditions of the vehicle in front. It can be distributed between the binocular cameras 701 and 702 and is symmetrically distributed with the far - infrared camera 704 on both sides of the active sensing component 510.

[0265] The far - infrared camera 704 is a camera system that uses infrared technology. It can generate images by detecting the thermal radiation (instead of visible light) emitted by objects (such as pedestrians or animals). It is mainly used to provide a clear field of view in low - light or completely dark environments. This camera is especially effective at night, in bad weather, or other conditions with limited visibility.

[0266] The above cameras (binocular cameras 701 and 702, long - focal - length camera 703, and far - infrared camera 704) can all be set to the external trigger mode. They receive external trigger signals through the coaxial cable 705 and transmit image signals to the communication adapter board 609. Among them, the external trigger signal can be provided by the processing device 52 corresponding to the module structure 600, or can be provided by the active sensing component 510.

[0267] The active sensing component 510 obtains some information about the detection area by emitting a laser beam and measuring the time it takes to reflect back. There are certain differences from the existing lidar systems. It does not contain the part of the large - computing - power dedicated processing chip inside, only retains some simple system logic control and ranging functions, and has no special algorithms for complex scenarios (such as dirt detection, high - reflection expansion suppression, ghost filtering, etc.) processing. It only outputs the original point cloud data for backend data fusion and is distributed in the middle of the accommodation cavity 603.

[0268] The combination of the accommodation cavity 603 and the bearing base plate 601 can form a closed space, which can wrap the active sensing component 510 and the passive sensing component 520 to prevent the sensing component 501 from being affected by the outside world. The front surface of the accommodation cavity 603 has a window 604, and the window 604 allows the active sensing component 510 to actively emit signals and receive external signals. In addition, the upper surface of the accommodation cavity 603 has heat - dissipation lines 605, and the heat - dissipation lines 605 are used for the heat dissipation of the accommodation cavity 603.

[0269] In some embodiments, the module structure 600 further includes: a positioning component 606, disposed at corresponding positions of the accommodation cavity 603 and the carrier bottom plate 601, for fixing the accommodation cavity 603 and the carrier bottom plate 601.

[0270] In some embodiments, the positioning component 606 is further used to fix the carrier bottom plate 601 to the application carrier of the module structure 600.

[0271] Specifically, the positioning component 606 is provided at the corner positions of the accommodation cavity 603 for fixing to the carrier bottom plate 601; in one example, the positioning component 606 is further used for connecting and positioning the carrier bottom plate 601 and the intelligent driving vehicle to which the module structure 600 is applied.

[0272] In some embodiments, the carrier bottom plate 601 is mainly used for the installation and positioning of different types of sensors, and the positioning holes 607 provided on its surface correspond one by one to the positioning components 606 on the accommodation cavity 603.

[0273] In some embodiments, the binocular cameras 701 and 702, the long - focal - length camera 703, and the far - infrared camera 704 are all composed of different optical lenses and image sensors. According to the different functions of the cameras, there are also certain differences in the optical lenses and image sensors. Among them, the design of the lens parameters directly affects the transmission and control of light, including view - angle control, chromatic aberration, glare, and reflection, and it is necessary to improve the clarity and contrast of the image as much as possible. The image sensor is mainly responsible for converting the optical signal into an electrical signal, and then packing it in pixel format and sending it to the processing device 52 for point - cloud processing.

[0274] In some embodiments, the transmitting component 201 is mainly responsible for emitting laser pulses, usually including a laser and its driving circuit, for generating a high - repetition - rate narrow - pulse laser signal, and the timing control signal can be provided by the receiving and processing module 122; the laser signal generated by the transmitting component 201 is usually relatively divergent and needs to be collimated by the lens assembly 202 to form a parallel light beam to ensure that the laser beam can maintain a high energy density and straight - line propagation, so as to improve the measurement accuracy and effective working distance; the scanning component 203 is used to control the scanning path and angle of the laser beam, so that it covers a predetermined spatial range, and records the angle information at the moment of laser emission in real time. Subsequently, the laser echo signal reflected by the target is usually relatively scattered and has weak energy, and needs to be focused by the lens assembly 202 again onto the focal plane of the photodetector of the receiving module, so as to make full use of the photosensitive area of the receiver and improve the detection performance of the overall system.

[0275] In some embodiments, the receiving and processing module 122 is mainly based on a highly integrated SPAD SoC chip, which integrates components such as a transmission timing control unit, a SPAD array, a front-end analog circuit, a high-precision clock sampling unit, a TDC histogram accumulation unit, a FIR filtering unit, a peak detection unit, a scanning control unit, an angle reading unit, and a data transmission unit. However, the receiving and processing module 122 does not include special algorithm processing in complex scenarios, and retains the original lidar data as much as possible to avoid the loss of information dimensions.

[0276] In some embodiments, the lidar may also be a pure solid-state lidar. In this case, the active sensing component 510 may not include the scanning component 203, and the SPAD SoC receiving module does not include the scanning control unit and the angle reading unit.

[0277] The module structure provided in this embodiment enables different types of sensors to be connected through physical structure design, which can simplify the process of joint extrinsic calibration and improve the accuracy of data spatial alignment of different types of sensors. At the same time, the physical distance between different sensors is shortened, and the active sensor provides a trigger synchronization signal to control the passive sensor to sample, making the synchronization scheme more flexible and improving the time alignment accuracy of the sensors.

[0278] Another embodiment of the present application also provides a method for generating an intelligent driving strategy, which is applied to the integrated sensing system 50 provided in the above embodiment, and includes: obtaining initial information obtained by at least two sensing components 501 monitoring a detection object; performing information fusion processing on the initial information to obtain point cloud information of the detection object.

[0279] In some embodiments, the method for generating an intelligent driving strategy further includes: obtaining an intelligent driving strategy based on the point cloud information corresponding to multiple target objects.

[0280] Specifically, after the processing device 52 generates point cloud information based on the initial information, it is further configured to generate an intelligent driving strategy according to the point cloud information.

[0281] In one example, the processing device 52 is implemented based on a domain control SoC. At this time, the domain control SoC also includes a point cloud processing function to calculate and obtain point cloud information based on the initial information uploaded by the sensing device.

[0282] In some embodiments, performing information processing on the initial information to obtain point cloud information of the detection object includes: obtaining point cloud information based on a preset algorithm.

[0283] In some embodiments, the preset algorithm includes at least one of a high-reflectivity expansion suppression, ghost filtering, dirt detection, and rain / fog / dust filtering algorithm.

[0284] Another embodiment of the present application further provides a computer-readable storage medium storing instructions that are adapted to be loaded by a processor to execute the method for generating an intelligent driving strategy provided in the above embodiment.

[0285] Another embodiment of the present application further provides a computer program product, including a computer program / instructions, which when executed by a processor implement the method for generating an intelligent driving strategy provided in the above embodiment.

[0286] Another embodiment of the present application further provides a controller, which includes a processor and a memory. The processor is connected to the memory. The memory is used to store a computer program, and the processor is used to execute the computer program to implement the integrated sensing system 50 provided in the above embodiment.

[0287] Another embodiment of the present application further provides a vehicle, which includes the integrated sensing system 50 provided in the above embodiment, or includes the module structure 600 provided in the above embodiment, or includes the computer-readable storage medium provided in the above embodiment, or includes the computer program product provided in the above embodiment, or includes the controller provided in the above embodiment.

[0288] Among them, the independent claim of the vehicle has all the beneficial effects of the above-mentioned minimum protection subject matter, which will not be elaborated herein. The vehicle may be a fuel vehicle, a plug-in hybrid vehicle, a new energy vehicle, etc., and the present application does not make specific limitations thereto.

[0289] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the detailed descriptions of other embodiments above, which will not be elaborated herein.

[0290] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only an example and does not constitute a limitation to the present application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to the present application. Such modifications, improvements, and corrections are proposed in the present application, so such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of the present application.

[0291] The above has introduced in detail an integrated sensing system, a module structure, a generation method, and related devices provided by the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. An integrated sensing system (50), characterized in that, Comprising: A sensing module (51), including at least two sensing components (501), where the sensing components (501) are used to monitor and obtain initial information of a detection object; A processing device (52), configured to receive the initial information and obtain point cloud information of the detection object based on the initial information corresponding to the at least two sensing components (501).

2. The integrated sensing system (50) according to claim 1, wherein, The processing device (52) is further configured to provide a clock synchronization signal to the sensing module (51), and the at least two sensing components (501) in the sensing module (51) perform clock calibration in response to the clock synchronization signal.

3. The integrated sensing system (50) according to claim 2, characterized in that, The processing device (52) includes: a clock module (504), connected to the sensing module (51), and the clock module (504) is configured to provide the clock synchronization signal.

4. The integrated sensing system (50) according to claim 1, wherein, The sensing module (51) includes at least one passive sensing component (520) and at least one active sensing component (510).

5. The integrated sensing system (50) according to claim 4, wherein, The active sensing component (510) is further configured to provide a synchronization trigger signal, and the passive sensing component (520) monitors and obtains the initial information in response to the synchronization trigger signal.

6. The integrated sensing system according to claim 5, wherein The active sensing component (510) provides the synchronization trigger signal based on the emission time of a detection signal and a preset delay.

7. The integrated sensing system according to claim 6, wherein, The synchronization trigger signal is provided once in each cycle of the detection signal.

8. The integrated sensing system according to claim 6, wherein The synchronization trigger signal is provided once in multiple cycles of the detection signal.

9. The integrated sensing system according to claim 7 or 8, characterized in that, In the cycle of the detection signal in which the synchronization trigger signal is provided, the delay time of the preset delay is the same.

10. The integrated sensing system (50) according to claim 4, wherein The initial information provided by the active sensing component (510) includes timestamp information.

11. The integrated sensing system (50) according to claim 1, wherein, The sensing module (51) further includes a serializer (530), and the processing device (52) further includes a deserializer (540). The serializer (530) is connected to the at least two sensing components (501), and the deserializer (540) is connected to the serializer (530).

12. The integrated sensing system (50) according to claim 11, characterized in that, The sensing module (51) includes multiple serializers (530), and the at least two sensing components (501) form multiple serial sensing links based on the multiple serializers (530).

13. The integrated sensing system (50) according to claim 11, characterized in that, The initial information provided by the at least two sensing components (501) is serially processed based on the serializer (530) to generate serial information, and the serial information is sent to the processing device (52).

14. The integrated sensing system (50) according to claim 13, wherein The processing device (52) performs deserialization processing on the serial information based on the deserializer (540) to obtain the initial information.

15. The integrated sensing system (50) according to claim 4, characterized in that, The processing device (52) includes: a data fusion module (503), configured to perform information fusion on the initial information obtained by the at least two sensing components (501) to obtain the point cloud information.

16. The integrated sensing system (50) according to claim 15, characterized in that, The data fusion module (503) outputs the point cloud information of the detection object based on the spatial cubic model matrix.

17. The integrated sensing system (50) according to claim 16, wherein, The data fusion module (503) includes: A first processing unit, configured to establish the spatial cubic model matrix and fill the initial information corresponding to each sensing component into the spatial cubic model matrix; A second processing unit, configured to output point cloud information of the detection object based on the spatial cubic model matrix.

18. The integrated sensing system (50) according to claim 17, wherein The data fusion module (503) further includes: A third processing unit, configured to convert the initial information corresponding to each sensing component (501) into the same coordinate system based on preset information; The first processing unit establishes the spatial cubic model matrix based on the coordinate system.

19. The integrated sensing system (50) according to claim 18, wherein, The preset information includes: internal parameters of the sensing component (501), and the corresponding relationship of the sensing component (501) in physical space.

20. The integrated sensing system (50) according to claim 18, wherein, The first processing unit establishes the spatial cubic model matrix based on the maximum field of view angle of the passive sensing component (520) and / or the spatial resolution of the active sensing component (510).

21. A module structure (600) is applied to form the sensing module (51) according to any one of claims 1 to 20, characterized in that, Comprising: A carrier base plate (601), at least two sensing components (501), and a processing module (602); The sensing component (501) and the processing module (602) are disposed on the carrier base plate (601); The processing module (602) is configured to control the sensing component (501) to detect and obtain initial information of the detection object, and transmit the initial information to the processing device (52).

22. The module structure (600) according to claim 21, characterized in that, The module structure (600) further includes: a receiving cavity (603), disposed on the carrier base plate (601) and forming a closed space with the carrier base plate (601), and the closed space is at least used to accommodate the sensing component (501).

23. The module structure (600) according to claim 22, wherein, The surface of the receiving cavity (603) has a window (604), and the window (604) is used for the sensing component (501) to obtain the initial information.

24. The module structure (600) according to claim 22 or 23, characterized in that, The surface of the receiving cavity (603) is further provided with heat dissipation patterns (605).

25. The module structure (600) according to claim 22, wherein, The module structure (600) further includes: a positioning component (606), disposed at corresponding positions of the receiving cavity (603) and the carrier base plate (601), for fixing the receiving cavity (603) and the carrier base plate (601).

26. The module structure (600) according to claim 25, wherein, The positioning component (606) is further used to fix the carrier base plate (601) on the application carrier of the module structure (600).

27. The module structure (600) according to claim 21, characterized in that, The sensing component (501) includes an active sensing component (510) and a passive sensing component (520).

28. A method for generating an intelligent driving strategy, applied to the integrated sensing system (50) according to any one of claims 1 to 20, characterized in that, Comprising: Obtain initial information obtained by at least two sensing components (501) monitoring a detection object; Perform information fusion processing on the initial information to obtain point cloud information of the detection object.

29. The method for generating an intelligent driving strategy according to claim 28, wherein Further comprising: Obtain an intelligent driving strategy based on the point cloud information corresponding to multiple target objects.

30. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions, and the instructions are suitable for being loaded by a processor to execute the method for generating an intelligent driving strategy according to any one of claims 28 or 29.

31. A computer program product comprising computer programs / instructions, characterized in that, When the computer program / instructions are executed by the processor, the method for generating an intelligent driving strategy according to any one of claims 28 or 29 is implemented.

32. A controller, characterized in that, Comprising a processor and a memory, the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program to implement the integrated sensing system (50) according to any one of claims 1 to 20.

33. A vehicle, characterized in that, Comprising the integrated sensing system (50) according to any one of claims 1 to 20, or comprising the module structure (600) according to any one of claims 21 to 27, or comprising the computer-readable storage medium according to claim 30, or comprising the computer program product according to claim 31, or comprising the controller according to claim 32.

Images