Circuit and method for storing data, chip and electronic device

Abstract

The present application relates to a circuit and method, chip and electronic device for storing data, wherein the circuit for storing data includes: a cache module, configured to cache point cloud data and audio data, wherein the point cloud data includes multiple point cloud frames, and the audio data includes multiple audio frames, and there is a corresponding relationship between the multiple point cloud frames and the multiple audio frames; a recognition result acquisition module, coupled with the cache module, configured to obtain recognition results for the multiple audio frames, and obtain recognition results for the target object; an assembly module, coupled with the recognition result acquisition module, configured to respond to the recognition result representation that there is a target object, and determine the association relationship between the target object and the multiple point cloud frames based on the corresponding relationship; an output module, coupled with the recognition result acquisition module and the assembly module, respectively, configured to output multiple point cloud frames, recognition results or multiple audio frames corresponding to the recognition results based on the association relationship. The present application can realize the targeted storage of effective data and improve the convenience of data retrieval.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a circuit and method for storing data, a chip and an electronic device. Background Art

[0002] With the rapid development of computer technology, artificial intelligence technology, etc., assisted driving (or autonomous driving) technology has begun to enter people's daily lives. Radar can provide environmental perception information in the form of point cloud data for determining driving strategies.

[0003] Point cloud data refers to a set of vectors in a three-dimensional coordinate system, which may contain color information (RGB) or reflection intensity information (Intensity), etc. Radar can generate tens of thousands of points per second, and the number of data points obtained in each scan can reach hundreds of thousands, millions or even more, with a data volume of tens of gigabytes (Gb) to thousands of Gb. Such a huge amount of data brings challenges to data storage and data retrieval. Summary of the invention

[0004] In order to solve or partially solve the problems existing in the related art, the present application provides a circuit and method for storing data, a chip and an electronic device, which can store valid data and improve the convenience of data retrieval.

[0005] In a first aspect, the present application provides a circuit for storing data, including: at least one of a cache module, a recognition result acquisition module, an assembly module, and an output module. The cache module is configured to cache point cloud data and audio data, the point cloud data includes multiple point cloud frames, the audio data includes multiple audio frames, and there is a corresponding relationship between the multiple point cloud frames and the multiple audio frames; the recognition result acquisition module is coupled to the cache module, and is configured to obtain recognition results for the multiple audio frames, and obtain recognition results for the target object; the assembly module is coupled to the recognition result acquisition module, and is configured to determine the association relationship between the target object and the multiple point cloud frames based on the corresponding relationship in response to the recognition result characterizing the existence of the target object; the output module is coupled to the recognition result acquisition module and the assembly module, respectively, and is configured to output multiple point cloud frames corresponding to the recognition result and at least one of the following based on the association relationship: the recognition result or the multiple audio frames corresponding to the recognition result, so as to write into the non-volatile memory.

[0006] The second aspect of the present application provides a chip, comprising a distance calculation unit and the above-mentioned circuit for storing data. The distance calculation unit is configured to calculate the distance between the object and the mobile platform based on the time difference between the emission time point of the laser and the reception time point of the reflected laser, and generate point cloud data; the circuit for storing data is configured to output the point cloud frame in which the target object exists in the point cloud data in association with at least one of the following: the recognition result of the target object or the audio frame in which the target object exists.

[0007] The third aspect of the present application provides a board, comprising the above-mentioned circuit for storing data or the above-mentioned chip, and further comprising a non-volatile memory.

[0008] A fourth aspect of the present application provides a radar, comprising at least one of the circuits, chips or boards for storing data as described above.

[0009] A fifth aspect of the present application provides a mobile platform, comprising the radar as described above.

[0010] A sixth aspect of the present application provides a method for storing data, including: caching point cloud data and audio data, the point cloud data including multiple point cloud frames, the audio data including multiple audio frames, and there being a corresponding relationship between the multiple point cloud frames and the multiple audio frames; obtaining recognition results for the multiple audio frames, and obtaining recognition results for the target object; in response to the recognition result characterizing the existence of the target object, determining an association relationship between the target object and the multiple point cloud frames based on the corresponding relationship; and outputting the multiple point cloud frames corresponding to the recognition result and at least one of the following: the recognition result or the multiple audio frames corresponding to the recognition result in an associative manner based on the association relationship, so as to write them into a non-volatile memory.

[0011] A seventh aspect of the present application provides an electronic device, comprising: a processor; and a memory, on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method as described above.

[0012] An eighth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor executes the above method.

[0013] A ninth aspect of the present application provides a computer program product, comprising an executable code, which implements the above method when the executable code is executed.

[0014] The technical solution provided by this application may have the following beneficial effects:

[0015] In certain embodiments of the present application, after the cache module caches multiple point cloud frames, it does not directly write the multiple point cloud frames into a non-volatile memory, but obtains the recognition result of the target object in the audio frame collected synchronously with the point cloud frame through the recognition result acquisition module, and the corresponding relationship between the point cloud frame, the audio frame and the target object is determined by the assembly module. In this way, the output module can output multiple point cloud frames corresponding to the recognition result and at least one of the following: the recognition result or the multiple audio frames corresponding to the recognition result in association with the corresponding relationship, so as to write useful (target object) point cloud frames, audio frames and corresponding recognition results into the non-volatile memory. The embodiments of the present application effectively reduce the waste of storage space caused by storing useless data, and facilitate the retrieval of the target point cloud frame and / or target audio frame based on the recognition result and the association relationship, thereby improving the retrieval efficiency and accuracy.

[0016] In certain embodiments of the present application, after obtaining the recognition result, the recognition result acquisition module determines the position of the target point cloud frame in the cached multi-frame point cloud frame based on the timestamp and correspondence of the target audio frame, which helps to improve the accuracy and efficiency of the retrieval results.

[0017] In some embodiments of the present application, the recognition model of the recognition result acquisition module outputs recognition results for each of the multiple preset target objects, and can retain the audio frames and point cloud frames corresponding to each target object. When the audio frame includes multiple target objects, the recognition results of the multiple target objects included in the audio frame will not be lost, thereby improving the integrity of the stored valid data.

[0018] In some embodiments of the present application, on the basis of ensuring the integrity of valid data, deduplication is performed on point cloud frames and / or audio frames that may be stored repeatedly. The deduplication operation avoids storing repeated point cloud frames and / or audio frames, further improving the effective utilization of storage space.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0021] Figure 1 It is a schematic diagram of an application scenario of a circuit and method for storing data, a chip, and an electronic device shown in an embodiment of the present application;

[0022] Figure 2It is a schematic diagram of another application scenario of a circuit and method for storing data, a chip, and an electronic device shown in an embodiment of the present application;

[0023] Figure 3 is a block diagram of a circuit for storing data according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of an application scenario shown in an embodiment of the present application;

[0025] Figure 5 is a block diagram of a recognition result obtaining module shown in an embodiment of the present application;

[0026] Figure 6 is a block diagram of a target audio frame timestamp determination unit shown in an embodiment of the present application;

[0027] Figure 7 is a block diagram of a frame number statistics circuit shown in an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of a recognition result for an audio frame according to an embodiment of the present application;

[0029] Fig. 9 is a schematic diagram of a target audio frame shown in an embodiment of the present application. ;

[0030] Fig.10 is a block diagram of a target object appearance frame timestamp determination unit shown in an embodiment of the present application;

[0031] Fig.11 is another block diagram of a circuit for storing data shown in an embodiment of the present application;

[0032] Fig.12 is a block diagram of a chip shown in an embodiment of the present application;

[0033] Fig.13 is a block diagram of a chip shown in another embodiment of the present application;

[0034] Fig.14 is a schematic diagram of the structure of a radar shown in an embodiment of the present application;

[0035] Fig.15 is a flow chart of a method for storing data shown in an embodiment of the present application;

[0036] Fig.16 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0038] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "an" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0040] In order to facilitate the understanding of the present application, some concepts involved in the present application are first explained.

[0041] Point cloud: Point cloud is a data set of points in a specific coordinate system. Point cloud data includes a variety of information, such as three-dimensional coordinates (X, Y, Z), color, classification value, intensity value, time, etc. Point cloud data can be used in autonomous driving, assisted driving, mapping, machine vision and other fields. Point cloud data can be generated by Light Detection and Ranging (LIDAR).

[0042] Vehicle-mounted laser radar: By emitting outgoing light (such as a laser beam) with a wavelength of about 900nm, the outgoing light will be reflected by the obstacle after encountering it. The processing unit calculates the distance between the obstacle and the vehicle-mounted laser radar based on the time difference between the reflected light and the outgoing light. In addition, the processing unit can also estimate the reflectivity of the target based on the cross-section of the reflected light signal obtained after receiving the reflected light, so as to generate point cloud data. Vehicle-mounted laser radar has the advantages of small size and high degree of integration.

[0043] In the fields of smart transportation / autonomous driving, the key point is to quickly and accurately perceive the surrounding environment of roads / unmanned vehicles. According to the road, vehicle position and obstacle information obtained by the sensor equipment, the road signal control is coordinated to improve the quality and efficiency of road management. The decision-making of unmanned vehicles is controlled, and the safe distance between unmanned vehicles is adjusted to ensure that vehicles can travel safely and reliably on the road. As one of the most important autonomous driving sensors, LiDAR can provide the decision-making system with the most important information about the location / size / motion of traffic participants in the fields of smart transportation / autonomous driving.

[0044] In the autonomous driving scenario, the system architecture suitable for the autonomous driving scenario may include mobile devices, networks, and the cloud. The cloud may include a server cluster or a distributed system consisting of multiple physical servers. Electronic devices such as sensors may be provided on the mobile device to obtain information about obstacles in the surrounding environment of the mobile device. Electronic devices may include radars (such as lidars), image sensors, etc.

[0045] Compared with image data, point cloud data has distance information, reflectivity information, etc., and can more comprehensively represent spatial location information. However, as shown above, the amount of point cloud data is huge. When point cloud data needs to be stored as evidence or backup data, it takes up a huge amount of storage space. This is unacceptable for application scenarios that continuously generate massive amounts of data and need to be stored on mobile platforms (such as autonomous driving scenarios).

[0046] Specifically, the LiDAR main control chip is the core component of the LiDAR, which is responsible for the collection, processing and transmission of all LiDAR information. However, the point cloud map collected by the LiDAR contains redundant and invalid information most of the time, and the massive amount of invalid information also makes it difficult to search for effective information. For example, if a user wants to find the relevant point cloud data of a certain type of object (such as an ambulance) in the point cloud of the driving record, a lot of manual search is required and there is a possibility of missed detection. This will cause a huge waste of on-board storage resources and great difficulties in searching for effective information.

[0047] Point cloud data includes unordered point cloud data, and it is difficult to retrieve target point cloud frames (such as point cloud frames collected in specific scenarios) from massive point cloud data. For example, it is difficult for users to find point cloud frames when the mobile platform passes through dangerous sections from the stored multi-frame point cloud to review driving habits. For example, it is difficult for users to find point cloud frames when the vehicle overtakes or is overtaken from the stored multi-frame point cloud to review dangerous driving behaviors. For example, it is difficult for users to find point cloud frames collected in scenes related to specific types of objects (such as special vehicles passing by such as police cars and ambulances) from the stored multi-frame point cloud to view more relevant information. For example, it is difficult for users to find point cloud frames when vehicles collide from the stored multi-frame point cloud to use as evidence. For example, useful point cloud data stored by users in non-volatile memory is overwritten by useless point cloud data collected subsequently, resulting in the loss of useful information that cannot be recovered.

[0048] The embodiments of the present application provide a circuit and method, a chip, and an electronic device for storing data. After collecting multiple point cloud frames, the multiple point cloud frames are not directly written into a non-volatile memory, but the recognition results for the target object in each point cloud frame are obtained, and the association relationship between the target object and the multiple point cloud frames is determined. In this way, the point cloud frames and / or audio frames containing the target object can be output in association with each other based on the association relationship, so that the useful point cloud frames (containing the target object), audio frames, and corresponding recognition results can be written into the non-volatile memory in association. In this way, the proportion of useful data stored can be effectively increased, and it is convenient to retrieve the target point cloud frame based on the recognition results and the association relationship, thereby improving the convenience and efficiency of retrieval.

[0049] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0050] Figure 1 It is a schematic diagram of an application scenario of a circuit and method for storing data, a chip, and an electronic device shown in an embodiment of the present application.

[0051] Figure 1The hardware structure of the vehicle 10 supporting assisted driving or automatic driving functions is shown. For example, at least one LIDAR 11 is mounted on the roof and / or the side of the vehicle body of the vehicle 10. The detection area of ​​the LIDAR 11 can be fixed, such as a certain LIDAR 11 can be used only for detection of a preset area. The detection area of ​​the LIDAR 11 can be adjustable, such as the laser radar on the vehicle body can scan multiple detection areas by adjusting the posture, etc., or by adjusting the field of view of the laser radar itself. The vehicle 10 is also provided with a sound sensor, such as a microphone or a microphone array. For example, the microphone is used to collect sound information. For example, the microphone array is used to collect sound information for multiple detection areas by means of beam nulling, etc., wherein the multiple detection areas of the microphone array can overlap at least partially with the multiple detection areas of the LIDAR 11 to improve the consistency of the objects included in the point cloud data and the audio data.

[0052] In addition, the vehicle 10 may be equipped with a plurality of millimeter wave radars in a manner surrounding the vehicle 10. For example, the vehicle 10 may be equipped with four millimeter wave radars to detect the left side in front of the vehicle, the right side in front of the vehicle, the left side behind the vehicle, and the right side behind the vehicle. The millimeter wave radars can detect the distance of an object in each detection area, and can detect the relative speed between the object and the vehicle 10.

[0053] Furthermore, the vehicle 10 may also be equipped with a positioning device 12 , such as a Beidou positioning device, a Global Positioning System (GPS), etc. The current position of the vehicle 10 can be determined by the positioning device 12 .

[0054] In addition, the vehicle 10 may also be equipped with an electronic control unit (ECU). The detection signal of at least one of the above-mentioned LIDAR 11, millimeter wave radar and positioning device 12 is sent to the ECU. The ECU can detect and identify obstacles (such as roadblocks, moving objects, trees, adjacent vehicles, etc. around the vehicle 10) based on these signals. In addition, the ECU can be physically divided into multiple units according to their functions, and this application will collectively refer to them as ECUs.

[0055] It should be noted that although the movable device is described as a car, such description is not restrictive and is applicable to a variety of movable devices, such as land robots, water robots, etc.

[0056] Figure 2 It is a schematic diagram of another application scenario of the circuit and method for storing data, chip and electronic device shown in one embodiment of the present application.

[0057] Figure 2 The aircraft 20 shown may be an aircraft with an assisted driving function and an automatic driving function. The aircraft 20 may include a sensor system 21 and a power mechanism 22 .

[0058] The sensing system 21 may include one or more sensors to sense at least one of the surrounding obstacles, spatial orientation, speed or acceleration of the aircraft 20. The types of sensors include, but are not limited to, ranging sensors, position sensors, motion sensors, inertial sensors or image sensors. The sensing data provided by the sensing system 21 can be used to control the spatial orientation, speed and / or acceleration of the aircraft 20. The sensing system 21 is used to collect relevant information of the aircraft 20. Different types of sensors can sense different types of signals or sense signals from different sources. For example, the sensor includes a radar, an inertial sensor, a GPS sensor, or a visual / image sensor (such as a camera).

[0059] The circuit and method for storing data, chip and electronic device of the embodiments of the present application can be applied to Figure 1 or Figure 2 The arrangement shown is any one or more electronic devices that generate point clouds, such as LIDAR 11, millimeter wave radar, etc., and need to store point cloud data.

[0060] In the above-mentioned application scenarios such as assisted driving, autonomous driving, and smart transportation, quickly and accurately perceiving the surrounding environment of movable equipment is a key point.

[0061] Here, the vehicle's autonomous driving scenario is used as an example for illustrative explanation. According to the vehicle position information, obstacle information, and road information perceived by the sensor system, road signal control is coordinated to improve the quality and efficiency of road management. Specifically, the corresponding autonomous driving vehicle decision can be determined based on the information perceived by the sensor system, and the safe distance between autonomous driving vehicles can be adjusted, so that the vehicle can drive safely and reliably on the road.

[0062] As one of the most important sensors for autonomous driving, LiDAR can provide the decision-making system with particularly important information such as the location, size, and motion information of traffic participants in the fields of smart transportation, assisted driving, and autonomous driving. LiDAR is equivalent to human eyes for autonomous vehicles. Non-volatile memory is equivalent to the human brain and needs to store the collected point cloud data. However, the storage space of non-volatile memory is limited, and the time required to retrieve the target point cloud frame increases with the amount of stored point cloud data. How to ensure that the valid point cloud data is stored completely and that the target point cloud frame can be easily retrieved has become an urgent problem to be solved.

[0063] Based on the above reasons, the present application aims to provide a technical solution that can utilize limited storage space to store as much useful point cloud data as possible and improve the convenience of data retrieval.

[0064] Figure 3 It is a block diagram of a circuit for storing data according to an embodiment of the present application.

[0065] See also Figure 3 The data storage circuit 300 may include: a cache module 310 , a recognition result acquisition module 320 , an assembly module 330 and an output module 340 .

[0066] Among them, the cache module 310 is configured to cache point cloud data and audio data, the point cloud data includes multiple point cloud frames, the audio data includes multiple audio frames, and there is a corresponding relationship between the multiple point cloud frames and the multiple audio frames. The cache module 310 refers to a memory that can perform high-speed data exchange. It exchanges data with the processor before the memory, so the rate is very fast. The cache module may include one or more cache units. For example, the first cache unit is used to cache point cloud data, and the second cache unit is used to cache audio data. For example, the first cache unit is used to cache point cloud data and audio data. The cache has the following main functions: pre-reading, writing, and temporary storage. Temporary storage and writing functions are at least partially involved in this application. The cache can be responsible for the storage and placement of point cloud data and audio data. This facilitates the identification of cached audio frames, the assembly of audio frames and point cloud frames, and the writing of target point cloud frames and\or target audio frames to non-volatile memory. In addition, the cache module 310 can also greatly reduce the time for data reading and writing to improve the efficiency of hard disk use. At the same time, the use of cache can also reduce the frequent reading and writing of non-volatile memory, increase the service life of non-volatile memory, and save more power.

[0067] The recognition result acquisition module 320 is coupled to the cache module 310 and is configured to obtain recognition results for multiple audio frames and obtain recognition results for the target object. Specifically, the recognition result acquisition module 320 can call the recognition result, or it can obtain the recognition result by itself through the recognition algorithm. For example, the cached point cloud data can be sent to the processor or the cloud for audio recognition, and the recognition result is called by the recognition result acquisition module 320. For example, the main control chip of the laser radar can use a trained neural network to process the audio data to obtain the recognition result.

[0068] The assembly module 330 is coupled to the recognition result acquisition module 320, and is configured to determine the association relationship between the target object and the multiple point cloud frames based on the correspondence relationship in response to the recognition result indicating the existence of the target object. Specifically, the recognition result and the corresponding audio frame can be bound, and since there is a correspondence between the audio frame and the point cloud frame, an association relationship between the target object and the multiple point cloud frames can be established. For example, if the recognition results of the i-th frame to the i+n-th frame indicate the existence of an ambulance (such as identifying the siren of an ambulance from the audio frame), the recognition result "ambulance" is used as the label of the point cloud frame corresponding to the i-th frame to the i+n-th frame audio frame.

[0069] The output module 340 is coupled to the recognition result acquisition module 320 and the assembly module 330 respectively, and is configured to output multiple point cloud frames corresponding to the recognition result and at least one of the following: the recognition result or multiple audio frames corresponding to the recognition result in association with the association relationship, so as to write to the non-volatile memory. The output module 340 can output the audio frame and its recognition result and the corresponding point cloud frame in association with other electronic components, such as non-volatile memory, processor, etc. For example, the recognition result "ambulance" and the point cloud frames corresponding to the i-th frame to the i+n-th frame audio frame are written to the hard disk in association.

[0070] It should be noted that, since some radars have strong directionality, such as when a laser radar only emits lasers within a specific angle range in the direction of travel, only point cloud data within the specific angle range can be obtained. The microphone is not limited by the specific angle range, which means that after the target object is identified from the audio frame, the target object may not exist in the point cloud frame collected at the same time. For example, an ambulance is traveling in parallel to the side of a movable platform. If the side of the movable platform is a blind spot of the laser radar (such as laser radars are set in front and behind the movable platform, but no laser radars are set on the side), in order to solve this problem, a microphone array can be used, and the beam direction of the microphone array should be kept consistent with the specific angle range of the laser radar as much as possible. For example, directivity can be achieved by zeroing the sound signal outside the specific angle range.

[0071] In addition, since the audio data includes sound intensity information, after determining that the recognition result indicates the existence of the target object, it is also possible to further determine whether the target object needs to be screened out based on the sound intensity information. For example, the current vehicle is in lane 1, the ambulance is in lane 4, and the sound intensity of the siren of the ambulance collected is relatively low. Although the recognition result of the audio data obtained by the microphone includes the ambulance, the ambulance is far away from the current vehicle, and the probability that the collected audio data and point cloud data need to be retained is relatively low. Therefore, there is no need to store the audio data and point cloud data, nor is there a need to build an association between the ambulance, the audio frame, and the point cloud frame, thereby reducing the consumed computing resources. In addition, the directionality of the sound can be further considered to further improve the accuracy of the determined point cloud data to be stored. For example, the ambulance is on the side of the mobile platform, and there is no laser radar on the side of the mobile platform. There is no ambulance in the point cloud collected by the laser radar in the driving direction. At this time, it is meaningless to save the point cloud data.

[0072] In addition to sound intensity information, audio data also has voiceprint features. For example, male voices, female voices, gasoline engine sounds, diesel engine sounds, electric motor sounds, collision sounds, etc. each have different voiceprint features. The target object can be determined according to actual needs, and then the presence of the target object can be identified from the audio frame.

[0073] Figure 4 It is a schematic diagram of an application scenario shown in an embodiment of the present application.

[0074] See also Figure 4 , there are cyclists and cars in front of the mobile platform. If the target object includes a bicycle bell, a car horn, a collision sound, or a sudden brake sound, once the above target object appears in the audio frame, the audio frame and the point cloud frame corresponding to the audio frame can be stored in a non-volatile memory for subsequent call.

[0075] In addition, the sound intensity information can be further integrated to determine whether the target object needs to be screened out. For example, although the sound of an emergency brake appears in the audio frame, the sound of the emergency brake is weak. If the sound intensity is less than a preset threshold, the audio frame and the corresponding point cloud frame may not be output. If the sound intensity is greater than the preset threshold, the audio frame and the corresponding point cloud frame may be output.

[0076] It should be noted that the embodiments of the present application do not exclude related technical solutions for image recognition or point cloud recognition of point cloud frames, and the two technical solutions can be used in combination. For example, after identifying the target object (the siren of the ambulance) in the audio frame, the ambulance can also be identified from the point cloud data. Since the point cloud data contains precise spatial position information, the distance between the ambulance and the mobile platform can be accurately calculated. If the distance is greater than the preset distance threshold, there is no need to retain the current point cloud frame and the current audio frame.

[0077] Point cloud data has a large volume, reaching several Gb or even Tb levels, and contains a lot of redundancy, which brings challenges to processing speed and storage. In addition, due to the high-speed driving of the car, the recognition speed is required to be high. The computing volume and processing speed of audio recognition are faster than those of point cloud recognition, which can better meet the needs of online recognition by reducing the amount of calculation and improving the recognition speed.

[0078] In addition, it can also be used in automated road inspections. For example, if sudden braking sounds, violent shock absorption sounds, etc. are frequently collected on a certain road section, the synchronously collected point cloud data can be stored to determine whether the road needs to be optimized based on the precise spatial position information in the point cloud data.

[0079] The data storage circuit provided in this embodiment can intelligently filter point cloud frames and / or audio frames where specific objects appear for storage. For example, based on edge computing of artificial intelligence, after the laser radar collects point cloud data, the point cloud frames are filtered based on the synchronously collected audio frames, and only valid point cloud data and / or audio frames where specific objects appear are retained and stored. This greatly reduces the demand for on-board storage and the difficulty of retrieving key information.

[0080] Figure 5 It is a block diagram of a recognition result obtaining module shown in an embodiment of the present application.

[0081] See also Figure 5 The above-mentioned recognition result obtaining module 320 may include: a frame number counting circuit 321, a target audio frame timestamp determining unit 322, and a target point cloud frame position determining unit 323.

[0082] The frame number counting circuit 321 is coupled to the buffer module 310 and is configured to count the frame numbers of the multiple point cloud frames. Specifically, the frame numbers of the point cloud frames can be counted by a counter, an accumulator, or the like.

[0083] The input end of the target audio frame timestamp determination unit 322 is coupled to the output end of the frame number statistics circuit 321 and the cache module 310 respectively, and is configured to determine the timestamp of the target audio frame based at least on the frame number and correspondence of the multi-frame point cloud frame, and the target audio frame includes the audio emitted by the target object.

[0084] The input end of the target point cloud frame position determination unit 323 is coupled with the output end of the frame number statistics circuit 321 and the target audio frame timestamp determination unit 322 respectively, and is configured to determine the timestamp of the target point cloud frame based on the timestamp and the corresponding relationship of the target audio frame, and determine the position of the target point cloud frame in the multi-frame point cloud frame based on the timestamp of the target point cloud frame and the frame number of the multi-frame point cloud frame.

[0085] In order to reduce the consumption of computing resources, a rough timestamp calculation method can be used. For example, if the frame rate of the point cloud frame is 30fps, then every 30 frames, add 1 second to the time and add timestamp +1. In order to improve the accuracy of the stored point cloud frames and the accuracy of the retrieval results, a more sophisticated timestamp calculation method can also be used. For example, the timestamp is increased by 1 for each frame received. Timestamps are added to each point cloud frame by counting the number of frames to ensure the uniqueness of the identification of each point cloud frame.

[0086] For example, if the target object (ambulance siren) appears in the current audio frame, then it is inferred that an ambulance appears in the point cloud frame. If the timestamp of the point cloud frame is timestamp x, and the sirens of ambulances appear in the audio frames corresponding to the subsequent 1000 point cloud frames, then the position of the target point cloud frame is: timestamp x to timestamp x+1000. The target point cloud frame can be found by finding the point cloud frames corresponding to timestamp x to timestamp x+1000.

[0087] The cache module may be composed of one or more cache units, and the multiple cache units may be arranged in the same area or separately. In some embodiments, the cache module includes an audio cache unit for caching audio data. Accordingly, the structure of the target audio frame timestamp determination unit may be as follows. Figure 6 It is a block diagram of a target audio frame timestamp determination unit shown in an embodiment of the present application.

[0088] See also Figure 6 The target audio frame timestamp determination unit 322 includes: an identification circuit 3221 and a target object appearance frame timestamp determination unit 3222 .

[0089] The input end of the recognition circuit 3221 is coupled to the audio frame buffer unit and is configured to recognize each frame in the multiple audio frames to obtain a recognition result for each target object in the current audio frame.

[0090] Specifically, the model topology and model parameters may be received to obtain a recognition model, so as to use the recognition model to process point cloud data or point cloud frames to obtain a recognition result for the target object.

[0091] The target object appearance frame timestamp determination unit 3222 is coupled to the frame number statistics circuit 321 and the recognition circuit 3221 respectively, and is configured to determine the timestamp of the audio frame where each target object appears based on the frame number and correspondence of the multi-frame point cloud frame and the recognition result of each target object in the current audio frame.

[0092] For example, the frame number of the point cloud frame when the target object is recognized from the audio frame is determined, and then the frame number of the point cloud frame is used as the timestamp of the audio frame where the target object appears.

[0093] The model parameters used by the recognition circuit can be obtained by training a recognition model with a specific model topology using training data. The topology of the recognition model includes but is not limited to: Linear Regression, Logistic Regression, Decision Trees, Naive Bayes, K-Nearest Neighbors, Support Vector Machines, Random Decision Forests or Bagging, Neural Networks, etc.

[0094] For example, a training data packet includes multiple audio frames, including an audio frame of a preset target object with a preset target object identifier. For example, audio frame 1 has an ambulance siren label, audio frame 2 has a police car siren label, audio frame 3 has an ambulance siren label and a police car siren label, etc. The training data can be divided into training data for multiple scenarios. The recognition model can be a recognition model composed of multiple recognition sub-models obtained by training multiple recognition sub-models respectively according to training data for different scenarios. The recognition model can be a model obtained by training the same recognition model using training data for different scenarios. Scenarios include but are not limited to: congested sections, highways, residential roads, national highways, windy areas, peak hours, night time, etc. This helps to improve the recognition accuracy of the recognition model.

[0095] Figure 7 It is a block diagram of a frame number statistics circuit shown in an embodiment of the present application.

[0096] See also Figure 7 , where the frame counting circuit is realized by an adder counter as an example, a binary adder counter can be formed by a trigger, such as a k-bit synchronous binary adder counter formed by a T trigger. For example, a 3-bit binary asynchronous adder composed of 3 falling edge triggered triggers T, the output signal of the trigger T is controlled by a pulse signal, and the counter starts counting from the state of Q2 Q1Q0 = 000.

[0097] Under the trigger of the falling edge of the counting input pulse CP, the output Q0 of the trigger FF0 will flip. 0 becomes 1 or 1 becomes 0. Since CP1 is taken from Q0, the output Q1 of the trigger FF1 will flip under the trigger of the falling edge of Q0. Similarly, since CP2=Q1, the output Q2 of the trigger FF2 will flip under the trigger of the falling edge of Q1. If the trigger T′ is triggered by the rising edge, an asynchronous binary addition counter can also be formed, but the carry pulse of each level of the trigger should be changed to the Qˉ terminal output. This is because when the output terminal Q of the low-order trigger changes from 1 to 0, the rising edge of the Qˉ terminal can just be used as a trigger pulse for the high-order.

[0098] Taking the subtraction counter as an example, if the triggers T′ are connected according to the binary subtraction rule, a binary subtraction counter can be obtained. According to the binary subtraction counting rule, if the low-order trigger is already 0, it should flip to 1 after inputting a subtraction counting pulse, and at the same time send a borrow signal to the high-order bit to flip the high-order bit.

[0099] In addition, it should be noted that counting can also be done by software. For example, the operation: n&(n–1) can be repeatedly performed until n becomes 0. The function of the operation: n&(n–1) is to remove the last 1 of n, that is, the last 1 of n becomes 0. For example, let n=11111, then the calculation process is as follows: n=11110, the count is 1; n=11100, the count is 2; n=11000, the count is 3; n=10000, the count is 4; n=00000, the count is 5.

[0100] In some embodiments, the target object appearance frame timestamp determination unit includes: a plurality of parallel timestamp determination subunits, each timestamp determination subunit corresponding to a target object.

[0101] Figure 8 It is a schematic diagram of the recognition result for an audio frame shown in an embodiment of the present application.

[0102] See also Figure 8 , taking the Long Short-Term Memory (LSTM) network as an example for illustrative explanation. LSTM is a time recursive neural network that is suitable for processing and predicting important events with relatively long intervals and delays in time series. The recognition result of LSTM can include confidence levels for multiple preset results. For example, the training data includes training samples for multiple target objects. If a target object is included in an audio frame, a label of the target object can be set for the audio frame. Figure 8The recognition results shown in the figure include the sound of an ambulance, a police car, a male scream, a female scream, a collision sound, an animal call, etc. After a frame of audio is input into the LSTM, the confidence for each preset target object is obtained. If the confidence of a preset target object is higher than the preset confidence threshold, the recognition result representing the preset target object included in the audio frame is output. It should be noted that multiple preset target objects can exist in one audio frame at the same time. In addition, users can split or combine the above categories according to usage requirements. For example, a male scream can be split into a boy scream and an adult male scream. For example, a male scream and a female scream can be combined into a scream.

[0103] Figure 8 If the confidence of the ambulance sound exceeds the confidence threshold, the recognition result corresponding to the ambulance sound is output. Since each preset target object has a corresponding timestamp determination subunit, the timestamp determination subunit corresponding to the ambulance sound obtains the frame number of the point cloud frame from the frame number statistics circuit when receiving the recognition result, and sets a timestamp for the audio frame in which the ambulance sound currently exists. In this way, an accurate timestamp can be set for each audio frame in which a target object exists. In addition, since each timestamp determination subunit corresponds to a target object, when there are multiple target objects in an audio frame, the same timestamp can be set for each target object of the current audio frame. In this way, when retrieving a certain target object, all timestamps corresponding to each target object can be accurately found, and then all audio frames in which a certain target object exists can be found based on all timestamps.

[0104] In some embodiments, in order to reduce the chip area occupied by the target object appearance frame timestamp determination unit and improve the signal processing speed, it can be achieved by multiplexing the timestamp determination subunit. Specifically, multiple preset target objects can reuse the same timestamp determination subunit. For example, the timestamp determination subunit may include a table lookup circuit. When the LSTM output result is 10000000, indicating the presence of an ambulance sound, the lookup table (Look-Up-Table, LUT for short) circuit selects the channel corresponding to the ambulance sound to be turned on, so as to set a timestamp for the current audio frame in which the ambulance sound exists.

[0105] Fig. 9 is a schematic diagram of a target audio frame shown in an embodiment of the present application.

[0106] See also Fig. 9In the current audio frame, there are two types of objects: footsteps and male screams. Among them, footsteps are not the target object, and male screams are the target object. There is a target object in the current audio frame, which is the target audio frame. Therefore, it is necessary to store the current audio frame and the audio frames in which there is at least one target object in multiple adjacent audio frames, as well as the corresponding point cloud frames in the non-volatile memory. This makes it easy to retrieve audio data and / or point cloud data as evidence to restore the scene at the time of the incident.

[0107] The target object occurrence frame timestamp determination unit can achieve: determining the audio frame where the target object exists from multiple preset target objects, and setting an accurate and unique timestamp for the audio frame where the target object exists. Figure 8 , setting a timestamp for the current audio frame in which the ambulance sound appears. For example, the target object appearance frame timestamp determination unit may include a multiplexer (MUX for short). The inputs of the multiplexer each correspond to a preset target object, and are configured to determine the existing target object from multiple preset target objects, so as to set a timestamp for the point cloud frame including the target object. Fig.10 , the multiplexer may be a selector. It should be noted that, in order to switch the path of the preset target object, at least one of a wiring switch, a lookup table circuit, etc. may also be used.

[0108] For example, the implementation logic of the two-choice MUX can be shown as formula (1).

[0109]

[0110] Among them, Y is the signal output by MUX, S is the control signal, NOT S, A and B are the two signals input to the MUX. In addition, a buffer (such as a single NOT gate) can be connected to the output of the multiplexer to provide a positive logic output (logic "1", high) at one terminal and a complementary negative logic output (logic "0", low) at the buffer output.

[0111] It should be noted that the logic operation circuits and combinations thereof shown above are only illustrative examples and are not limited here. It should be noted that logic operations include but are not limited to: "OR", "AND", "NOT", "NOR", "NAND", "XOR", etc. Logic operation circuits include but are not limited to: "OR operation circuit", "AND operation circuit", "NOT operation circuit", "OR NOR operation circuit", "NAND operation circuit", "XOR operation circuit", etc. These logic operation circuits may also be referred to as logic operation units. In order to realize functions such as counting, threshold comparison, and determination of target point cloud frames, a combination of multiple logic operation circuits may be used to achieve this. Accordingly, a combination of multiple logic operation circuits may be used to achieve this, which is not limited here.

[0112] Fig.11 is another block diagram of a circuit for storing data according to an embodiment of the present application.

[0113] See also Fig.11 The target point cloud frame position determination unit may include: a plurality of parallel target object position determination subunits, each target object position determination subunit having a one-to-one corresponding and coupled timestamp determination subunit, and coupled with the frame number statistics circuit 321, configured to determine the frame position range of the target object in the neighboring point cloud frames of the current point cloud frame from the cached point cloud data, and output the frame position range to the assembly module.

[0114] In this embodiment, the frame position may refer to a certain frame, such as the 0th frame, the i-th frame, etc., where i is a positive integer greater than or equal to 0. The setting of the timestamp facilitates the determination of the correspondence between multiple data streams. For example, if the recognition result output by the LSTM indicates that both the sound of an ambulance and the sound of a police car exist in the current audio frame, it is necessary to set the identifier 1 for the current audio frame with the sound of an ambulance so as to facilitate the subsequent search for the current audio frame, and set the identifier 2 for the current audio frame with the sound of a police car so as to facilitate the subsequent search for the current audio frame. The timestamp can synchronize identifiers 1 and 2 to the current audio frame and the current point cloud frame.

[0115] In addition, in order to reduce the chip area occupied by the target point cloud frame position determination unit and improve the signal processing speed, it can be achieved by multiplexing the target object position determination subunit. Specifically, multiple parallel timestamp determination subunits can reuse the same target object position determination subunit. For example, the target object position determination subunit may include a table lookup circuit, and when a timestamp determination subunit outputs a timestamp, the table lookup circuit selects the channel corresponding to the timestamp determination subunit to be turned on to determine the frame position range corresponding to the timestamp.

[0116] It should be noted that the specific value of the timestamp of the 0th audio frame of the audio stream (also called the stream timestamp) is not very important. For example, the stream timestamp of the 0th frame of the audio stream is 0 (it can also be the current system time), and the audio frame duration is 20ms; the stream timestamp of the 1st frame is 20, and the 2nd frame is incremented in sequence. There are errors in the timestamp conversion between different time bases. When converting, it should be ensured that the timestamp after conversion is not less than the timestamp before conversion. For another example, if the sampling frequency is 80000Hz, the timestamp unit is 1 / 80000, assuming that 1 second is divided into 80000 time blocks. If 25 frames are sent per second, the transmission of each frame occupies 80000÷25=3200 time blocks. Therefore, if the timestamp increment is defined as the time interval between two adjacent frames, the timestamp increment should be 3200. The conversion between timestamp and frame position range can be achieved based on the above relationship.

[0117] In some embodiments, in order to reduce the risk of repeatedly storing the same point cloud frame and improve the effective utilization of storage space, the above-mentioned circuit for storing data may also include: a deduplication module, coupled to the plurality of target object position determination subunits respectively, configured to perform deduplication processing on the frame position range to obtain a deduplication frame position range. For example, please refer to Figure 8 , if the target objects "police car sound" and "ambulance sound" exist in the current point cloud frame at the same time, the two timestamp determination subunits corresponding to the two target objects both output timestamps, or the two target object position determination subunits corresponding to the two target objects both output frame position ranges, resulting in the frame position range being recorded twice, causing the current point cloud frame to be stored twice, wasting limited storage space. In this embodiment, after determining the correspondence between the target objects, video frames, and point cloud frames, the frame positions that appear multiple times for a point cloud frame are deduplicated to ensure that the point cloud frames with the same frame positions are only stored once, effectively reducing the possibility of repeated storage of the same point cloud frames, and helping to improve the effective utilization of storage space.

[0118] In some embodiments, the circuit for storing data may further include: an encryption module. The encryption module is coupled to the assembly module 330 and is configured to encrypt multiple point cloud frames corresponding to the target object to obtain encrypted multiple point cloud frames to be stored. Specifically, the point cloud data may be encrypted by hardware encryption or software encryption.

[0119] For example, hardware encryption can be implemented in the following way. The laser radar main control chip (hereinafter referred to as the main control chip) generates a random number, sends the random number to the encryption chip, and the encryption chip encrypts it and sends it back to the main control chip. The main control chip determines whether the data is the expected random data. If it is, the program of the main control chip continues to execute, otherwise it reports an error or stops output, such as stopping writing point cloud data to the non-volatile memory.

[0120] Taking the logic encryption chip as an example, it can be protected and authenticated by an electrically erasable programmable read-only memory (EEPROM) and peripheral logic circuits through a simple identification (ID) number or a logic authentication password; this method is low-cost and simple to develop, but the protection capability needs to be improved. Taking the encryption chip with a fixed algorithm and a comparison authentication mode as an example, the comparison mode is adopted; the program in the protected CPU is complete, and there are several isolated comparison points between it and the encryption chip. Taking the chip core as an example, the encryption chip has become part of the software product, and the program in the encryption chip cannot be read or copied, making it difficult to crack.

[0121] In a specific embodiment, the encryption module may include: a key generation unit and an encryption unit. The key generation unit is coupled to the recognition result acquisition module, and is configured to generate a secret key based on at least part of the numerical value in the probability of the recognition result. For example, if the confidence that the sound of an ambulance exists in the current point cloud frame is 93.32567%, then the values ​​from the 4th to the 6th digits can be taken as the basic values, and then a specific operation (such as a hash operation) is performed on the basic values ​​to generate an encryption key. The secret key obtained in this way is highly random and has a high level of security. The encryption unit is coupled to the key generation unit, and is configured to encrypt multiple point cloud frames and / or audio frames containing target objects based on the secret key.

[0122] In addition, software encryption can also be performed in the following manner. Before sending information, call the information security module to encrypt the information, and then send the encrypted data. After the encrypted data reaches the recipient, it is decrypted and restored by the corresponding decryption software. The software encryption method has the following advantages: there are already standard security application programming interface (Application Programming Interface, referred to as API) products, easy to implement, and good compatibility.

[0123] In some embodiments, the output module 340 may include a write control unit to write the target point cloud frame into a non-volatile memory, wherein the non-volatile memory includes but is not limited to at least one of the following: an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash memory, etc.

[0124] Specifically, the input end of the write control unit is coupled to the reorganization module or the encryption module, and the output end of the write control unit is coupled to the non-volatile memory, and is configured to write the recognition results and multiple point cloud frames and multiple audio frames corresponding to the recognition results into the non-volatile memory based on the association relationship.

[0125] For example, in scenarios such as traffic accidents, in addition to using the video data recorded by the dashcam as evidence, the stored audio frames with the target object and the corresponding point cloud data can also be used as evidence in this embodiment. Compared with the image information that does not have distance information, the point cloud data stored in this embodiment also stores accurate distance information, and the audio data contains the sound information of the scene, which can better restore the scene of the traffic accident and provide higher quality evidence. In addition, in this embodiment, only high-value audio data and / or point cloud data are saved, and useless audio data (such as audio frames without target objects) and / or point cloud frames are not retained as much as possible, thereby improving the effective utilization of storage space. Compared with the storage method of overwriting old data with cached new data in related technologies, the risk of the target object's point cloud data being overwritten and unable to be restored is effectively reduced.

[0126] For example, in a backtracking scenario, since the corresponding relationship between the position ranges of the target object, audio frame, and point cloud frame has been determined before storing the audio data and / or point cloud data, the user can quickly and efficiently retrieve the target point cloud frame based on the target object, thereby improving the retrieval efficiency and the accuracy of the retrieval results.

[0127] The data storage circuit provided in the present application can be applied to multiple fields such as unmanned driving, drones, exploration, inspection, etc. The data storage circuit provided in the present application can store effective data in a targeted manner and improve the convenience of data retrieval.

[0128] Another aspect of the present application also provides a chip.

[0129] Fig.12 It is a block diagram of a chip shown in an embodiment of the present application.

[0130] See also Fig.12 The chip 1200 includes: a distance calculation unit 1210 and the above-mentioned data storage circuit 300.

[0131] The distance calculation unit 1210 is configured to calculate the distance between the object and the mobile platform based on the time difference between the emission time point of the laser and the reception time point of the reflected laser, and generate point cloud data.

[0132] The circuit 300 storing data is configured to output the point cloud frame in which the target object exists in the point cloud data in association with at least one of the following: a recognition result of the target object or an audio frame in which the target object exists.

[0133] The components of the data storage circuit 300 may refer to the relevant descriptions in the above-mentioned embodiments, and will not be described in detail here.

[0134] Fig.13It is a block diagram of a chip shown in another embodiment of the present application.

[0135] See also Fig.13 In some embodiments, the chip 1200 may further include: a laser receiving unit.

[0136] Among them, the laser receiving unit is coupled with the distance calculation unit 1210 and is configured to receive the laser reflected by the object. When the laser encounters an obstacle, it will be reflected and then received by the laser receiving unit. The laser receiving unit creates a set of point clouds based on the time when each laser beam is emitted and returned. The principle is to measure the propagation distance (Time of Flight, referred to as TOF) between the laser radar and the target object, analyze the reflected energy size, amplitude, frequency and phase of the reflected spectrum on the surface of the target object, and output the point cloud, thereby presenting the accurate three-dimensional structure information of the target object.

[0137] Specifically, the laser receiving unit can be composed of a receiving optical system, a photodetector, an amplifier (preamplifier, main amplifier), etc. The laser is calibrated by an optical lens and received by a photosensitive device (photodiode). After the photodiode receives light, it will generate a photocurrent of corresponding intensity depending on the light intensity. The current is amplified by the amplifier to output an electrical signal.

[0138] In some embodiments, the chip 1200 may further include a laser emitting unit.

[0139] The laser emitting unit is configured to emit laser light. Fig.13 ,The laser emitted by the laser emitting unit is reflected by the target object (such as a passerby) and then received by the laser receiving unit to form point cloud data.

[0140] The laser emitting unit works by emitting layers of laser beams outward. The more layers there are, the higher the accuracy will be, but it may also lead to a larger sensor size. For example, a high-quality lidar can emit more than 200 laser beams per second.

[0141] For the wavelength of the laser, laser transmitters with wavelengths of around 900nm and 1550nm can be used. High power can achieve a longer detection distance, and long wavelength can also improve anti-interference capabilities. Safety is ensured by limiting power and pulse time.

[0142] In some embodiments, a non-volatile memory may be integrated into the chip to improve the security of the point cloud data. In addition, in order to reduce the chip size or reduce the chip cost, an external non-volatile memory may be used to store the audio data and / or the point cloud data.

[0143] For example, the chip 1200 may further include a non-volatile memory. The non-volatile memory is coupled to the data storage circuit 300 and is configured to store multiple point cloud frames corresponding to the target object, or to store the target object and multiple point cloud frames corresponding to the target object.

[0144] In some embodiments, an audio acquisition unit may be integrated into the chip to improve chip integration and data security. In addition, in order to reduce chip size or chip cost, audio data collected by an external audio acquisition unit may also be called.

[0145] For example, the chip 1200 may further include a microphone or a microphone array. If a microphone array is provided, the beam range of the microphone array may be similar to or identical to the beam range of the emitted laser beam, so as to improve the consistency between the objects appearing in the audio frame and the objects appearing in the point cloud frame.

[0146] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0147] Another aspect of the present application further provides a radar. The radar may include the chip or board as described above.

[0148] Fig.14 It is a schematic diagram of the structure of a radar shown in an embodiment of the present application.

[0149] See also Fig.14 The radar 1400 may include a chip or board as shown above, and the chip and / or board may include a circuit for storing data as described above. The circuit may be arranged on a board 1410, and a plurality of chips, such as a central control chip, may be arranged on the board 1410. The board 1410 may be arranged in a housing 1420.

[0150] Specifically, the radar 1400 may include a circuit for storing data as described above, or a chip as described above. The radar may be a laser radar, a millimeter wave radar, an ultrasonic radar, etc. The radar may be a scanning radar or a non-scanning radar.

[0151] The key components of LiDAR include control hardware DSP (digital signal processor), laser driver, laser emitting light emitting diode, transmitting optical lens, receiving optical lens, APD (avalanche optical diode), TIA (variable transconductance amplifier) ​​and detector according to the signal chain of signal processing. Among them, except for the transmitting and receiving optical lenses, all are electronic components. With the rapid evolution of semiconductor technology, the performance is gradually improved while the cost is rapidly reduced. Optical components account for the higher cost of LiDAR.

[0152] Another aspect of the present application also provides a method for storing data.

[0153] Fig.15 It is a flow chart of a method for storing data shown in one embodiment of the present application.

[0154] See also Fig.15 , the method for storing data may include operations S1510 to S1540.

[0155] In operation S1510, point cloud data and audio data are cached, the point cloud data includes multiple point cloud frames, the audio data includes multiple audio frames, and there is a corresponding relationship between the multiple point cloud frames and the multiple audio frames.

[0156] In operation S1520, a recognition result for the multiple audio frames is obtained to obtain a recognition result for the target object.

[0157] In operation S1530 , in response to the recognition result indicating that the target object exists, an association relationship between the target object and the multiple point cloud frames is determined based on the correspondence relationship.

[0158] In operation S1540, a plurality of point cloud frames corresponding to the recognition result and at least one of the recognition result or a plurality of audio frames corresponding to the recognition result are output in association with each other based on the association relationship to be written into a non-volatile memory.

[0159] In some embodiments, the above method may further include at least one of the following operations.

[0160] For example, in response to the recognition result indicating that the target object does not exist, writing of the point cloud data and / or the corresponding audio data into the non-volatile memory is prohibited.

[0161] For example, in response to the recognition result indicating that the target object does not exist, the point cloud data obtained after the current moment overwrites the point cloud data in the cache where the target object does not exist, and / or the audio data obtained after the current moment overwrites the audio data in the cache where the target object does not exist.

[0162] In some embodiments, the target objects may also be classified, for example, key target objects (such as target objects that are closer to the laser radar), candidate target objects (such as target objects that are closer to the laser radar), etc. This facilitates the classified storage of audio data and / or point cloud data of different types of target objects, so as to further improve the security of point cloud data of key target objects.

[0163] For example, the associated data of the key target object (such as audio data and point cloud data) is stored in the first storage area. The associated data stored in the first storage area cannot be overwritten and can only be cleared by a user deletion operation. If the first storage area is full, the associated data of the key target object can be stored in the second storage area.

[0164] The associated data of the candidate target object can be stored in the second storage area. The associated data stored in the second storage area can be overwritten, but is stored in a first-in-first-out manner. The above method further improves the security and reliability of the associated data.

[0165] The above method is exemplarily described below in conjunction with a circuit or chip for storing data.

[0166] Please also see Fig.11 or Fig.13 Before executing the operation of this method, it is necessary to train the neural network (such as LSTM) with the training data of the target object (such as the point cloud frame) to obtain model parameters, such as generating a parameter table of the trained neural network.

[0167] The laser emitting unit emits laser, the laser receiving unit receives the reflected laser, and the distance calculation unit calculates the distance by the time difference between receiving laser and emitting laser. The microphone collects the sound of the vehicle's surrounding environment and sends it to the audio frame buffer unit for buffering.

[0168] The point cloud frame buffer unit is responsible for storing the point cloud image obtained by the laser radar scanning, and sending the cached point cloud file to the frame counting circuit and assembly module. The audio frame buffer unit sends the cached audio file to the recognition circuit.

[0169] The frame number counting circuit is responsible for calculating the frame number of the point cloud frame and the synchronization information of the audio frame, counting and recording the relationship between the audio frame number and the point cloud frame number, and then sending the statistical results to the timestamp determination subunit and the target object position determination subunit.

[0170] The recognition circuit is responsible for reading the parameter table of the trained neural network, performing audio recognition on the audio frame, and sending the recognition result to the timestamp determination subunit. For example, the recognition result of the LSTM of the recognition circuit can identify n target objects. If the current audio recognition result is that target object 1 (such as the sound of an ambulance) and target object 3 exist in the audio, then the recognition results of target object 1 and target object 3 in the current audio frame are sent to the timestamp determination subunit corresponding to target object 1 and the timestamp determination subunit corresponding to target object 3, respectively.

[0171] The timestamp determination subunit is responsible for querying the timestamp information of the current audio frame where the target object appears in the audio file according to the recognition result and the timestamp information of the current point cloud frame sent by the frame number statistics circuit, and then sending the timestamp information to the corresponding target object position determination subunit.

[0172] The target object position determination subunit is responsible for querying the frame position of the nearest decoded random read point of the current point cloud frame where the target object appears based on the timestamp of the current audio frame and the point cloud file. The frame position of the decoded random read point found is then sent to the de-overlapping determination frame unit.

[0173] The de-overlapping frame determination unit is responsible for removing the overlapping parts of the frame positions of the decoding random read points determined by the target object position determination subunit where multiple target objects appear, obtaining the frame positions of the decoding random read points that need to be intercepted in the end, and sending them to the assembly module.

[0174] The assembly module reads the frame position of the random read point according to the decoding, and sends its content to the encryption unit. The encryption unit is responsible for encrypting the intercepted new audio file and / or point cloud file according to the key, and the key comes from the key generation unit. Among them, the source information of the key generation unit comes from the decimal part of the probability data of the recognition result of the recognition circuit. Then, a hash operation is performed on it to obtain the encrypted key. In this way, a highly random and highly secure key is obtained. The write control unit is responsible for writing the encrypted audio file and / or point cloud file to an external non-volatile memory.

[0175] The laser radar main control chip in this embodiment can realize the function of intelligently screening audio frames and / or point cloud frames of specific objects for storage. This embodiment can be based on edge computing of artificial intelligence. After the laser radar acquires the point cloud, the content is screened, and only the point cloud frames and / or audio frames where valid objects and specific objects appear are retained and stored. This greatly reduces storage requirements and also greatly reduces the difficulty of retrieving key information. In addition, the random key generation and encryption mechanism protects the security and reliability of the screening result files.

[0176] Another aspect of the present application further provides a device for storing data. The device for storing data may include multiple modules, and the function of each module may refer to the function of each module in the circuit for storing data, which will not be described in detail here.

[0177] Fig.16 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application.

[0178] See also Fig.16 The electronic device 1600 may include a memory 1610 and a processor 1620. In addition, the electronic device 1600 may be provided with at least one of a clock generation circuit and a radar.

[0179] The processor 1620 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0180] The memory 1610 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. Among them, ROM may store static data or instructions required by the processor 1620 or other modules of the computer. The permanent storage may be a readable and writable storage device. The permanent storage may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1610 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1610 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0181] The memory 1610 stores executable codes, and when the executable codes are processed by the processor 1620 , the processor 1620 may execute part or all of the above-mentioned methods.

[0182] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0183] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0184] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A circuit for storing data, characterized in that: include: A cache module is configured to cache point cloud data and audio data, wherein the point cloud data includes multiple point cloud frames, and the audio data includes multiple audio frames, and there is a corresponding relationship between the multiple point cloud frames and the multiple audio frames; A recognition result obtaining module, coupled to the cache module, configured to obtain a plurality of recognition results for a plurality of audio frames, wherein the plurality of recognition results include a recognition result indicating the existence of a target object; an assembling module, coupled to the recognition result obtaining module, and configured to determine, in response to the recognition result of the existence of the target object, an association relationship between the target object and the target point cloud frame based on the corresponding relationship; The target point cloud frame is a point cloud frame corresponding to the target audio frame among the multiple point cloud frames, and the target audio frame is an audio frame corresponding to the recognition result of the existence of the target object; An output module is coupled to the recognition result acquisition module and the assembly module respectively, and is configured to output the target point cloud frame and at least one of the following: the recognition result of the target object or the target audio frame in association with the association relationship, so as to be written into a non-volatile memory.

2. The circuit according to claim 1, characterized in that The recognition result obtaining module comprises: A frame number counting circuit, coupled to the cache module, and configured to count the frame numbers of multiple point cloud frames; a target audio frame timestamp determination unit, whose input end is respectively coupled to the frame number counting circuit and the output end of the cache module, and is configured to determine the timestamp of the target audio frame based at least on the frame number of multiple frames of the point cloud frames and the corresponding relationship, wherein the target audio frame includes the audio emitted by the target object; A target point cloud frame position determination unit, whose input end is respectively coupled to the output end of the frame number statistics circuit and the target audio frame timestamp determination unit, is configured to determine the timestamp of the target point cloud frame based on the timestamp of the target audio frame and the corresponding relationship, and to determine the position of the target point cloud frame in multiple frames of the point cloud frames based on the timestamp of the target point cloud frame and the frame number of the multiple frames of the point cloud frames.

3. The circuit according to claim 2, characterized in that The cache module includes an audio frame cache unit; The target audio frame timestamp determination unit comprises: an identification circuit, the input end of which is coupled to the audio frame buffer unit, and is configured to identify each frame of the multiple audio frames to obtain an identification result for each target object in the current audio frame; The target object appearance frame timestamp determination unit is coupled to the frame number statistics circuit and the recognition circuit respectively, and is configured to determine the timestamp of the audio frame where each target object appears based on the frame number of multiple frames of the point cloud frames, the corresponding relationship and the recognition results of each target object in the current audio frame.

4. The circuit according to claim 3, characterized in that The target object appearance frame timestamp determination unit comprises: a plurality of parallel timestamp determination subunits, each of which corresponds to one of the target objects.

5. The circuit according to claim 4, characterized in that The target point cloud frame position determination unit comprises: A plurality of parallel target object position determination subunits, each target object position determination subunit having a one-to-one corresponding and coupled timestamp determination subunit, and coupled with the frame number statistics circuit, are configured to determine the frame position range of the target object in the neighboring point cloud frames of the current point cloud frame from the cached point cloud data, and output the frame position range to the assembly module.

6. The circuit according to claim 5, characterized in that Also includes: The deduplication module is coupled to the plurality of target object position determination subunits respectively, and is configured to perform deduplication processing on the frame position range to obtain a deduplicated frame position range.

7. The circuit according to claim 1, characterized in that The target point cloud frame includes multiple frames, and also includes: The encryption module is coupled to the assembling module and is configured to encrypt multiple frames of the target point cloud frames to obtain multiple encrypted frames of point cloud frames to be stored.

8. The circuit according to claim 7, characterized in that The encryption module comprises: a key generating unit, coupled to the recognition result obtaining module, and configured to generate a secret key based on at least a portion of the numerical value of the probability of the recognition result; An encryption unit is coupled to the key generation unit and is configured to encrypt multiple frames of the target point cloud frames based on the secret key.

9. A chip, characterized in that: include: a distance calculation unit configured to calculate the distance between the object and the mobile platform based on a time difference between a time point of emission of the laser and a time point of reception of the reflected laser, and generate point cloud data; The circuit for storing data according to any one of claims 1 to 8 is configured to output the target point cloud frame in the point cloud data in association with at least one of: the recognition result of the target object or the target audio frame. 10 . A radar, comprising the circuit for storing data according to claim 1 , or the chip according to claim 9 .

11. A method for storing data, characterized in that: include: Cache point cloud data and audio data, wherein the point cloud data includes multiple point cloud frames, and the audio data includes multiple audio frames, and there is a corresponding relationship between the multiple point cloud frames and the multiple audio frames; Obtaining a plurality of recognition results for the plurality of audio frames, the plurality of recognition results including a recognition result indicating the presence of a target object; In response to the recognition result of the existence of the target object, determining an association relationship between the target object and the target point cloud frame based on the corresponding relationship; The target point cloud frame is a point cloud frame corresponding to the target audio frame among the multiple point cloud frames, and the target audio frame is an audio frame corresponding to the recognition result of the existence of the target object; The target point cloud frame and at least one of the following: the recognition result of the target object or the target audio frame are output in association with each other based on the association relationship to be written into a non-volatile memory.

12. The method according to claim 11, characterized in that The plurality of recognition results also include a recognition result in which the target object does not exist, and further include: In response to the recognition result that the target object does not exist, prohibiting the point cloud data and / or the corresponding audio data from being written into the non-volatile memory; or In response to the recognition result that the target object does not exist, the point cloud data obtained after the current moment overwrites the point cloud data of the target object that does not exist in the cache, and / or the audio data obtained after the current moment overwrites the audio data of the target object that does not exist in the cache.

13. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to perform the method according to claim 11 or 12.

Images