Target recognition method, recognition device and electronic equipment based on dynamic vision sensor and laser radar fusion

Abstract

The present application discloses a target recognition method, recognition device and electronic device based on the fusion of dynamic visual sensor and laser radar. The dynamic visual sensor is used to asynchronously capture the moving events in the picture at a very high frame rate to obtain the initial visual image frame set; the initial point cloud image set is captured by the low frequency of the laser radar, and the data is aligned in time by processing the frames of the visual image and the point cloud image synchronously, and the target visual image set and the target point cloud image set are aligned in space. The fusion of time and space consistency takes up less hardware resources, has a high execution rate, and ensures real-time performance. The target point cloud image set after spatial alignment can be projected onto the target visual image set for fusion processing to obtain a visual fusion matrix, and the visual fusion matrix is ​​input into a convolutional neural network based on a storage and computing integrated architecture to perform target recognition to meet the accuracy requirements. The digital part of the storage and computing can greatly reduce the cumulative energy consumption in the matrix multiplication and accumulation operation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a target recognition method, a recognition device and an electronic device based on the fusion of a dynamic vision sensor and a laser radar. Background Art

[0002] In various target recognition scenarios, due to the complex environment, it is particularly important for the recognition device to perceive the surrounding environment information. In order to identify the targets in the surrounding environment, target recognition can be performed through visual perception, radar perception, etc.

[0003] If a DVS (Dynamic Vision Sensor) camera is used for target recognition, it can accurately track the movement of objects or cameras compared to an RGB camera, but the resolution of the DVS camera is low and the accuracy is not high.

[0004] If lidar is used for target recognition, since the working principle of lidar is to detect objects through the emission, reflection and reception of harmless infrared light beams, the point cloud generated by lidar is data that expresses three dimensions in space. However, the effective information of radar point cloud data is too sparse, the information density is not high, and the image integrity is not high. Summary of the invention

[0005] The purpose of this application is to provide a target recognition method, recognition device and electronic device based on the fusion of dynamic vision sensor and laser radar, so as to at least solve the technical problems that the above-mentioned dynamic vision sensor has low resolution and low precision when recognizing targets, and the effective information is too sparse, the information density is not high, and the image integrity is not high when the laser radar recognizes targets. The many technical effects that can be produced by the optional technical solutions among the many technical solutions provided by the present invention are described in detail below.

[0006] To achieve the above objectives, in a first aspect, the present application provides a target recognition method based on the fusion of a dynamic vision sensor and a laser radar, and the target recognition method based on the fusion of a dynamic vision sensor and a laser radar comprises:

[0007] Obtaining an initial visual image frame set of a dynamic visual sensor and an initial point cloud image frame set of a laser radar;

[0008] According to the timestamp, the visual images in the initial visual image frame set and the point cloud images in the initial point cloud image frame set are subjected to frame synchronization processing to obtain a target visual image and a target point cloud image with matching timestamps, thereby obtaining a target visual image set and a target point cloud image set;

[0009] Spatially aligning the target visual image set and the target point cloud image set;

[0010] In the computing unit of the storage-computing integrated architecture, the spatially aligned target visual image is projected onto the target point cloud image to perform fusion processing and matrix processing to obtain a visual fusion matrix;

[0011] In the convolutional neural network with the storage-computing integrated architecture, multiplication and accumulation operations are performed on the visual fusion matrix to perform target recognition and obtain a target recognition result.

[0012] In some embodiments, the target recognition method based on the fusion of dynamic visual sensor and laser radar further includes:

[0013] Extracting a visual timestamp of each initial visual image frame of the initial visual image frame set, and extracting a point cloud timestamp of each initial point cloud image frame of the initial point cloud image frame set;

[0014] Obtaining a first average time delay between the dynamic vision sensor and a host clock, and obtaining a second average time delay between the laser radar and the host clock;

[0015] The visual timestamp is corrected based on the first average time delay to obtain a first corrected timestamp, and the point cloud timestamp is corrected based on the second average time delay to obtain a second corrected timestamp.

[0016] In some embodiments, the visual images in the initial visual image frame set and the point cloud images in the initial point cloud image frame set are subjected to frame synchronization processing according to the timestamps to obtain target visual images and target point cloud images with matching timestamps, thereby obtaining a target visual image set and a target point cloud image set, including:

[0017] Determine a reference frame rate based on the acquisition frame rate of the initial visual image frame set by the dynamic vision sensor and the acquisition frame rate of the initial point cloud image frame set by the laser radar;

[0018] Setting a frame time difference threshold based on the reference frame rate;

[0019] Calculating a timestamp difference based on the first corrected timestamp and the second corrected timestamp;

[0020] Using a sliding window method, if the timestamp difference is less than or equal to the frame time difference threshold, the visual image corresponding to the first corrected timestamp is used as a target visual image matched by the timestamp, and the point cloud image corresponding to the second corrected timestamp is used as a target point cloud image matched by the timestamp;

[0021] Based on the target visual image frame set and the target point cloud image frame set with matching timestamps, a target set including the target visual image frame set and the target point cloud image frame set is updated.

[0022] In some embodiments, determining the reference frame rate based on the acquisition frame rate of the initial visual image frame set by the dynamic vision sensor and the acquisition frame rate of the initial point cloud image frame set by the laser radar includes:

[0023] If the acquisition frame rate of the initial point cloud image frame set by the laser radar is lower than the acquisition frame rate of the initial visual image frame set by the dynamic vision sensor, the acquisition frame rate of the initial point cloud image frame set by the laser radar is used as the reference frame rate.

[0024] In some embodiments, updating the target set including the target visual image frame set and the target point cloud image frame set based on the target visual image frame set and the target point cloud image frame set with matching timestamps comprises:

[0025] The target visual image and the target point cloud image are fused to obtain a current target set; the current target set includes a current target visual image set and a current target point cloud image set;

[0026] The current target set is combined with the historical target set to obtain an updated target set.

[0027] In some embodiments, the target recognition method based on the fusion of dynamic visual sensor and laser radar further includes:

[0028] If the timestamp difference is greater than the frame time difference threshold, the visual image corresponding to the first corrected timestamp is discarded.

[0029] In some embodiments, the target recognition method based on the fusion of dynamic visual sensor and laser radar further includes:

[0030] If the timestamp difference is greater than the frame time difference threshold, searching for a third timestamp and a fourth timestamp corresponding to the visual image; wherein the third timestamp is less than the first correction timestamp, the fourth timestamp is greater than the third timestamp, and the time difference between the third timestamp and the fourth timestamp is at least the acquisition time interval of the dynamic vision sensor;

[0031] Performing interpolation calculation on the third timestamp and the fourth timestamp to obtain a fifth timestamp having a visual image;

[0032] Calculating a difference between the fifth timestamp and the second correction timestamp;

[0033] If the difference between the fifth timestamp and the second corrected timestamp is less than or equal to the frame time difference threshold, the visual image corresponding to the fifth timestamp is used as the target visual image matched by the timestamp, and the point cloud image corresponding to the second corrected timestamp is used as the target point cloud image matched by the timestamp.

[0034] In some embodiments, spatially aligning the target visual image set and the target point cloud image set includes:

[0035] Acquire the visual coordinate system of the dynamic visual sensor, the laser coordinate system of the laser radar, and the pixel coordinates of the dynamic visual sensor when taking a visual image;

[0036] Calculating transformation matrices between the visual coordinate system and the laser coordinate system and the pixel coordinates respectively;

[0037] Based on the number of point clouds of the laser radar, calculate a u coordinate set and a v coordinate set in the pixel coordinates;

[0038] Substitute the u coordinate set, the v coordinate set, the coordinate matrix formed by all n points of the laser radar, and the all-1 matrix into the transformation matrix to obtain a conversion matrix;

[0039] Based on the transformation matrix, the target visual image set and the target point cloud image set are spatially aligned.

[0040] In some embodiments, in the convolutional neural network of the storage-computation integrated architecture, multiplication and accumulation operations are performed on the visual fusion matrix to perform target recognition and obtain a target recognition result, including:

[0041] In the convolutional neural network of the storage-computation integrated architecture, image feature extraction is performed on the visual fusion matrix to obtain an image feature matrix;

[0042] After loading the image feature matrix in parallel as row voltages, multiplying the row voltages by weighted conductances and summing them using currents on columns to generate an output vector;

[0043] Target recognition is performed based on the output vector to obtain a target recognition result.

[0044] In a second aspect, the present application provides an identification device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors are used to execute the one or more computer programs stored in the memory, so that the one or more processors execute a target recognition method based on the fusion of dynamic visual sensors and lidar as described in any one of the first aspects.

[0045] In a third aspect, the present application provides an electronic device, wherein the electronic device comprises the identification device described in the second aspect.

[0046] Implementing one of the above technical solutions of this application has the following advantages or beneficial effects:

[0047] The target recognition method, recognition device and electronic device based on the fusion of dynamic visual sensor and laser radar of the present application use the dynamic visual sensor to asynchronously capture the moving object in the picture at a very high frame rate to obtain the initial visual image frame set; use the low frequency of the laser radar to capture the initial point cloud image set, and realize the temporal alignment of the data by synchronizing the frames of the visual image and the point cloud image, align the target visual image set and the target point cloud image set in space, and the fusion of time and space consistency takes up less hardware resources, has a high execution rate, and ensures real-time performance, so that the spatially aligned target visual image can be projected onto the target point cloud image in the computing unit of the storage and computing integrated architecture to perform fusion processing and matrix processing to obtain a visual fusion matrix. In this way, the information of the movement of the target object in the lens can be obtained, and the high information density brought by multiple frames can be increased, while the power consumption can be greatly reduced. The solution of integrating the dynamic visual sensor and the laser radar device can take into account the advantages of both solutions at the same time, and make up for the defects of the independent operation of the two solutions, so as to improve the recognition effect of the target object.

[0048] Moreover, in the convolutional neural network with the integrated storage and computing architecture, multiplication and accumulation operations are performed on the visual fusion matrix to perform target recognition and obtain target recognition results. The convolutional neural network with the integrated storage and computing architecture is used to replace the traditional computing unit to meet the accuracy requirements. The digital part of the storage and computing can greatly reduce the cumulative energy consumption in the matrix multiplication and accumulation operations and greatly save the chip area. Its analog operation has greater advantages in nonlinear operators and improves the operation speed.

[0049] This application is based on target recognition using dynamic vision sensors and lidars, and has higher recognition efficiency, lower power consumption and latency, and better robustness. It can be highly adaptable and adaptable to various scenarios and site conditions such as cloudy and rainy weather (climate) and changes in tunnel light (road environment). BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. It is obvious that the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0051] Figure 1It is a flowchart of a target recognition method based on the fusion of a dynamic vision sensor and a laser radar according to an embodiment of the present application;

[0052] Figure 2 It is a schematic diagram of a process of matching a target visual image set and a target point cloud image set in an embodiment of the present application;

[0053] Figure 3 It is a schematic diagram of the frame synchronization process of an embodiment of the present application;

[0054] Figure 4 is a schematic diagram of a visual fusion signal according to an embodiment of the present application;

[0055] Figure 5 It is a schematic diagram of the storage-computing integrated architecture of an embodiment of the present application;

[0056] Figure 6 It is a schematic block diagram of a target recognition method based on the fusion of a dynamic vision sensor and a lidar according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, wherein various exemplary embodiments that may be used to implement the present application are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods, and devices that are consistent with some aspects disclosed in the present application as detailed in the attached claims, and other embodiments may also be used, or the embodiments listed herein may be modified in structure and function without departing from the scope and essence of the present application.

[0058] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "plurality" means two or more. The terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0059] In order to illustrate the technical solution described in the present application, a specific embodiment is provided below, and only the parts related to the embodiment of the present application are shown.

[0060] Embodiment 1:

[0061] like Figure 1 As shown, the present application provides a target recognition method based on the fusion of dynamic vision sensor and laser radar, and the target recognition method based on the fusion of dynamic vision sensor and laser radar includes the following steps S10 to S50.

[0062] S10: Acquire an initial visual image frame set of the dynamic visual sensor and an initial point cloud image frame set of the laser radar.

[0063] DVS (Dynamic Vision Sensor) camera, also known as EVS (Event Vision Sensor) event camera, is an imaging (sensor) camera that responds to local brightness changes in the field of view. Unlike traditional RGB cameras, each pixel in the dynamic vision sensor operates independently and asynchronously, generating feedback signals when the brightness changes, otherwise remaining silent. Among them, RGB represents three color channels, namely the colors of the red, green, and blue channels.

[0064] Specifically, each pixel of a dynamic vision sensor stores a reference brightness when capturing a visual image, and constantly compares the reference brightness with the current brightness. If the difference between the reference brightness and the current brightness exceeds a set threshold, the pixel will reset its reference brightness and generate an event Event, which contains a discrete data packet of the pixel position and timestamp, as well as the polarity of the brightness change ("+ / -": brighter / darker). The output of a dynamic vision sensor is an asynchronous data stream driven by changes in scene illumination, denoted by t k At the pixel coordinate The brightness change at is expressed by the following formula:

[0065] ;

[0066] in, is a small time interval. When the brightness change exceeds the preset threshold ±C, an event will be triggered. The brightness change polarity is recorded as p k , brightness enhancement p k =+1, brightness decreases p k = -1, the final event is expressed as .

[0067] Compared with RGB cameras, dynamic vision sensors can respond in microseconds, have a 120dB dynamic range, and have less underexposure / overexposure and motion blur. The independence of pixels allows dynamic vision sensors to handle bright and dim scenes at the same time. This allows it to track objects or camera movements more accurately.

[0068] However, dynamic vision sensors have low resolution and low accuracy.

[0069] The full name of LiDAR is Light Detection and Ranging, also known as optical radar. The working principle of LiDAR is the echo time (Time of Flight, abbreviated as TOF) measurement method. It analyzes the reflected energy size, amplitude, frequency and phase of the reflected spectrum on the surface of the target object, outputs a point cloud, and thus presents the precise three-dimensional structure information of the target object. The working principle of LiDAR is to detect objects by emitting, reflecting and receiving infrared light beams that are harmless to humans and animals. The speed of light is LightSpeed, and the echo time TOF is T f , the distance measurement result D is calculated using the following formula:

[0070] ;

[0071] The point cloud generated by the LiDAR device is a three-dimensional data in the space, which is expressed in the form of depth map (i.e. the distance measurement result, which expresses the distance between the object and the camera in grayscale). Compared with the two-dimensional image captured by the RGB camera, the three-dimensional image can achieve background decoupling of natural objects with the help of the third dimension information. Point cloud data is the most common and basic three-dimensional model.

[0072] However, the disadvantages of conventional commercial LiDAR solutions are that effective information is too sparse, information density is not high, and image integrity is not high. Moreover, the power consumption of LiDAR already accounts for a considerable proportion in the hardware system of image recognition, and high power consumption is also a major drawback of LiDAR applications.

[0073] Based on the advantages of dynamic vision sensors in accurately tracking objects and the advantages of lidar in presenting accurate three-dimensional structural information of target objects, if the visual images taken by the dynamic vision sensor and the point cloud images detected by the lidar are fused, it can not only make up for the defects of the dynamic vision sensor and the lidar, but also effectively combine the advantages of the two, enrich the final output image information, improve the frame rate, and increase the information density of a single frame, thereby effectively improving the accuracy of target recognition.

[0074] In this application, the initial visual image frame set of the dynamic visual sensor is represented by D, which is D={d 1 ,d 2 ,……,d m}.

[0075] The initial point cloud image frame set of the laser radar is represented by L, which is L={l 1 ,l 2 ,……,l n}. Where l is calculated by formula 2.

[0076] In addition, the visual timestamp of the visual image collected by the dynamic visual sensor is Td={t c1 , t c2 ,……,t cm}, the point cloud timestamp of the initial point cloud image acquired by the lidar is Tl={t l1 ,t l2 ,……,t ln}.

[0077] When identifying a target, since the dynamic vision sensor can asynchronously capture moving objects in the picture at an extremely high frame rate, the subsequent signal fusion solution can fuse the relevant information with the point cloud image captured by the lidar at a low frequency. This can not only obtain information about the movement of objects in the lens, but also increase the high information density brought by multiple frames, while greatly reducing power consumption. It can take into account the advantages of visual images that can identify bright and dim scenes to accurately track targets and the three dimensions of point cloud images for accurate ranging, and make up for the defects of dynamic vision sensors and lidar when used separately in target identification.

[0078] S20: According to the timestamp, the visual images in the initial visual image frame set and the point cloud images in the initial point cloud image frame set are subjected to frame synchronization processing to obtain a target visual image and a target point cloud image with matching timestamps, thereby obtaining a target visual image set and a target point cloud image set.

[0079] Since the dynamic visual sensor and LiDAR are not completely synchronized with the host clock, there is a delay. In order to accurately calculate the visual timestamp of each frame's initial visual image and the point cloud timestamp of each frame's initial point cloud image, such as Figure 2 As shown, firstly, the visual timestamp of each frame of the initial visual image and the point cloud timestamp of each frame of the initial point cloud image need to be corrected, and then frame synchronization is performed. In some embodiments, the target recognition method based on the fusion of dynamic visual sensor and laser radar can also include steps S21 to S23.

[0080] S21: extracting a visual timestamp of each frame of the initial visual image of the initial visual image frame set, and extracting a point cloud timestamp of each frame of the initial point cloud image of the initial point cloud image frame set;

[0081] S22: Obtain a first average time delay between the dynamic vision sensor and a host clock, and obtain a second average time delay between the laser radar and the host clock;

[0082] S23: Correcting the visual timestamp based on the first average time delay to obtain a first corrected timestamp, and correcting the point cloud timestamp based on the second average time delay to obtain a second corrected timestamp.

[0083] Specifically, the host clock is the clock of the central control unit (host) of the identification device or the unified clock of the server. Usually, there is a delay due to the signal propagation delay caused by the physical distance between the sensor and the host, or due to the inherent delay in the sensor processing events, so compensation needs to be made.

[0084] The visual timestamp of each frame of initial visual image collected by the dynamic visual sensor is Td={t c1 , t c2 ,……,t cm}, the point cloud timestamp of the initial point cloud image acquired by the lidar is Tl={t l1 ,t l2 ,……,t ln}, the first average delay between the dynamic vision sensor and the host clock is δ D , the second average delay δ between the lidar and the host clock L .

[0085] When correcting the visual timestamp, the first average delay can be subtracted from the visual timestamp to obtain the first corrected timestamp, i.e., Td-δ D .

[0086] Similarly, when correcting the point cloud timestamp, the second average delay can be subtracted from the point cloud timestamp to obtain the second corrected timestamp, i.e., Tl-δ L .

[0087] It is understandable that the timestamp correction may be performed before frame synchronization or after frame synchronization, and the present invention is not limited thereto.

[0088] By correcting the timestamp, the accuracy of the analysis and decision-making of the time-based visual images and point cloud images is guaranteed, which is conducive to more accurately acquiring the visual images and point cloud images with matching timestamps.

[0089] In some implementations, step S20 may include steps S201 to S205.

[0090] S201: determining a reference frame rate based on a frame rate at which the dynamic vision sensor acquires the initial visual image frame set and a frame rate at which the laser radar acquires the initial point cloud image frame set;

[0091] S202: Setting a frame time difference threshold based on the reference frame rate;

[0092] S203: Calculating a timestamp difference based on the first corrected timestamp and the second corrected timestamp;

[0093] S204: using a sliding window method, if the timestamp difference is less than or equal to the frame time difference threshold, taking the visual image corresponding to the first corrected timestamp as a target visual image for timestamp matching, and taking the point cloud image corresponding to the second corrected timestamp as a target point cloud image for timestamp matching;

[0094] S205: Based on the target visual image frame set and the target point cloud image frame set with matching timestamps, updating the target set including the target visual image frame set and the target point cloud image frame set.

[0095] Specifically, since there is a difference between the acquisition frame rate of the dynamic vision sensor for the initial visual image frame set and the acquisition frame rate of the laser radar for the initial point cloud image frame set, in order to find the initial visual image and initial point cloud image with the closest timestamps, the reference frame rate is first determined based on the acquisition frame rate of the dynamic vision sensor for the initial visual image frame set and the acquisition frame rate of the laser radar for the initial point cloud image frame set.

[0096] When determining the reference frame rate, a sensor with a lower frame rate is usually selected. In practice, it is highly likely that the acquisition frame rate of the initial point cloud image frame set by the laser radar is lower than the acquisition frame rate of the initial visual image frame set by the dynamic vision sensor. The lower the acquisition frame rate, the higher the acquisition frequency and the smaller the time interval. Therefore, if the acquisition frame rate of the initial point cloud image frame set by the laser radar is lower than the acquisition frame rate of the initial visual image frame set by the dynamic vision sensor, the acquisition frame rate of the initial point cloud image frame set by the laser radar is taken as the reference frame rate, denoted as f min .

[0097] On the contrary, in some special cases, such as when the brightness of the environment changes frequently, resulting in a smaller time interval for the dynamic vision sensor to collect data, that is, a smaller collection frame rate, the collection frame rate of the initial point cloud image frame set by the laser radar is greater than the collection frame rate of the initial visual image frame set by the dynamic vision sensor. In this case, the collection frame rate of the initial visual image frame set by the dynamic vision sensor can be used as the reference frame rate f min .

[0098] In determining the reference frame rate f min After that, based on the reference frame rate f min Set the frame time difference threshold, denoted as δ, for example, the frame time difference threshold Among them, when the acquisition frame rate of the dynamic vision sensor is used as the reference frame rate, It is the time interval for the dynamic vision sensor to collect visual images. Taking half of the time interval for the dynamic vision sensor to collect visual images as a reference, the frame time difference threshold has a higher accuracy.

[0099] Then, based on the first corrected timestamp and the second corrected timestamp, a timestamp difference is calculated. di Indicates that the second correction timestamp is T lj Indicates that the timestamp difference Δt is the first correction timestamp T di and the second correction timestamp T lj The absolute value of the difference between is expressed by Formula 3:

[0100] Formula 3;

[0101] After obtaining the timestamp difference Δt, the sliding window method is used. If the timestamp difference Δt is less than or equal to the frame time difference threshold δ, it means that the first corrected timestamp T di The corresponding visual image and the second correction time stamp T lj If the corresponding point cloud image frames meet the fusion conditions and belong to the same scene, it is determined that the first correction timestamp and the second correction timestamp match each other. Then, the corresponding visual image and point cloud image are the visual image and point cloud image with matching timestamps, so that the first correction timestamp T di The corresponding visual image is used as the target visual image, and the second correction time stamp T lj The corresponding point cloud image is used as the target point cloud image. Otherwise, the first correction timestamp and the second correction timestamp do not match, and the corresponding visual image and point cloud image are visual images and point cloud images with mismatched timestamps, and no fusion is required.

[0102] like Figure 3 As shown, Figure 3 A simplified diagram showing frame synchronization. Figure 3 In the example, the laser radar point cloud image is used as a reference, and the sliding window is used to find the visual image frames that satisfy the corresponding point cloud images of each frame, and to find a point cloud image (target point cloud image) corresponding to three visual images (target visual images) in the frame. In addition, the timestamps of the target point cloud image and the corresponding target visual image are closest, indicating that the timestamps of the target point cloud image and the target visual image match.

[0103] Since the timestamps of the target point cloud image and the corresponding target visual image are closest, it means that the dynamic visual sensor and the lidar have captured the same scene, and the fusion conditions are met, so fusion can be performed.

[0104] In some embodiments, the target recognition method based on the fusion of dynamic visual sensor and laser radar may also include:

[0105] If the timestamp difference is greater than the frame time difference threshold, the visual image corresponding to the first corrected timestamp is discarded.

[0106] Specifically, if the timestamp difference Δt is greater than the frame time difference threshold δ, it means that the visual image corresponding to the first correction timestamp and the point cloud image corresponding to the second correction timestamp do not belong to the same scene, that is, the image frames of the two sensors do not match and do not need to be fused. At this time, the visual image corresponding to the first correction timestamp is discarded. It can be understood that the discard here only means that the visual image is not fused.

[0107] In some implementations, the target recognition method based on the fusion of dynamic visual sensor and laser radar further includes:

[0108] If the timestamp difference is greater than the frame time difference threshold, searching for a third timestamp and a fourth timestamp corresponding to the visual image; wherein the third timestamp is less than the first correction timestamp, the fourth timestamp is greater than the third timestamp, and the time difference between the third timestamp and the fourth timestamp is at least the acquisition time interval of the dynamic vision sensor;

[0109] Performing interpolation calculation on the third timestamp and the fourth timestamp to obtain a fifth timestamp having a visual image;

[0110] Calculating a difference between the fifth timestamp and the second correction timestamp;

[0111] If the difference between the fifth timestamp and the second corrected timestamp is less than or equal to the frame time difference threshold, the visual image corresponding to the fifth timestamp is used as the target visual image matched by the timestamp, and the point cloud image corresponding to the second corrected timestamp is used as the target point cloud image matched by the timestamp.

[0112] Specifically, if the timestamp difference Δt is greater than the frame time difference threshold δ, it means that the first corrected timestamp has no corresponding visual image, but corresponds to a point cloud image. At this time, some linear motion equations need to be updated. For example, the two timestamps corresponding to the visual image can be found, which are the third timestamp and the fourth timestamp. There is at least a visual image between these two timestamps. Then, interpolation calculations are performed on these two timestamps to obtain the fifth timestamp with the visual image. Then, referring to the relationship between the first corrected timestamp, the second corrected timestamp and the frame time difference threshold, the matching visual image and point cloud image are found.

[0113] Based on the target visual image frame set and the target point cloud image frame set whose timestamps match, a target set including the target visual image frame set and the target point cloud image frame set is updated, the target visual image frame set and the target point cloud image frame set are sets that meet the fusion condition, and the target visual image frame set is represented by D i Indicates that the target point cloud image frame set is represented by L j Denotes the combination of the target visual image frame set and the target point cloud image frame set, using (D i , L j )express.

[0114] In some embodiments, updating a target set including a target visual image frame set and a target point cloud image frame set based on the target visual image frame set and the target point cloud image frame set with matching timestamps may include:

[0115] The target visual image and the target point cloud image are fused to obtain a current target set; the current target set includes a current target visual image set and a current target point cloud image set;

[0116] The current target set is combined with the historical target set to obtain an updated target set.

[0117] Specifically, since both the initial visual image set and the initial point cloud image set are multi-frame images, multiple rounds of recognition will be performed according to the sliding window method to identify all targets. Therefore, before the sliding window processing, the historical target set is an empty set. When the target set is recognized for the first time, it is (D i , L j ) At this time, i and j are the i-th frame or j-th frame of the visual image and point cloud image recognized in the first round, respectively. Then, the target set is (D i , L j ). When the target set is identified multiple times in the following, the target visual image set and the target point cloud image set determined each time are the current target visual image set and the current target point cloud image set respectively. Then, the updated target set can be expressed by the following formula:

[0118] ;

[0119] in, represents the updated target set, represents the historical target set, represents the current target set that meets the fusion conditions, Represents the merger of the historical target set and the current target set.

[0120] By performing frame synchronization processing on the visual images in the initial visual image frame set and the point cloud images in the initial point cloud image frame set, the spatiotemporal alignment of data is achieved, thereby ensuring the spatiotemporal consistency of the target visual image frame set and the target point cloud image frame set.

[0121] S30: spatially aligning the target visual image set and the target point cloud image set.

[0122] In order to enable the target point cloud image to be projected onto the target visual image for fusion, corresponding coordinate transformation is also required to ensure the alignment of the target visual image and the target point cloud image in space.

[0123] In some implementations, step S30 may include:

[0124] Acquire the visual coordinate system of the dynamic visual sensor, the laser coordinate system of the laser radar, and the pixel coordinates of the dynamic visual sensor when taking a visual image;

[0125] Calculating transformation matrices between the visual coordinate system and the laser coordinate system and the pixel coordinates respectively;

[0126] Based on the number of point clouds of the laser radar, calculate a u coordinate set and a v coordinate set in the pixel coordinates;

[0127] Substitute the u coordinate set, the v coordinate set, the coordinate matrix formed by all n points of the laser radar, and the all-1 matrix into the transformation matrix to obtain a conversion matrix;

[0128] Based on the transformation matrix, the target visual image set and the target point cloud image set are spatially aligned.

[0129] Specifically, when spatially aligning and fusing the target visual image and the target point cloud image, corresponding coordinate transformation is required.

[0130] The visual coordinate system of the dynamic vision sensor is: (x d ,y d ,z d ), the laser coordinate system of the laser radar is (x l ,y l ,z l ), the pixel coordinates when the dynamic vision sensor captures the visual image are (u,v,1).

[0131] Then, the transformation matrices between the visual coordinate system and the laser coordinate system and the pixel coordinates are calculated using the following formula:

[0132] .

[0133] Among them, s represents the scale factor, M1 represents the internal parameter matrix, which is the parameter related to the internal optical axis imaging of the dynamic sensor, M2 represents the external parameter matrix of the dynamic sensor, and R represents the rotation factor. represents the translation vector. Since the image does not contain z-axis information, that is, the z-axis is always 0, the above formula can be simplified to the following formula:

[0134] ;

[0135] Based on the number of point clouds n of the laser radar, calculate the u coordinate set in the pixel coordinates , v coordinate set , the u coordinate set , v coordinate set , the coordinate matrix composed of all n points of the laser radar And the matrix of all 1s Substitute the transformation matrix to obtain the conversion matrix T transform . Where T stands for transpose.

[0136] And the transformation matrix T transform Finally, the least squares method is used to calculate:

[0137]

[0138] So as to obtain the transformation matrix T transform Moreover, based on the transformation matrix, the target point cloud image of the lidar and the target visual image of the dynamic sensor can be placed in a unified coordinate system, thereby achieving the spatial unification of the target visual image and the target point cloud image.

[0139] It is understandable that the unification of the visual image and the point cloud image in space can further refer to the current conventional spatial alignment of images and point clouds, which will not be elaborated here.

[0140] After the target visual image and the target point cloud image are aligned in time and space, the target visual image can be projected onto the target point cloud image to form fused data. Due to the low frame rate of the LiDAR, the fused single-frame image has better sparsity, which can be better compatible with the convolutional neural network with a storage-computing integrated architecture, improve the frame rate of the fused data, and provide richer data. Moreover, since the LiDAR carries depth information, the fused data can also make up for the low resolution of the dynamic vision sensor. This fusion method can achieve the effect that can only be achieved by simply improving the properties of the LiDAR device itself (resulting in higher power consumption) with lower power consumption.

[0141] S40: In the computing unit of the storage-computing integrated architecture, the spatially aligned target visual image is projected onto the target point cloud image to perform fusion processing and matrix processing to obtain a visual fusion matrix.

[0142] The fused signal is Figure 4 As shown in the figure, after the initial point cloud image frame set is aligned with the initial visual image frame set through temporal and spatial consistency, a visual fusion signal is obtained, and the visual fusion signal is transmitted to the storage and computing integrated computing unit for processing, which can improve the energy efficiency ratio by more than 5 times compared with the traditional computing mode, and its smaller area can be more flexible and convenient in terminal deployment and commercial applications. For the storage and computing core, it is only necessary to store the trained model parameters in the storage and computing core in advance, and then input the visual fusion matrix to obtain the corresponding output result after processing.

[0143] S105: In the convolutional neural network of the storage-computing integrated architecture, multiplication and accumulation operations are performed on the visual fusion matrix to perform target recognition and obtain a target recognition result.

[0144] The convolutional neural network of the storage and computing integrated architecture of this application is as follows Figure 5 shown.

[0145] exist Figure 5 In the example, the input includes columns 0 to N, an analog-to-digital converter that converts image features into voltage, and an operation matrix , when calculating, image features are extracted from the visual fusion matrix with better sparsity to obtain an image feature matrix, and the image feature matrix is ​​loaded in parallel as a row voltage ( Figure 5 In the WL direction), the multiplication is performed in an analog manner, that is, the input row voltage is multiplied by the weight conductance and summed with the current on the column ( Figure 5 BL direction in the image), generate an output vector; perform target recognition based on the output vector to obtain a target recognition result.

[0146] When obtaining the image feature matrix, traditional computing devices cannot efficiently utilize the characteristics of matrix sparsification itself. Therefore, by using a convolutional neural network based on a storage and computing integrated architecture to replace the traditional computing unit, under the accuracy requirements of the current mainstream technical solutions, the digital part of the storage and computing can greatly reduce the energy consumption of multiplication operations in matrix multiplication and accumulation operations, and the analog part can significantly reduce the accumulation energy consumption in matrix multiplication and accumulation operations, and greatly save chip area. In addition, analog operations have a natural advantage in the operation of nonlinear operators. The fused sparse image feature matrix is ​​efficiently calculated through analog-digital mixed storage and computing, which greatly reduces energy consumption and improves computing speed.

[0147] like Figure 6 As shown, Figure 6It is a flowchart of the data fusion and processing of this application, including modules such as image input, data time alignment, coordinate (space) alignment, data fusion, recognition in convolutional neural network and output of recognition results.

[0148] In the embodiment of the present application, a dynamic visual sensor is used to asynchronously capture moving objects in the picture at an extremely high frame rate to obtain an initial visual image frame set; the initial point cloud image set is captured by the low frequency of the laser radar, and the data is aligned in time by synchronizing the frames of the visual image and the point cloud image, and the target visual image set and the target point cloud image set are aligned in space. The fusion of time and space consistency takes up less hardware resources, has a high execution rate, and ensures real-time performance. Therefore, the spatially aligned target visual image can be projected onto the target point cloud image in the computing unit of the storage and computing integrated architecture to perform fusion processing and matrix processing to obtain a visual fusion matrix. In this way, the information on the movement of the target object in the lens can be obtained, and the high information density brought by multiple frames can be increased, while greatly reducing power consumption. The solution of integrating the dynamic visual sensor and the laser radar device can take into account the advantages of both solutions at the same time, make up for the defects of the independent operation of the two solutions, and improve the recognition effect of the target.

[0149] Moreover, in the convolutional neural network with the integrated storage and computing architecture, multiplication and accumulation operations are performed on the visual fusion matrix to perform target recognition and obtain target recognition results. The convolutional neural network with the integrated storage and computing architecture is used to replace the traditional computing unit to meet the accuracy requirements. The digital part of the storage and computing can greatly reduce the cumulative energy consumption in the matrix multiplication and accumulation operations and greatly save the chip area. Its analog operation has greater advantages in nonlinear operators and improves the operation speed.

[0150] This application is based on target recognition using dynamic vision sensors and lidars, and has higher recognition efficiency, lower power consumption and latency, and better robustness. It can be highly adaptable and adaptable to various scenarios and site conditions such as cloudy and rainy weather (climate) and changes in tunnel light (road environment).

[0151] Embodiment 2:

[0152] Those skilled in the art can understand that all or part of the features / steps of the above-mentioned method embodiments can be implemented by methods, data processing systems or computer programs, and these features can be implemented without hardware, entirely by software, or by a combination of hardware and software. The aforementioned computer program can be stored in one or more computer-readable storage media, and the storage medium stores a computer program. When the computer program is executed (such as a processor), the steps of the above-mentioned target recognition method embodiment based on the fusion of dynamic visual sensors and laser radar are executed.

[0153] The aforementioned storage media that can store program codes include: static hard disks, solid-state hard disks, random access memories (SRAM), electrically erasable programmable read-only memories (EEPROM), erasable programmable read-only memories (EPROM), programmable read-only memories (PROM), read-only memories (ROM), optical storage devices, magnetic storage devices, flash memories, magnetic disks or optical disks and / or combinations of the above devices, that is, they can be implemented by any type of volatile or non-volatile storage devices or combinations thereof.

[0154] Embodiment three:

[0155] The present application also provides an identification device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors are used to execute the one or more computer programs stored in the memory, so that the one or more processors execute a target recognition method based on the fusion of a dynamic visual sensor and a lidar as described in any one of Embodiment 1.

[0156] The recognition device of the present invention may be a chip, and the recognition device provided by the present invention may be, for example, a large model recognition device. When the recognition device is a chip, each module in the chip may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in or independent of the processor in the computing device in hardware form, or may be stored in the memory in the computing device in software form, so that the processor can call and execute the operations corresponding to the above modules. The invention can be applied to fields such as artificial intelligence and metaverse that require high precision, high reliability, low power consumption, and low area cost computing power, artificial intelligence training or reasoning chip products, autonomous driving chips, VR chips, robot built-in chips, and deep learning applications of parallel computing. It can be applied to fields such as artificial intelligence and metaverse that require high precision, high reliability, low power consumption, and low area cost computing power, such as artificial intelligence training or reasoning chips, autonomous driving chips, VR chips, robot built-in chips, and other fields.

[0157] The present application also provides an electronic device, the electronic device including the identification device. The electronic device can be a series of electronic devices such as a humanoid robot, a smart phone, a tablet computer, a wearable electronic device, a smart home electronic product, and an industrial device.

[0158] Since the implementation of the electronic device is described in detail in the first embodiment, it will not be repeated here.

[0159] The above description is only the preferred embodiment of the present invention. It is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the protection scope of the present invention.

Claims

1. A target recognition method based on the fusion of dynamic visual sensor and laser radar, characterized in that: include: Obtaining an initial visual image frame set of a dynamic visual sensor and an initial point cloud image frame set of a laser radar; According to the timestamp, the visual images in the initial visual image frame set and the point cloud images in the initial point cloud image frame set are subjected to frame synchronization processing to obtain a target visual image and a target point cloud image with matching timestamps, thereby obtaining a target visual image set and a target point cloud image set; Spatially aligning the target visual image set and the target point cloud image set; In the computing unit of the storage-computing integrated architecture, the spatially aligned target visual image is projected onto the target point cloud image to perform fusion processing and matrix processing to obtain a visual fusion matrix; wherein the fused single-frame image has sparseness; In the convolutional neural network with the integrated storage and computing architecture, the visual fusion matrix is ​​multiplied and accumulated to perform target recognition and obtain a target recognition result; wherein the digital part of the storage and computing reduces the multiplication operation energy consumption in the matrix multiplication and accumulation operation, and the analog part reduces the accumulation energy consumption in the matrix multiplication and accumulation operation; The target recognition method based on the fusion of dynamic visual sensor and laser radar also includes: Extracting a visual timestamp of each initial visual image frame of the initial visual image frame set, and extracting a point cloud timestamp of each initial point cloud image frame of the initial point cloud image frame set; Obtaining a first average time delay between the dynamic vision sensor and a host clock, and obtaining a second average time delay between the laser radar and the host clock; Correcting the visual timestamp based on the first average time delay to obtain a first corrected timestamp, and correcting the point cloud timestamp based on the second average time delay to obtain a second corrected timestamp; The method of performing frame synchronization processing on the visual images in the initial visual image frame set and the point cloud images in the initial point cloud image frame set according to the timestamps to obtain target visual images and target point cloud images with matching timestamps, thereby obtaining a target visual image set and a target point cloud image set, includes: Determine a reference frame rate based on the acquisition frame rate of the initial visual image frame set by the dynamic vision sensor and the acquisition frame rate of the initial point cloud image frame set by the laser radar; Setting a frame time difference threshold based on the reference frame rate; Calculating a timestamp difference based on the first corrected timestamp and the second corrected timestamp; Using a sliding window method, if the timestamp difference is less than or equal to the frame time difference threshold, the visual image corresponding to the first corrected timestamp is used as a target visual image matched by the timestamp, and the point cloud image corresponding to the second corrected timestamp is used as a target point cloud image matched by the timestamp; Based on the target visual image frame set and the target point cloud image frame set whose timestamps match, updating the target set including the target visual image frame set and the target point cloud image frame set; In the convolutional neural network of the storage-computation integrated architecture, multiplication and accumulation operations are performed on the visual fusion matrix to perform target recognition and obtain a target recognition result, including: In the convolutional neural network of the storage-computation integrated architecture, image feature extraction is performed on the visual fusion matrix to obtain an image feature matrix; After the image feature matrix is ​​loaded in parallel as row voltages, the row voltages are multiplied by weighted conductances and summed using currents on columns to generate an output vector; Target recognition is performed based on the output vector to obtain a target recognition result.

2. The target recognition method based on the fusion of dynamic visual sensor and laser radar according to claim 1 is characterized in that: The determining of the reference frame rate based on the acquisition frame rate of the initial visual image frame set by the dynamic vision sensor and the acquisition frame rate of the initial point cloud image frame set by the laser radar comprises: If the acquisition frame rate of the initial point cloud image frame set by the laser radar is lower than the acquisition frame rate of the initial visual image frame set by the dynamic vision sensor, the acquisition frame rate of the initial point cloud image frame set by the laser radar is used as the reference frame rate.

3. The target recognition method based on the fusion of dynamic visual sensor and laser radar according to claim 1 is characterized in that: The updating of the target set including the target visual image frame set and the target point cloud image frame set based on the target visual image frame set and the target point cloud image frame set that match the timestamps comprises: The target visual image and the target point cloud image are fused to obtain a current target set; the current target set includes a current target visual image set and a current target point cloud image set; The current target set is combined with the historical target set to obtain an updated target set.

4. The target recognition method based on the fusion of dynamic visual sensor and laser radar according to claim 1 is characterized in that: The target recognition method based on the fusion of dynamic visual sensor and laser radar also includes: If the timestamp difference is greater than the frame time difference threshold, the visual image corresponding to the first corrected timestamp is discarded.

5. The target recognition method based on the fusion of dynamic visual sensor and laser radar according to claim 1 is characterized in that: The target recognition method based on the fusion of dynamic visual sensor and laser radar also includes: If the timestamp difference is greater than the frame time difference threshold, searching for a third timestamp and a fourth timestamp corresponding to the visual image; wherein the third timestamp is less than the first correction timestamp, the fourth timestamp is greater than the third timestamp, and the time difference between the third timestamp and the fourth timestamp is at least the acquisition time interval of the dynamic vision sensor; Performing interpolation calculation on the third timestamp and the fourth timestamp to obtain a fifth timestamp having a visual image; Calculating a difference between the fifth timestamp and the second correction timestamp; If the difference between the fifth timestamp and the second corrected timestamp is less than or equal to the frame time difference threshold, the visual image corresponding to the fifth timestamp is used as the target visual image matched by the timestamp, and the point cloud image corresponding to the second corrected timestamp is used as the target point cloud image matched by the timestamp.

6. The target recognition method based on the fusion of dynamic visual sensor and laser radar according to claim 1 is characterized in that: The spatially aligning the target visual image set and the target point cloud image set comprises: Acquire the visual coordinate system of the dynamic visual sensor, the laser coordinate system of the laser radar, and the pixel coordinates of the dynamic visual sensor when taking a visual image; Calculating transformation matrices between the visual coordinate system and the laser coordinate system and the pixel coordinates respectively; Based on the number of point clouds of the laser radar, calculate a u coordinate set and a v coordinate set in the pixel coordinates; Substitute the u coordinate set, the v coordinate set, the coordinate matrix formed by all n points of the laser radar, and the all-1 matrix into the transformation matrix to obtain a conversion matrix; Based on the transformation matrix, the target visual image set and the target point cloud image set are spatially aligned.

7. An identification device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, and one or more processors for executing one or more computer programs stored in the memory, so that one or more processors execute a target recognition method based on the fusion of a dynamic visual sensor and a lidar as described in any one of claims 1 to 6.

8. An electronic device, characterized in that: The electronic device comprises the identification device according to claim 7.

Images