Modal fusion method based on event measurement function

Abstract

The present invention belongs to the field of image processing technology, and in particular to a modal fusion method based on event metric function. It includes: S1: using a DAVIS event camera to collect images of a target to be detected, and obtaining an event stream and a color image; S2: calculating the measurement value of each pixel point of the event stream, and performing an aggregation operation on the calculated measurement value of each pixel point to obtain an event tensor; S3: constructing an event integral model based on the event tensor and the color image, and obtaining a potential frame using the event integral model; S4: fusing the event tensor with the potential frame to obtain a fused image with a time series. The present invention improves the image quality of the fused image by adopting a multi-channel connection and sharpening filter operation mode.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a modal fusion method based on event metric function. Background Art

[0002] The event stream data generated by traditional event cameras mostly focuses on the contour information of objects, while ignoring the details and texture information of objects. Therefore, traditional event image-based detection tasks are prone to missed detection or false detection. The tasks related to autonomous driving focus on detection scenes such as pedestrians and cars, but the current detection accuracy is low, especially for target detection in complex scenes. In complex scenes, the RGB images or grayscale images generated by traditional standard cameras will have motion blur, overexposure and distortion, which greatly reduces the detection performance of target detection tasks in this environment. Summary of the invention

[0003] In view of this, the present invention aims to provide a modal fusion method based on event metric function to solve the loss of temporal granularity easily caused by common event image stacking or voxel aggregation. The present invention can retain all the spatiotemporal information and temporal granularity of the original event sequence to the greatest extent, which is of great significance to the practical application of event cameras in high-speed scenes.

[0004] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0005] A modal fusion method based on event measurement function specifically includes the following steps:

[0006] S1: Use the DAVIS event camera to collect images of the target to be detected and obtain event streams and color images;

[0007] S2: Calculate the measurement value of each pixel point in the event stream, and aggregate the calculated measurement value of each pixel point to obtain the event tensor;

[0008] S3: construct an event integration model based on the event tensor and the color image, and obtain the latent frame using the event integration model;

[0009] S4: Fuse the event tensor with the latent frame to obtain a fused image with time series.

[0010] Furthermore, in step S1, the triggering formula of the event flow is:

[0011] ;

[0012] in, is the nth event, is the position coordinate of the nth event on the image sensor of the event camera, is the timestamp information of the nth event, is the polarity of the nth event, is a four-dimensional real vector space;

[0013] The calculation formula for color image is:

[0014] ;

[0015] in, is the color image produced during the exposure time, ( ) is the exposure time, T is the exposure time for generating a color image, is the potential frame, is the pixel on the image sensor, is the middle moment of the exposure time used to generate the color image.

[0016] Furthermore, step S2 specifically includes the following steps:

[0017] S21: Calculate the measurement value of each pixel in the event stream based on the kernel function :

[0018] ;

[0019]

[0020] in, is the Dirac function, describing the impulse properties of the event stream, k is the kernel function, is the mathematical expression of the nth event, The measurement function assigned to the nth event, and They are Dot and The burst impulse response at time and The kernel function is used to measure Dot and The degree of impulse response at the moment;

[0021] S22: Aggregate the calculated measurement values ​​of each pixel to obtain the event tensor:

[0022] ;

[0023] in, is the event tensor, is an aggregate function.

[0024] Furthermore, step S3 specifically includes the following steps:

[0025] S31: Constructing event integration model based on event tensor and color image:

[0026] ;

[0027] in, To take the logarithm of the latent frame, To take the logarithm of a color image;

[0028] S32: Calculating latent frames using event integration model :

[0029] ;

[0030] Among them, c is a hyperparameter, are potential frames at different times f.

[0031] Furthermore, step S4 specifically includes the following steps:

[0032] S41: The mixed image is obtained by fusing the event tensor and the latent frame as follows: :

[0033] ;

[0034] S42: assigning the mixed image to the R channel and the G channel, and assigning the complete latent frame to the B channel to obtain a fused image with a time series.

[0035] Compared with the prior art, the invention can achieve the following beneficial effects:

[0036] (1) The present invention creates the modal fusion method based on event metric function, which fuses the event stream and the color image, and highlights the contour information of the target object. At the same time, the present invention can generate data of different forms on the same image by randomly selecting the metric function. The image fusion method proposed by the present invention has a simple and independent fusion mode, which can be applied to a simple downstream network or used alone for subsequent tasks.

[0037] (2) The present invention creates the modal fusion method based on event metric function, which reduces the blur of the fused image by sharpening the latent frame. In the process of image fusion, the present invention improves the image quality of the fused image by adopting multi-channel connection and sharpening filtering operation.

[0038] (3) The present invention creates the modal fusion method based on the event metric function, which enables the latent frame and the event tensor to share the same measurement function and aggregation function. In the process of image fusion, the event tensor only needs to be calculated once, thereby reducing the computational cost of image fusion. The present invention obtains a fused image with better image quality and clearer image by analyzing the essential information attributes of the event tensor and the color image. The present invention solves the image defects in complex scenes. Due to the application of two modal data, the event tensor and the color image are complementary to each other to a certain extent, and the lack of targets in low light and low saturation conditions is solved. At the same time, the target blurring under high-speed motion is also solved, and no major changes are produced in conventional scenes. It is suitable for downstream tasks in various scenes. The generation rate of the fused image of the present invention is relatively fast, and it can be applied to the training process of deep neural networks. Since the fused image has image details at different angles, the neural network can break through the performance bottleneck. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:

[0040] Figure 1 A flow chart of a modal fusion method based on event metric function described in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0042] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0043] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0044] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0045] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0046] The present invention analyzes the two modal data generated by the DAVIS event camera (model DAVIS346), namely, color image and event stream, from a mechanism perspective, and constructs an event tensor by assigning metric attributes to each event point itself. At the same time, a latent frame image is constructed by filtering the color image, and finally the event tensor and the latent frame are integrated to form a fused image. This fused image has the advantages of both event stream and color image modalities, including texture details and strengthening object contours. More importantly, this fusion strategy enriches the characteristics of the data and improves the detection accuracy of the detection model based on the deep learning training strategy.

[0047] First, the data types of the two modes, event stream and color image, are introduced:

[0048] (1) Event flow: each pixel , receiving a logarithmic brightness change within a certain period of time, it will trigger an intensity information , which can be called an event:

[0049] ;

[0050] in, For polarity, , is the threshold for event trigger sensitivity, For the same pixel Location, Variation of intensity over time.

[0051] The event flow triggered by the above formula is:

[0052] ;

[0053] (2) Color image: The pixel structure of the DAVIS event camera provides a color image (grayscale image or RGB color image) within a certain exposure interval. Depending on the length of the exposure time, the color image may produce motion blur. Therefore, the color image can be described by the integral of the intensity information within the exposure time:

[0054] ;

[0055] in, is the color image produced during the exposure time, ( ) is the exposure time.

[0056] like Figure 1 As shown, the present invention proposes a modal fusion method based on event metric function, which specifically includes the following steps:

[0057] S1: Use the DAVIS event camera to collect images of the target to be detected and obtain event streams and color images.

[0058] In step S1, the triggering formula of the event flow is:

[0059] ;

[0060] in, is the nth event, is the position coordinate of the nth event on the image sensor of the event camera, is the timestamp information of the nth event, is the polarity of the nth event, is a four-dimensional real vector space;

[0061] The calculation formula for color image is:

[0062] ;

[0063] in, is the color image produced during the exposure time, ( ) is the exposure time, T is the exposure time for generating a color image, is the potential frame, is the pixel on the image sensor, is the middle moment of the exposure time used to generate the color image.

[0064] S2: Calculate the measurement value of each pixel point in the event stream, and perform aggregation operations on the calculated measurement values ​​of each pixel point to obtain the event tensor.

[0065] Step S2 specifically includes the following steps:

[0066] S21: Calculate the measurement value of each pixel in the event stream based on the kernel function , is the mathematical expression of the nth event, The measurement function assigned to the nth event:

[0067] ;

[0068]

[0069] in, is the Dirac function, describing the impulse properties of the event stream, k is the kernel function, is the mathematical expression of the nth event, The measurement function assigned to the nth event, and They are Dot and The burst impulse response at time and The kernel function is used to measure Dot and The degree of impulse response at the moment;

[0070] because is the Dirac pulse function, which cannot be calculated, so a kernel function k is needed to The function is converted into a calculable form, and the calculated measurement value of the point is the pixel value of the event tensor.

[0071] Represents the polarity of this event. Positive polarity means that the brightness of the point increases, and negative polarity means that the brightness of the point decreases. Represents the measurement function assigned to the nth event. This measurement function represents the essential attributes of the event, such as the event count measurement function, event polarity measurement function, etc.

[0072] S22: Aggregate the calculated measurement values ​​of each pixel to obtain the event tensor:

[0073] ;

[0074] in, is the event tensor, is an aggregate function.

[0075] S3: An event integration model is constructed based on the event tensor and the color image, and the latent frame is obtained using the event integration model.

[0076] Step S3 specifically includes the following steps:

[0077] S31: Constructing event integration model based on event tensor and color image:

[0078] ;

[0079] in, To take the logarithm of the latent frame, To take the logarithm of a color image;

[0080] S32: Calculating latent frames using event integration model :

[0081] ;

[0082] Among them, c is a hyperparameter, are potential frames at different times f.

[0083] Represents a logarithmic relationship between latent frames, f is a fixed moment, t is a continuously changing moment, and this formula expresses the relationship between latent frames and event tensors at different moments.

[0084] S4: Fuse the event tensor with the latent frame to obtain a fused image with time series.

[0085] Step S4 specifically includes the following steps:

[0086] S41: The mixed image is obtained by fusing the event tensor and the latent frame as follows: :

[0087] ;

[0088] S42: assigning the mixed image to the R channel and the G channel, and assigning the complete latent frame to the B channel to obtain a fused image with a time series.

[0089] The latent frame is proposed based on the event image, which describes the relationship between the color image and the event stream. When a pixel receives a large amount of intensity information and appears overexposed or pixel ghosting, it means that excessive intensity information has been received. Therefore, it is necessary to use the discrete intensity information contained in the event stream for smoothing, which is equivalent to performing a deblurring operation on the color image.

[0090] At exposure time In the context of the summation, we define the event tensor It is the accumulation of information of each event point within a certain period of time:

[0091] ;

[0092] in, Represents pixel The event tensor in discrete form is represented as:

[0093] ;

[0094] As time accumulates, the tensor intensity of the event tensor in the summation sense increases proportionally, because under the set threshold conditions, the events triggered by the DAVIS event camera have the same intensity information. Therefore, the concept of potential frame can be used to describe this proportional effect:

[0095] ;

[0096] in, for The potential frame of the image at a certain moment is described by a logarithmic intensity as the integrated intensity increases within a certain exposure time.

[0097] The DAVIS event camera generates sparse and discrete events, but isolated event points cannot generate any information. There is only a small event tensor formed by the aggregation of events within the exposure time. In order to better describe the scene and target features, this invention assigns a metric function to each event to represent the essential attributes of the event, and by replacing different metric functions, the aggregated event tensor is used to generate different feature information. The measurement value of the pixel point :

[0098] ;

[0099] in, is the Dirac function, describing the impulse characteristics of the event, As a measurement function, it describes the essential properties of event measurement. When performing actual calculations, the Dirac function cannot be directly involved in the calculation, so the kernel function is needed to assist the calculation:

[0100] ;

[0101] The kernel function can be a Gaussian kernel function or a triangular kernel function, and the kernel function will affect the measurement value of the event. To calculate the measurement value of the event, it is necessary to aggregate the measurement value at each pixel point. Therefore, an aggregation function is introduced here. The aggregation function is for the event measurement value within a short time window, so:

[0102] ;

[0103] in, is an aggregation function, and the aggregation function includes summation, averaging, variance, maximum value, etc. When defining a potential frame, the event tensor in the sense of summation used in the present invention is defined by taking summation as the aggregation function.

[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0105] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A modal fusion method based on event metric function, characterized in that: The specific steps include: S1: Use the DAVIS event camera to collect images of the target to be detected and obtain event streams and color images; The triggering formula of the event flow is: ； in, is the nth event, is the position coordinate of the nth event on the image sensor of the event camera, is the timestamp information of the nth event, is the polarity of the nth event, is a four-dimensional real vector space; The calculation formula of the color image is: ； in, is the color image produced during the exposure time, ( ) is the exposure time, T is the exposure time for generating a color image, is the potential frame, is the pixel on the image sensor, is the middle moment of the exposure time for generating a color image; S2: Calculate the measurement value of each pixel point of the event stream, and perform aggregation operation on the calculated measurement value of each pixel point to obtain an event tensor; S21: Calculate the measurement value of each pixel in the event stream based on the kernel function : ； in, is the Dirac pulse function, describing the pulse properties of the event stream, k is the kernel function, is the mathematical expression of the nth event, The measurement function assigned to the nth event, and They are Point and The burst impulse response at time and The kernel function is used to measure Point and The degree of impulse response at the moment; S22: Aggregate the calculated measurement values ​​of each pixel to obtain the event tensor: ； in, is the event tensor, is an aggregate function; S3: constructing an event integration model based on the event tensor and the color image, and obtaining a potential frame using the event integration model; S4: Fusing the event tensor with the latent frame to obtain a fused image with a time series.

2. The modal fusion method based on event metric function according to claim 1, characterized in that: Step S3 specifically includes the following steps: S31: Constructing an event integration model based on the event tensor and the color image: ； in, To take the logarithm of the latent frame, To take the logarithm of a color image; S32: Calculate potential frames using the event integration model : ； Among them, c is a hyperparameter, are potential frames at different times f.

3. The modal fusion method based on event metric function according to claim 2, characterized in that: Step S4 specifically includes the following steps: S41: The event tensor and the potential frame are fused by the following formula to obtain a mixed image: : ； S42: assigning the mixed image to the R channel and the G channel, and assigning the complete latent frame to the B channel to obtain a fused image with a time series.

Images