Color image determination method and device, storage medium and electronic device

Abstract

The invention discloses a color image determination method and device, a storage medium and electronic equipment, and the method comprises the steps: separating a first brightness item meeting a target condition from a first color image shot by a color camera, the target condition is used for indicating that the first brightness item and an event signal collected by an event camera are aligned pixel by pixel; a second color image is determined through the first brightness item and the event signal, and the definition of the second color image is higher than that of the first color image. By adopting the technical scheme, the problem that the quality of the finally recovered image is poor by using a color and brightness separation method based on the deblurring algorithm of the event camera in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular, to a method and device for determining a color image, a storage medium, and an electronic device. Background Art

[0002] In the process of shooting with a color camera, object movement or camera shake during the shutter opening time will cause the captured color image to be blurred. Event cameras can generate event streams with high temporal resolution by detecting the brightness changes of each pixel, and are therefore widely used in the field of image deblurring.

[0003] However, in related technologies, event camera-based deblurring methods usually directly perform linear transformation on color images to separate grayscale information and color information. Since the linear separation process does not consider the relationship with the event signal trigger, it does not successfully decouple the intensity information directly related to the event signal in the color image, resulting in shape distortion and color overflow in the restored image.

[0004] In the related art, an event camera-based deblurring algorithm uses a color and brightness separation method, resulting in a problem of poor quality of the image finally restored, and no effective solution has been proposed yet. Summary of the invention

[0005] Embodiments of the present application provide a method and device for determining a color image, a storage medium, and an electronic device.

[0006] According to an embodiment of the embodiments of the present application, a method for determining a color image is provided, including: separating a first brightness item that meets a target condition from a first color image taken by a color camera, wherein the target condition is used to indicate that the first brightness item and an event signal collected by an event camera are aligned pixel by pixel; determining a second color image through the first brightness item and the event signal, wherein the clarity of the second color image is higher than that of the first color image.

[0007] In an exemplary embodiment, separating a first brightness item that meets a target condition from a first color image captured by a color camera comprises: decomposing the first color image through a trained event trigger constraint to obtain the first brightness item, wherein the trained event trigger constraint is a constraint obtained by training the event trigger constraint through a first loss function; wherein the event trigger constraint is used for pixel-by-pixel alignment of the first brightness item and the event signal, and the first loss function is determined at least by a perceptual loss between a third color image and a fourth color image; wherein the third color image is an image captured simultaneously with the fifth color image, the clarity of the third color image is higher than the clarity of the fifth color image, the fourth color image is an image obtained after a deblurring operation is performed on the fifth color image, the clarity of the fourth color image is higher than the clarity of the fifth color image, the fifth color image is a pre-captured image, and the difference between the clarity of the fifth color image and the clarity of the first color image is less than a preset threshold.

[0008] In an exemplary embodiment, determining the second color image using the first brightness item and the event signal includes: fusing the event signal integral corresponding to the event signal with the first brightness item to obtain a second brightness item; and determining the second color image according to the second brightness item.

[0009] In an exemplary embodiment, fusing an event signal integral corresponding to the event signal with the first brightness item to obtain a second brightness item includes: determining an event signal integral corresponding to the event signal; and fusing the event signal integral and the first brightness item through a cross-attention mechanism to obtain the second brightness item.

[0010] In an exemplary embodiment, determining the event signal integral corresponding to the event signal includes: determining a forward event integral and a reverse event integral corresponding to the event signal; marking the forward event integral by a first position encoding vector in a transformer model to obtain a marked forward event integral, and marking the reverse event integral by a second position encoding vector in the transformer model to obtain a marked reverse event integral, wherein the transformer model includes the cross-attention mechanism; weighted fusion of the marked forward event integral and the marked reverse event integral to obtain the event signal integral.

[0011] In an exemplary embodiment, determining the second color image according to the second brightness item includes: separating a first color item from the first color image, wherein the first color item retains smooth color information; and determining the second color image according to the second brightness item and the first color item.

[0012] In an exemplary embodiment, separating a first color item from the first color image includes: decomposing the first color image through a trained gradient constraint to obtain the first color item, wherein the trained gradient constraint is a constraint obtained by training the gradient constraint through a second loss function; wherein the second loss function is determined by a third brightness item of a sixth color image and a second color item of a seventh color image, and the sixth color image and the seventh color image each include one of the following: a pre-captured fifth color image, a third color image captured simultaneously with the fifth color image, wherein a difference between a clarity of the fifth color image and a clarity of the first color image is less than a preset threshold, and the clarity of the third color image is higher than that of the fifth color image.

[0013] In an exemplary embodiment, determining the second color image based on the second brightness item and the first color item includes: determining an edge map of the first color image through the second brightness item; fusing the edge map with the event signal to obtain a fusion feature; correcting the first color item through the fusion feature to obtain a third color item; and determining the second color image through the second brightness item and the third color item.

[0014] According to another embodiment of the embodiments of the present application, a device for determining a color image is also provided, including: a separation module, used to separate a first brightness item that meets a target condition from a first color image taken by a color camera, wherein the target condition is used to indicate that the first brightness item and an event signal collected by an event camera are aligned pixel by pixel; a determination module, used to determine a second color image through the first brightness item and the event signal, wherein the clarity of the second color image is higher than that of the first color image.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for determining a color image when running.

[0016] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the color image determination method through the computer program.

[0017] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, wherein the computer program is executed by a processor to perform the method.

[0018] In an embodiment of the present application, after separating the first brightness item that is pixel-by-pixel aligned with the event signal collected by the event camera from the first color image captured by the color camera, a second color image with higher clarity than the first color image can be determined through the event signal and the first brightness item. Through the embodiment of the present application, the problem of poor quality of the image finally restored by the color and brightness separation method used in the deblurring algorithm based on the event camera in the related art can be solved. Therefore, by separating the first brightness item that is pixel-by-pixel aligned with the event signal, the image quality of the second color image finally restored is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 is a flow chart of a method for determining a color image according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of a method for determining a color image according to an optional embodiment of the present application;

[0023] Figure 3 is a structural block diagram of a color image determination device according to an embodiment of the present application (I);

[0024] Figure 4 is a structural block diagram of a color image determination device according to an embodiment of the present application (II);

[0025] Figure 5 It is a schematic diagram of an electronic device for a method for determining a color image according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] In this embodiment, a method for determining a color image is provided, which is applied to a device provided with a color camera and an event camera. Figure 1 : is a flowchart of a method for determining a color image according to an embodiment of the present application, the process comprising the following steps:

[0029] Step S102, separating a first brightness item that meets a target condition from a first color image captured by a color camera, wherein the target condition is used to indicate that the first brightness item and an event signal collected by an event camera are aligned pixel by pixel;

[0030] Optionally, a color camera refers to a camera capable of capturing and recording color images, and the color camera includes at least a traditional camera and a digital camera. Specifically, a traditional camera, also known as a film camera, is mainly characterized by using film as a photosensitive material to form an image. A digital camera uses a digital photosensitive element (such as CCD or CMOS) to capture an image.

[0031] Step S104: determining a second color image by using the first brightness item and the event signal, wherein the clarity of the second color image is higher than that of the first color image.

[0032] It should be noted that the first color image refers to an image that visually lacks clarity or sharpness, usually manifested as unclear edges, blurred or difficult to distinguish details, and the second color image refers to an image that visually has high clarity and sharpness, rich details, clear edges and high resolution.

[0033] Optionally, devices equipped with color cameras and event cameras include but are not limited to driving recorders, robots, etc.

[0034] For a driving recorder, during the use of the driving recorder, a color camera arranged on or associated with the driving recorder collects a first color image used to characterize the road and traffic conditions in front of the vehicle; if the first color image is directly presented on the display screen of the driving recorder, the first color image may not be clear. However, through the scheme of the embodiment of the present application, the driving recorder can obtain an event signal corresponding to the first color image from an event camera arranged on or associated with the driving recorder. Optionally, the first color image and the event signal corresponding to the first color image can be obtained from the color camera and the event camera by the central processor deployed in the driving recorder. Then, when the central processor of the driving recorder obtains the first color image and the event signal, the central processor can decompose the first brightness item aligned with the event signal pixel by pixel from the first color image, and then process the first color image into a second color image with higher clarity than the first color image through the first brightness item and the event signal, and send the second color image to the display screen of the driving recorder for display. As a result, the driver can accurately and clearly know the road and traffic conditions ahead from the display screen of the driving recorder even when moving at high speed.

[0035] For robot vision, in general, color cameras and event cameras can be deployed on the robot as visual sensors of the robot. In the case of robot motion, the first color image directly collected by the color camera may be blurred, and the first color image directly used for robot vision may reduce the robot's visual perception ability. However, through the scheme of the embodiment of the present application, when the robot's brain (e.g., central processing unit) perceives (i.e., collects) the first color image through the color camera, the brain will obtain the event signal corresponding to the first color image from the event camera; and then separate the first brightness item that is pixel-by-pixel aligned with the event signal from the first color image, and determine the second color image with a higher clarity than the first color image through the first brightness item and the event signal, so that the robot can more accurately identify objects, perform target detection and tracking, and perform tasks such as scene understanding and navigation. In addition, clear images also help robots perform various tasks more accurately and improve work efficiency and accuracy.

[0036] For example, for a sweeping robot, the second color image obtained by the brain with higher clarity than the first color image will be used to more intelligently plan the cleaning path, avoid obstacles, and perform other operations.

[0037] The movement of the robot causes the color camera arranged on the robot to collect a first color image, and then an event signal corresponding to the first color image of the event camera arranged on the robot can be obtained to execute the above-mentioned color image determination method.

[0038] Through the above steps, after separating the first brightness item that is pixel-by-pixel aligned with the event signal collected by the event camera from the first color image taken by the color camera, the second color image with higher clarity than the first color image can be determined through the event signal and the first brightness item. Through the embodiment of the present application, the problem of poor quality of the image finally restored by the color and brightness separation method used in the deblurring algorithm based on the event camera in the related art can be solved. Therefore, by separating the first brightness item that is pixel-by-pixel aligned with the event signal, the image quality of the second color image finally restored is better.

[0039] In some optional embodiments, the first brightness item and the first color item can be separated from the first color image, and the operation of separating the first brightness item and the operation of separating the first color item can be performed simultaneously or successively, which is not limited in the embodiments of the present application.

[0040] In the case of performing the operations successively or simultaneously but separating the first brightness item and the first color item separately from the first color image, the scheme for separating the first brightness item includes: decomposing the first color image by means of a trained event trigger constraint to obtain the first brightness item, wherein the trained event trigger constraint is a constraint obtained by training the event trigger constraint by means of a first loss function; wherein the event trigger constraint is used for pixel-by-pixel alignment of the first brightness item and the event signal, and the first loss function is determined by at least a perceptual loss between a third color image and a fourth color image; wherein the third color image is an image acquired simultaneously with the fifth color image, the clarity of the third color image is higher than the clarity of the fifth color image, the fourth color image is an image obtained by performing a deblurring operation on the fifth color image, the clarity of the fourth color image is higher than the clarity of the fifth color image, the fifth color image is a pre-acquired image, and the difference between the clarity of the fifth color image and the clarity of the first color image is less than a preset threshold.

[0041] Therefore, the embodiment of the present application performs a separation operation on the first color image through event triggering constraints, so that the separated first brightness item is aligned pixel by pixel with the event signal of the event camera.

[0042] In the case of performing the method sequentially or simultaneously, but separating the first brightness item and the first color item from the first color image alone, the method of separating the first color item comprises: decomposing the first color image by a trained gradient constraint to obtain the first color item, wherein the trained gradient constraint is a constraint obtained by training the gradient constraint by a second loss function; wherein the second loss function is determined by a third brightness item of a sixth color image and a second color item of a seventh color image, wherein the sixth color image and the seventh color image both comprise one of the following: a pre-collected fifth color image, a third color image collected simultaneously with the fifth color image, wherein the difference between the clarity of the fifth color image and the clarity of the first color image is less than a preset threshold, and the clarity of the third color image is higher than the clarity of the fifth color image. It should be noted that the gradient constraint is used to ensure the smoothness of the first color item.

[0043] Therefore, the embodiment of the present application performs a separation operation on the first color image through gradient constraints, so that the separated first color items retain smooth color information, which is beneficial for the subsequent generation of a second color image based on the first color items with smooth color information to avoid color overflow and have better image quality.

[0044] Optionally, when the first brightness item and the first color item are separated from the first color image simultaneously, the separation scheme includes: using an objective function to separate the first color item and the first brightness item from the first color image simultaneously. The objective function includes both an event trigger constraint and a gradient constraint. The objective function is a function trained by the first loss function and the second loss function.

[0045] Optionally, the objective function including both event triggering constraints and gradient constraints is shown in formula (1):

[0046]

[0047] Where B represents the first color image, B g represents the first brightness term, B c represents the first color item, represents the image merging operation, represents the gradient operator, represents the regularization term, where e represents the event signal. More specifically, represents the gradient constraint, Represents the event trigger constraint. Optionally, the event trigger constraint refers to formula (2):

[0048]

[0049] Among them, DebluR represents the deblurring operation, S g Indicates the brightness item of the clear image (equivalent to the third color image).

[0050] Furthermore, the first loss function for training the event trigger constraint is specifically determined by the perceptual loss and the mean square error between the third color image and the fourth color image. The first loss function is shown in formula (3):

[0051]

[0052] Among them, β 1 , β 2 is the weight parameter, is the mean square error, is the perception error, S rec is the reconstructed image (equivalent to the fourth color image), and S is the clear image (equivalent to the third color image).

[0053] Further, the second loss function for training the gradient constraint is determined by the Hadamard product between the third brightness item of the sixth color image and the second color item of the seventh color image; more specifically, by the difference between the Hadamard product and the third image. The second loss function is shown in formula (4):

[0054]

[0055] Among them, X 1 , X 2 It represents any one of a blurred image (equivalent to the fifth color image. It should be noted that in the training process, the fifth color image used is also represented as B) and a clear image (represented as S). is the weight parameter, and ⊙ represents the Hadamard product. represents the second color term, Represents the third brightness term.

[0056] Therefore, the embodiment of the present application can decompose the first brightness item and the first color item from the first color image through the objective function including the gradient constraint and the event trigger constraint, and ensure that the first brightness item and the event signal are aligned pixel by pixel, and ensure that the first color item has smooth color information, which provides a good foundation for the subsequent further processing of the first brightness item and the first color item. In addition, the operation of separating the first brightness item and the first color item at the same time also saves time, making the processing efficiency of the first color image higher.

[0057] In an optional embodiment, the above step S104 can be implemented by the following scheme, including: fusing the event signal integral corresponding to the event signal with the first brightness item to obtain a second brightness item; and determining the second color image according to the second brightness item.

[0058] In an exemplary embodiment, fusing an event signal integral corresponding to the event signal with the first brightness item to obtain a second brightness item includes: determining an event signal integral corresponding to the event signal; and fusing the event signal integral and the first brightness item through a cross-attention mechanism to obtain the second brightness item.

[0059] It should be noted that event signal integration refers to integrating the event signal generated by the event camera sensor to obtain richer and more accurate information. The event signal is a high-frequency signal related to time. By integrating the event signal, information such as the total amount or duration of the event can be obtained. After decomposing the first color image, the first brightness item aligned pixel by pixel with the event signal is integrated and fused with the event signal corresponding to the event signal, and a second brightness item deblurred by the event signal can be obtained.

[0060] By fusing the event signal integral with the first brightness term, the information of the event camera and the color camera can be combined to achieve a more comprehensive scene perception and understanding of the scene captured by the first color image, thereby obtaining a second color image with higher clarity than the first color image.

[0061] Optionally, the second brightness item is obtained by fusing the event signal integral and the first brightness item through a cross attention mechanism, including: extracting a first feature map of the first color image, and extracting a second feature map of the event signal integral; fusing the first feature map and the second feature map to obtain a third feature map of the event signal. For details, see formula (5); finally, determining the second brightness item through the third feature map and the first brightness item.

[0062]

[0063] Among them, X e is the second characteristic graph of the event signal integration, X b is the first feature map of the first color image, W {Q，K，V} is the weight matrix in the attention mechanism, W p is the weight matrix, and Atten represents the attention mechanism. The third characteristic map representing the optimized event signal integral.

[0064] Optionally, the first feature map and the second feature map may be extracted through a convolutional layer.

[0065] Furthermore, determining the event signal integral corresponding to the event signal includes: determining a forward event integral and a reverse event integral corresponding to the event signal; marking the forward event integral by a first position encoding vector in a transformer model to obtain a marked forward event integral, and marking the reverse event integral by a second position encoding vector in the transformer model to obtain a marked reverse event integral, wherein the transformer model includes the cross-attention mechanism; weighted fusion of the marked forward event integral and the marked reverse event integral to obtain the event signal integral.

[0066] Among them, the architecture of the transformer model and the architecture of the cross-attention mechanism included in the transformer model can adopt any architecture in the relevant technology, and the embodiments of the present application are not limited to this.

[0067] It should be noted that the forward event integral refers to the integration from the moment the event occurs, gradually accumulating the size of the event signal; while the reverse event integral is the integration from the moment the event ends, gradually reducing the size of the event signal. The forward event integral and the reverse event integral are relative concepts, and are used to obtain the overall information of the event signal in the embodiments of the present application.

[0068] The purpose of marking the forward event integral and the reverse event integral by the position coding vector is to clearly distinguish the two when processing the event signal, and to facilitate subsequent data processing and analysis. The position coding vector can embed the information of the forward event integral and the reverse event integral into the event signal, making the processing process more intuitive and accurate. Among them, the position coding vector includes: a first position coding vector and a second position coding vector.

[0069] In an optional embodiment, determining the second color image according to the second brightness item includes: separating a first color item from the first color image, wherein the first color item retains smooth color information; and determining the second color image according to the second brightness item and the first color item.

[0070] Furthermore, determining the second color image according to the second brightness item and the first color item includes: determining an edge map of the first color image through the second brightness item; fusing the edge map with the event signal to obtain a fusion feature; correcting the first color item through the fusion feature to obtain a third color item; and determining the second color image through the second brightness item and the third color item.

[0071] Specifically, the second brightness term can clearly outline the boundary of the internal area of ​​the first color image. By performing edge detection on the second brightness term using the Laplace operator, an edge map can be extracted; the edge map is combined with the event signal through the convolution layer and the squeeze-and-excitation block to obtain a fusion feature. Then, the fusion feature and the first color term are input into a preset color compensation network to obtain a third color term. The second brightness term and the third color term are re-fused to obtain a deblurred second color image corresponding to the first color image.

[0072] Therefore, the embodiment of the present application enhances the edge information of the image by extracting the edge map from the second brightness item, which is beneficial to avoid the shape distortion of the second color image obtained based on the edge map. In summary, the embodiment of the present application finally obtains a second color image that avoids shape distortion and color overflow and has higher clarity than the first color image, and solves the problem that the color and brightness separation method used in the deblurring algorithm based on the event camera in the related art ultimately restores the image with poor quality. The technical effect of obtaining a high-quality second color image based on event signal recovery is achieved.

[0073] In order to better understand the process of the above-mentioned color image determination method, the implementation method flow of the above-mentioned color image determination is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of the present application.

[0074] In order to solve the problem of poor quality of the image finally restored by the color and brightness separation method used in the deblurring algorithm based on the event camera in the related art, an optional embodiment of the present application proposes a method for determining a color image, and the execution flow of the method is shown as follows: Figure 2 As shown, the following steps are included:

[0075] Step 1: Decomposing the blurred color image (equivalent to the first color image in the above embodiment, and equivalent to Figure 2 The blurred image B) is decomposed into a brightness item (equivalent to the first brightness item in the above embodiment) and a color item (equivalent to the first color item in the above embodiment).

[0076] Specifically, the event-triggered image decomposition strategy uses the optimized gradient constraint and event-triggered constraint to decompose the fuzzy color image into brightness term and color term based on the proposed objective function, and makes the brightness term pixel-by-pixel alignment with the event signal, while the color term retains the smooth color information. In the optimization of the objective function, the gradient constraint and event-triggered constraint are converted into the minimization of the mean square error for the grayscale image (i.e., the brightness term), and the gradient descent algorithm is used to achieve the decomposition of the color image.

[0077] Furthermore, the image decomposition based on event-triggered perception includes:

[0078] The objective function corresponding to the event-triggered perceptual image decomposition can realize a decomposition method that can decompose the luminance term that is precisely aligned with the event triggering process (equivalent to the event signal) from the blurred color image, and separate the color information that does not overlap with the event signal (equivalent to the first color term). Since the event signal records the motion information in the luminance space, and the smooth area is not easy to trigger the event, the decomposed luminance term B g For a single-channel image, extract the blurred brightness information and align the blurred brightness information with the grayscale event signal trigger process. The decomposed color item B c is a three-channel image to preserve smooth blurred color information, color term B c The content recorded in does not overlap with the information recorded by the event signal. Therefore, the calculation method of the objective function to be optimized is shown in formula (1).

[0079] Combining formula (1), it can be seen that the objective function requires that the two decomposed items (i.e., the first brightness item and the first color item) can reconstruct the original image (i.e., the first color image). At the same time, there are two constraints, namely, the gradient constraint and the event-triggered constraint. The gradient constraint ensures the smoothness of the color item, and the event-triggered constraint ensures the pixel-level alignment of the brightness item and the event signal by evaluating the deblurring effect. The calculation method of the event-triggered constraint is shown in formula (2).

[0080] The optimization of the above objective function is achieved using two loss functions. For the gradient constraint, it is achieved by recombining the decomposed terms. The calculation method of the corresponding second loss function is shown in formula (4). For the event-triggered constraint, it is necessary to evaluate the effect of image deblurring. This is achieved by using the mean square error and perceptual loss of the reconstructed image and the clear image. The calculation method of the corresponding first loss function is shown in formula (3).

[0081] For example, Figure 2 As shown in FIG. 1 , the decomposition process of the blurred color image may include: using a decoder network composed of 6 convolutional layers to decompose the blurred color image, decomposing the blurred color image into brightness terms and color terms based on a combination of loss functions and a gradient descent method, and realizing pixel-by-pixel alignment of the brightness term with the event signal, while retaining smooth color information in the color term.

[0082] Step 2: The brightness deblurring method based on bidirectional event fusion is used to effectively remove blur from the brightness item.

[0083] Optionally, for the blurred brightness item obtained by decomposition (equivalent to the first brightness item in the above embodiment), the brightness deblurring method of bidirectional event fusion is based on the forward and reverse integration of the event signal (equivalent to the forward event integration and the reverse event integration) to extract the motion information in the event signal. The fusion of the forward and reverse integrations is marked by different position encoding vectors in the transformer model (Transformer), and the cross-attention mechanism is used to fuse the event signal integration and brightness item features to reconstruct the deblurred brightness item (equivalent to the second brightness item in the above embodiment).

[0084] Optionally, in order to achieve effective extraction of event signal motion information, an optional embodiment of the present application proposes a bidirectional event integral fusion model, which uses the forward and reverse event integrals included in the bidirectional event integral to deblur the blurred color image. There is a correlation relationship between the bidirectional event integral and the blurred image and the clear image. The correlation relationship is shown in formula (6) and formula (7):

[0085]

[0086] Among them, S represents a clear image, E(t_x,t_y) represents the positive accumulation of time signals between the trigger time t_x and t_y, and ε pre Positive event quadratic integration;

[0087] Among them, S' also represents a clear image, E'(t_x,t_y) represents the reverse accumulation of the time signal between the trigger time t_x and t_y, ε post Reverse event quadratic integration.

[0088] Furthermore, the bidirectional event integrals (including the forward event integral and the reverse event integral in the above embodiment) are weighted and combined with the blurred brightness term B g Combined, the deblurred brightness image I can be restored rec (equivalent to the second brightness item).

[0089] For example, Figure 2 As shown, through Figure 2The U-Net architecture in the invention includes an image processing network (EvLumin-Net) based on an event camera to fuse the features of the blurred image and the event stream (i.e., the event signal) to generate the event signal integral feature E. After that, the first position coding vector and the second position coding vector are used to realize the marking of the forward and reverse event integral features, wherein the first position coding vector and the second position coding vector are the same vector; a transformer model (Transformer) with a cross attention mechanism is used to fuse the bidirectional event integral, and finally a deblurred brightness image is generated. The architecture of the cross attention mechanism and the transformer model adopts the original architecture, but at the same time introduces the feature map of the blurred color image (equivalent to the first feature map in the above embodiment) as a key to improve the fusion effect of the event double integral. The feature map of the blurred color image and the feature map of the event integral (equivalent to the second feature map in the above embodiment) are extracted through a convolution layer. The calculation method of the fusion of the feature map of the blurred color image and the feature map of the event integral is shown in formula (5).

[0090] Optionally, the second brightness item may be calculated by using only the original integral signal of the time signal in combination with the first brightness item. The calculation method may refer to formula (8).

[0091] S = B ·∫∫edt; (8)

[0092] Where S represents a clear image, B represents a blurred image, and e represents an event signal. In the case of formula (8), B can be replaced by the first brightness term and S can be replaced by the second brightness term.

[0093] Step 3: Correct the color overflow at the edge of the image through the color compensation strategy guided by the edge map.

[0094] After obtaining the deblurred brightness item, an optional embodiment of the present application guides the correction of the color item based on the edge information of the deblurred brightness item. The edge map is calculated from the deblurred brightness item using the Laplacian operator, and the clear edge restored by the deblurred brightness item is used to guide the color compensation network to correct the color item, thereby reducing the image color overflow phenomenon.

[0095] Specifically, the correction of the color item in the optional embodiment of the present application also needs to be combined with the event signal. However, the color item decomposed from the blurred color image is a three-channel image, and the event signal is recorded in a monochrome space, which can overlap with the brightness item as a single-channel image, but cannot overlap with the color item as a three-channel image. Therefore, in the process of correcting the color item, the problem that the monochrome event signal and the color information do not overlap needs to be solved.

[0096] In order to solve the problem that the monochrome event signal and the color information do not overlap, an optional embodiment of the present application proposes a color compensation strategy based on edge map guidance. Since the event signal is recorded in a monochrome space, it is not feasible to directly combine the event signal with the color information. After obtaining the deblurred brightness item, this brightness item can clearly outline the boundary of the internal area of ​​the image. The edge map calculated from the deblurred brightness item is used to guide the color compensation process, thereby solving the problem that the color information and the event information do not overlap and cannot effectively guide the recovery.

[0097] For example, Figure 2 As shown, the color item is corrected by the color compensation network (ImgColor-Net) included in the U-Net architecture as shown in 2. First, the deblurred brightness item I rec The edge map is extracted from the image, and the edge map is obtained by edge detection using the Laplace operator. Before fusing the color term features, the edge map is combined with the event signal using a convolutional layer and a squeeze and excitation block, and then these fused features are input into the ImgColor-Net network to reconstruct the color term C rec (equivalent to the third color term).

[0098] Optional, for example Figure 2 The training process of the network shown includes:

[0099] A two-stage training strategy is adopted for training. In the first stage, the decoder network and EvLumin-Net are trained using decomposition loss (equivalent to the second loss function) and reconstruction loss (equivalent to the first loss function). When calculating the reconstruction loss, the deblurred brightness term is directly combined with the decomposed color term. By jointly optimizing image decomposition and brightness deblurring, the decoder network is trained in a task-oriented manner.

[0100] In the second stage, the parameters of the decoder network and EvLumin-Net are frozen, and ImgColor-Net is trained using the reconstruction loss. Training is implemented on the NVIDIA RTX 3090 GPU using the PyTorch platform. In each stage of training, the network is trained for 100 rounds using the AdamW optimizer and the cosine learning rate scheduler. The initial learning rate is set to 1×10 of the first stage. -3 and 1×10 in the second stage -4 .

[0101] Optionally, all convolution modules in the EvLumin-Net network and the ImgColor-Net network can be replaced by modules in the Transformer model.

[0102] Step 4: Re-integrate the deblurred brightness term and color term to achieve deblurring of the blurred color image.

[0103] By combining the deblurred color term with the brightness term, a clearer image can be generated. Therefore, the optional embodiment of the present application can effectively eliminate the modal difference between the monochrome event signal and the color image signal by decomposing the blurred color image, processing the different decomposed terms in steps, and finally recombining them into an image, thereby achieving effective restoration of the blurred color image.

[0104] In summary, the optional embodiment of the present application can decouple the brightness item and the non-overlapping color item triggered by the event signal after decomposing the blurred color signal through an objective function including two constraints, thereby improving the utilization efficiency of the event signal. In the process of deblurring the brightness item obtained by decomposition, the strategy of two-way event fusion can improve the reconstruction quality of the brightness item and avoid the problem of shape distortion. And by extracting the edge map from the deblurred brightness item and guiding the compensation of the decomposed color item through the edge map, the color overflow problem can be effectively avoided.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0106] Figure 3 is a structural block diagram of a color image determination device according to an embodiment of the present application (I); Figure 3 As shown, including:

[0107] A separation module 32, configured to separate a first brightness item that meets a target condition from a first color image captured by a color camera, wherein the target condition is used to indicate that the first brightness item and an event signal collected by an event camera are aligned pixel by pixel;

[0108] The determination module 34 is configured to determine a second color image according to the first brightness item and the event signal, wherein the second color image has a higher definition than the first color image.

[0109] Through the above device, after separating the first brightness item that is pixel-by-pixel aligned with the event signal collected by the event camera from the first color image taken by the color camera, the second color image with higher clarity than the first color image can be determined through the event signal and the first brightness item. Through the embodiment of the present application, the problem of poor quality of the image finally restored by the color and brightness separation method used in the deblurring algorithm based on the event camera in the related art can be solved. Therefore, by separating the first brightness item that is pixel-by-pixel aligned with the event signal, the image quality of the second color image finally restored is better.

[0110] In an exemplary embodiment, the separation module 32 is further used to: decompose the first color image through a trained event trigger constraint to obtain the first brightness item, wherein the trained event trigger constraint is a constraint obtained by training the event trigger constraint through a first loss function; wherein the event trigger constraint is used for pixel-by-pixel alignment of the first brightness item and the event signal, and the first loss function is determined at least by the perceptual loss between the third color image and the fourth color image; wherein the third color image is an image captured simultaneously with the fifth color image, the clarity of the third color image is higher than the clarity of the fifth color image, the fourth color image is an image obtained after performing a deblurring operation on the fifth color image, the clarity of the fourth color image is higher than the clarity of the fifth color image, the fifth color image is a pre-captured image, and the difference between the clarity of the fifth color image and the clarity of the first color image is less than a preset threshold.

[0111] In an exemplary embodiment, Figure 4 As shown, the determination module 34 also includes a fusion unit 342 and a determination unit 344, wherein the fusion unit is used to fuse the event signal integral corresponding to the event signal with the first brightness item to obtain a second brightness item; and the determination unit is used to determine the second color image according to the second brightness item.

[0112] In an exemplary embodiment, the fusion unit is further used to: determine an event signal integral corresponding to the event signal; and fuse the event signal integral and the first brightness item through a cross-attention mechanism to obtain the second brightness item.

[0113] In an exemplary embodiment, the fusion unit is also used to: determine the forward event integral and the reverse event integral corresponding to the event signal; mark the forward event integral by the first position encoding vector in the transformer model to obtain the marked forward event integral, and mark the reverse event integral by the second position encoding vector in the transformer model to obtain the marked reverse event integral, wherein the transformer model includes the cross-attention mechanism; weightedly fuse the marked forward event integral and the marked reverse event integral to obtain the event signal integral.

[0114] In an exemplary embodiment, the determining unit is further configured to: separate a first color item from the first color image, wherein the first color item retains smooth color information; and determine the second color image according to the second brightness item and the first color item.

[0115] In an exemplary embodiment, the determination unit is also used to: decompose the first color image through a trained gradient constraint to obtain the first color item, wherein the trained gradient constraint is a constraint obtained by training the gradient constraint through a second loss function; wherein the second loss function is determined by a third brightness item of the sixth color image and a second color item of the seventh color image, and the sixth color image and the seventh color image each include one of the following: a pre-captured fifth color image, a third color image captured simultaneously with the fifth color image, wherein the difference between the clarity of the fifth color image and the clarity of the first color image is less than a preset threshold, and the clarity of the third color image is higher than the clarity of the fifth color image.

[0116] In an exemplary embodiment, the determination unit is also used to: determine the edge map of the first color image through the second brightness item; fuse the edge map with the event signal to obtain a fusion feature; correct the first color item through the fusion feature to obtain a third color item; and determine the second color image through the second brightness item and the third color item.

[0117] An embodiment of the present application further provides a storage medium, which includes a stored program, wherein the program executes any of the above methods when it is run.

[0118] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0119] S1, separating a first brightness item that meets a target condition from a first color image captured by a color camera, wherein the target condition is used to indicate that the first brightness item and an event signal collected by an event camera are aligned pixel by pixel;

[0120] S2. Determine a second color image using the first brightness item and the event signal, wherein the clarity of the second color image is higher than that of the first color image.

[0121] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0122] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0123] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0124] S1, separating a first brightness item that meets a target condition from a first color image captured by a color camera, wherein the target condition is used to indicate that the first brightness item and an event signal collected by an event camera are aligned pixel by pixel;

[0125] S2. Determine a second color image using the first brightness item and the event signal, wherein the clarity of the second color image is higher than that of the first color image.

[0126] An embodiment of the present application further provides a computer program product, including a computer program, which is used by a processor to execute the steps in any of the above method embodiments.

[0127] Optionally, in this embodiment, the computer program product may be executed by a processor to perform the following steps:

[0128] S1, separating a first brightness item that meets a target condition from a first color image captured by a color camera, wherein the target condition is used to indicate that the first brightness item and an event signal collected by an event camera are aligned pixel by pixel;

[0129] S2. Determine a second color image using the first brightness item and the event signal, wherein the clarity of the second color image is higher than that of the first color image.

[0130] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0131] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0132] Optionally, the embodiment of the present application further provides another electronic device for implementing the above-mentioned color image determination method, such as Figure 5 As shown, the electronic device includes a memory 502 and a processor 504. The memory 502 stores a computer program, and the processor 504 is configured to execute the steps in any of the above method embodiments through the computer program.

[0133] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network, or may be located in a driving recorder, a robot, or other device.

[0134] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0135] S1, separating a first brightness item that meets a target condition from a first color image captured by a color camera, wherein the target condition is used to indicate that the first brightness item and an event signal collected by an event camera are aligned pixel by pixel;

[0136] S2. Determine a second color image using the first brightness item and the event signal, wherein the clarity of the second color image is higher than that of the first color image.

[0137] Alternatively, a person skilled in the art may understand that: Figure 5 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 5 The structure of the electronic device is not limited. Figure 5 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 5 Different configurations are shown.

[0138] Among them, the memory 502 can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for determining a color image in the embodiment of the present invention. The processor 504 executes various functional applications and data processing by running the software programs and modules stored in the memory 502, that is, realizing the above-mentioned method for determining a color image. The memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 502 may further include a memory remotely located relative to the processor 504, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. Among them, as an example, Figure 5 As shown, the memory 502 may include but is not limited to the separation module 32 and the determination module 34 in the color image determination device. In addition, it may also include but is not limited to other module units in the color image determination device, which will not be repeated in this example.

[0139] Optionally, the transmission device 506 is used to receive or send data via a network. Specific examples of the network may include wired networks and wireless networks. In one example, the transmission device 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 506 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0140] In addition, the electronic device further includes: a display 508 for displaying the first color image, the second color image, and the processing process of the first color image, etc.; and a connection bus 510 for connecting various module components in the electronic device.

[0141] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0142] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for determining a color image, characterized in that: include: Separating a first brightness item that meets a target condition from a first color image captured by a color camera, wherein the target condition is used to indicate that the first brightness item and an event signal collected by an event camera are aligned pixel by pixel; A second color image is determined by the first brightness item and the event signal, wherein the definition of the second color image is higher than that of the first color image.

2. The method for determining a color image according to claim 1, characterized in that: The step of separating a first brightness item that meets a target condition from a first color image captured by a color camera comprises: Decomposing the first color image by using the trained event trigger constraint to obtain the first brightness item, wherein the trained event trigger constraint is a constraint obtained by training the event trigger constraint by using the first loss function; The event trigger constraint is used for pixel-by-pixel alignment of the first brightness item and the event signal, and the first loss function is determined at least by a perceptual loss between a third color image and a fourth color image; Among them, the third color image is an image collected at the same time as the fifth color image, the clarity of the third color image is higher than the clarity of the fifth color image, the fourth color image is an image obtained after performing a deblurring operation on the fifth color image, the clarity of the fourth color image is higher than the clarity of the fifth color image, the fifth color image is a pre-collected image, and the difference between the clarity of the fifth color image and the clarity of the first color image is less than a preset threshold.

3. The method for determining a color image according to claim 1 or 2, characterized in that: The determining of the second color image by using the first brightness item and the event signal comprises: Fusing the event signal integral corresponding to the event signal and the first brightness item to obtain a second brightness item; The second color image is determined according to the second brightness item.

4. The method for determining a color image according to claim 3, characterized in that: The event signal integral corresponding to the event signal and the first brightness item are integrated to obtain a second brightness item, including: determining an event signal integral corresponding to the event signal; The event signal integral and the first brightness item are fused through a cross attention mechanism to obtain the second brightness item.

5. The method for determining a color image according to claim 4, characterized in that: Determining an event signal integral corresponding to the event signal includes: Determine a forward event integral and a reverse event integral corresponding to the event signal; Marking the forward event integral by a first position encoding vector in a transformer model to obtain a marked forward event integral, and marking the reverse event integral by a second position encoding vector in the transformer model to obtain a marked reverse event integral, wherein the transformer model includes the cross-attention mechanism; The event signal integral is obtained by weighted fusion of the marked forward event integral and the marked reverse event integral.

6. The method for determining a color image according to any one of claims 3 to 5, characterized in that: Determining the second color image according to the second brightness item includes: Separating a first color item from the first color image, wherein the first color item retains smooth color information; The second color image is determined according to the second brightness item and the first color item.

7. The method for determining a color image according to claim 6, characterized in that: Separating a first color item from the first color image includes: Decomposing the first color image by using a trained gradient constraint to obtain the first color item, wherein the trained gradient constraint is a constraint obtained by training the gradient constraint by using a second loss function; Among them, the second loss function is determined by the third brightness item of the sixth color image and the second color item of the seventh color image, and the sixth color image and the seventh color image each include one of the following: a pre-captured fifth color image, and a third color image captured simultaneously with the fifth color image, wherein the difference between the clarity of the fifth color image and the clarity of the first color image is less than a preset threshold, and the clarity of the third color image is higher than the clarity of the fifth color image.

8. The method for determining a color image according to claim 6 or 7, characterized in that: Determining the second color image according to the second brightness item and the first color item includes: determining an edge map of the first color image using the second brightness item; Fusing the edge map with the event signal to obtain a fused feature; Correcting the first color item by using the fusion feature to obtain a third color item; The second color image is determined by the second brightness item and the third color item.

9. A device for determining a color image, characterized in that: include: A separation module, used to separate a first brightness item that meets a target condition from a first color image captured by a color camera, wherein the target condition is used to indicate that the first brightness item and an event signal collected by an event camera are aligned pixel by pixel; A determination module is used to determine a second color image according to the first brightness item and the event signal, wherein the second color image has a higher definition than the first color image.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 8 when executed.

11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

12. A computer program product, comprising a computer program, characterized in that The computer program is used by a processor to execute the method described in any one of claims 1 to 8.

Images