Event image processing method, processing model training method and related equipment

Abstract

The embodiment of the invention provides an event image processing method, a processing model training method and related equipment, and belongs to the technical field of digital image signal processing. The processing method comprises the following steps: inputting event features of an event image into an encoder to carry out encoding processing under a plurality of different scales to obtain a plurality of intermediate features; performing residual processing on the intermediate features output under the last scale to obtain a target semantic feature, and inputting the target semantic feature and each intermediate feature into a brightness decoder to perform decoding processing under the corresponding scale to obtain a plurality of brightness features; inputting the target semantic feature and each brightness feature into a color decoder for decoding processing under a corresponding scale to obtain a plurality of color features; splicing the brightness features and the color features under a preset scale, and outputting a target color image; and identifying a target object in the target color image to obtain a target identification result. According to the invention, the accuracy of identifying the target object in the event image can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of digital image signal processing, and in particular to a method for processing event images, a method for training a processing model, and related equipment. Background Art

[0002] Event-based camera is a new type of bionic asynchronous sensor. Different from the instantaneous static images captured by traditional cameras, event cameras capture "events". "Events" can be simply understood as "changes in pixel brightness", that is, the output of the event camera is the changes in pixel brightness.

[0003] In the related art, in order to realize the recognition of event images, it is necessary to retrain the pre-built model based on a large number of event image training sets and verification sets (collectively referred to as "event image datasets"). However, due to the working principle and output characteristics of event cameras, the number of event image datasets is usually very limited. Therefore, the trained model cannot recognize the target objects in the event images well, which makes the recognition results corresponding to the target objects output by the model have poor accuracy. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method for processing event images, a method for training a processing model and related equipment, aiming to improve the accuracy of identifying target objects in event images.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for processing an event image, the method comprising:

[0006] Acquire an event image and input the event image into a pre-trained event image processing model;

[0007] Extract event features from the event image, input the event features into an encoder for encoding at multiple scales, and output intermediate features at multiple scales;

[0008] Perform residual processing on the intermediate features output at the last scale to obtain the target semantic features, and input the target semantic features and each intermediate feature into the brightness decoder for decoding processing at the corresponding scale, and output brightness features at multiple different scales;

[0009] Input the target semantic features and each brightness feature into the color decoder for decoding at the corresponding scale, and output color features at multiple different scales;

[0010] Splice the brightness features and color features at a preset scale and output the target color image corresponding to the event image;

[0011] The target object in the target color image is identified to obtain a target recognition result.

[0012] In some embodiments, the event images include positive event images and negative event images;

[0013] Extract event features from event images, including:

[0014] Performing feature extraction on the positive event image to obtain a plurality of positive event features, and performing feature extraction on the negative event image to obtain a plurality of negative event features;

[0015] The positive event features and negative event features are fused to obtain the event features.

[0016] In some embodiments, the intermediate features include a first intermediate feature and a second intermediate feature;

[0017] The target semantic features and each intermediate feature are input into the brightness decoder for decoding at the corresponding scale, and brightness features at multiple different scales are output, including:

[0018] Deconvolution is performed on the target semantic feature based on the brightness decoder to obtain an initial first brightness feature;

[0019] Based on the first intermediate feature at the corresponding scale, the initial first brightness feature is decoded to obtain a first brightness feature;

[0020] Performing deconvolution processing on the first brightness feature to obtain an initial second brightness feature;

[0021] Based on the second intermediate feature at the corresponding scale, the initial second brightness feature is decoded to obtain the second brightness feature.

[0022] In some embodiments, based on the first intermediate feature at the corresponding scale, the initial first brightness feature is decoded to obtain the first brightness feature, including:

[0023] The initial first brightness feature and the first intermediate feature at the corresponding scale are concatenated, and the concatenated result is processed based on the attention module of the event image processing model to obtain the first sub-feature;

[0024] The results of performing global average pooling processing and activation processing on the first sub-feature are added respectively, and the XOR result of the added result and the first sub-feature is calculated to obtain an updated first sub-feature;

[0025] The updated first sub-feature and the initial first brightness feature are superimposed to obtain a first brightness feature.

[0026] In some embodiments, the target semantic features and each brightness feature are input into a color decoder for decoding processing at corresponding scales, and color features at multiple different scales are output, including:

[0027] Deconvolution is performed on the target semantic feature based on the color decoder to obtain an initial first color feature;

[0028] Based on the first brightness feature at the corresponding scale, the initial first color feature is decoded to obtain the first color feature;

[0029] Performing deconvolution processing on the first color feature to obtain an initial second color feature;

[0030] Based on the second brightness feature at the corresponding scale, the initial second color feature is decoded to obtain the second color feature.

[0031] In some embodiments, decoding the initial first color feature based on the first brightness feature at the corresponding scale to obtain the first color feature includes:

[0032] Performing a deconvolution process on the superimposed first brightness feature and the initial first color feature at the corresponding scale to obtain a deconvolution result;

[0033] The deconvolution result is regularized to obtain a regularized result, and the regularized result is activated to obtain a first color feature.

[0034] To achieve the above purpose, a second aspect of an embodiment of the present application proposes a training method for a processing model, the method comprising:

[0035] Acquire a sample event image, and input the sample event image into an initial event image processing model;

[0036] Extracting sample event features from the sample event image, and inputting the sample event features into an encoder for encoding processing at multiple different scales, and outputting sample intermediate features at multiple different scales;

[0037] Perform residual processing on the sample intermediate features output by the last convolutional layer to obtain sample semantic features, and input the sample semantic features and each sample intermediate feature into the brightness decoder for decoding processing at the corresponding scale, and output sample brightness features at multiple different scales;

[0038] The sample semantic features and each sample brightness feature are input into the color decoder for decoding processing at the corresponding scale, and the sample color features at multiple different scales are output;

[0039] Splice the sample brightness features and sample color features at a preset scale, output a sample color image corresponding to the sample event image, and identify the sample object in the sample color image to obtain a sample identification result;

[0040] A sample loss value is calculated based on the sample recognition result, and model parameters of the event image processing model are adjusted based on the sample loss value to obtain a trained event image processing model.

[0041] In some embodiments, splicing the sample brightness features and the sample color features at a preset scale to output a sample color image corresponding to the sample event image includes:

[0042] The sample brightness features and sample color features at a preset scale are stitched together to obtain a sample color image in the LAB color space;

[0043] The sample color image is subjected to nonlinear transformation processing to obtain an updated sample color image in the RGB color space.

[0044] In some embodiments, the sample loss value is calculated based on the sample recognition result, including:

[0045] Acquire a sample verification image corresponding to the sample event image, wherein the sample event image and the sample verification image include the same sample object;

[0046] Identify the sample object in the sample verification image to obtain the sample verification result, and compare the difference between the sample recognition result and the sample verification result to obtain the cross entropy loss value;

[0047] Performing a feature extraction operation on the sample color image to obtain a first semantic feature, and performing a feature extraction operation on the sample verification image to obtain a second semantic feature;

[0048] Compare the difference between the first semantic feature and the second semantic feature to obtain a semantic alignment loss value;

[0049] Performing logarithmic operation on the first semantic feature and the second semantic feature to obtain a contrastive learning loss value;

[0050] The cross entropy loss, semantic alignment loss, and contrastive learning loss are superimposed to obtain the sample loss.

[0051] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application provides a device for processing event images, the device comprising:

[0052] An acquisition module, used for acquiring event images and inputting the event images into a pre-trained event image processing model;

[0053] A feature extraction module is used to extract event features from the event image, and input the event features into an encoder for encoding processing at multiple different scales, and output intermediate features at multiple different scales;

[0054] The brightness feature determination module is used to perform residual processing on the intermediate features output at the last scale to obtain the target semantic features, and input the target semantic features and each intermediate feature into the brightness decoder for decoding processing at the corresponding scale, and output brightness features at multiple different scales;

[0055] A color feature determination module is used to input the target semantic features and each brightness feature into a color decoder for decoding processing at a corresponding scale, and output color features at multiple different scales;

[0056] An output module is used to stitch brightness features and color features at a preset scale and output a target color image corresponding to the event image;

[0057] The recognition module is used to recognize the target object in the target color image and obtain the target recognition result.

[0058] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the event image processing method of the first aspect or the processing model training method of the second aspect.

[0059] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the event image processing method of the first aspect or the processing model training method of the second aspect.

[0060] The event image processing method, processing model training method and related equipment proposed in the present application obtain an event image and input the event image into a pre-trained event image processing model; extract event features in the event image, and input the event features into an encoder for encoding processing at multiple scales, and output intermediate features at multiple scales; by encoding event features at different scales, multiple scale information in the image can be captured, including global structure and local details. This multi-scale fusion method can significantly improve the richness of features, thereby enabling the processing model to more comprehensively understand the information content in the event image, thereby improving the accuracy of the decoding results obtained by the subsequent decoder output; residual processing is performed on the intermediate features output at the last scale to obtain the target semantic features, and the target semantic features and each intermediate feature are input into a brightness decoder for decoding processing at the corresponding scale, and multiple scales are output. The target semantic features and each brightness feature are input into a color decoder for decoding processing at the corresponding scale, and color features at multiple different scales are output; the brightness features and color features at the preset scale are spliced ​​to output a target color image corresponding to the event image; since color and brightness are two important and relatively independent attributes in an image, specifically, brightness reflects the brightness of an image, while color is related to the specific wavelength of light reflected or emitted from the surface of an object, the processing of color and brightness is separated into two independent decoders, and based on the color decoder and the brightness decoder, a high-quality target color image that meets the requirements of the pre-trained model processing is generated with minimal computing power resources; thereafter, the target object in the target color image is identified to obtain a target recognition result; since the target recognition result is obtained based on the pre-trained model recognition in the related art, the present application improves the accuracy of identifying the target object in the event image. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a schematic diagram of an optional application scenario of the event image processing device provided in an embodiment of the present application;

[0062] Figure 2 is an optional flowchart of the method for processing event images provided in an embodiment of the present application;

[0063] Figure 3 is an optional schematic diagram of an event image processing model provided in an embodiment of the present application;

[0064] Figure 4 is an optional flowchart of the training method of the processing model provided in the embodiment of the present application;

[0065] Figure 5 This is an optional processing schematic diagram of the training method of the processing model provided in the embodiment of the present application;

[0066] Figure 6 is an optional schematic diagram of comparing a sample image with a verification image provided in an embodiment of the present application;

[0067] Figure 7 It is a schematic diagram of an optional functional module of the event image processing device provided in an embodiment of the present application;

[0068] Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0072] First, some nouns involved in this application are analyzed:

[0073] RGB Color Image (RGB image for short), where RGB is the abbreviation of Red, Green, and Blue, representing the three color channels in the image. These three color channels can generate almost all colors through the combination of different intensities, thus forming a colorful image.

[0074] S-shaped growth curve (Sigmoid activation function), the Sigmoid function is often used as an activation function of a neural network, which maps the variable to between 0 and 1 and interprets the output as the probability that the sample belongs to a certain category as the output of the hidden layer neuron.

[0075] Broadcasting addition between matrices is an important concept in matrix operations, which allows addition operations to be performed between matrices of unequal shapes or between a matrix and a vector.

[0076] ReLU function (Rectified Linear Unit function) is a commonly used activation function. The output of ReLU function is: f(x) = max(0,x), that is, when x is greater than or equal to 0, the output is x, otherwise the output is 0. ReLU function has the advantages of simple calculation, fast training speed, and stable convergence.

[0077] The event camera is a new type of bionic asynchronous sensor. Different from the instantaneous static images captured by traditional cameras, the event camera captures "events". "Events" can be simply understood as "changes in pixel brightness", that is, the event camera outputs changes in pixel brightness.

[0078] In the related art, in order to realize the recognition of event images, it is necessary to retrain the pre-built model based on a large number of event image training sets and verification sets (collectively referred to as "event image datasets"). However, due to the working principle and output characteristics of event cameras, the number of event image datasets is usually very limited. Therefore, the trained model cannot recognize the target objects in the event images well, which makes the recognition results corresponding to the target objects output by the model have poor accuracy.

[0079] Based on this, the embodiments of the present application provide a method for processing event images, a method for training a processing model, and related equipment, aiming to improve the accuracy of identifying target objects in event images.

[0080] The event image processing method, processing model training method and related equipment provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the event image processing method in the embodiments of the present application is described.

[0081] It should be noted that in the embodiments of the present application, when it comes to information related to user characteristics such as user basic information or user identity, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or consent will be obtained first. After the user's separate permission or consent is clearly obtained, the necessary data for the normal operation of the embodiments of the present application will be obtained. For example, when the embodiments of the present application obtain event images or sample event images taken by the user, the consent of the relevant user will be obtained first; or, if the obtained event images or sample event images involve privacy including but not limited to the user's portrait privacy, personal information, etc., the consent of the relevant user will be obtained first, otherwise the event images or sample event images used in the embodiments of the present application cannot be obtained.

[0082] The event image processing method, processing model training method and related equipment provided in the embodiments of the present application are specifically explained through the following embodiments. First, the application scenario of the event image processing device in the embodiments of the present application (for the convenience of description, it may also be referred to as "processing device" below) is described, wherein a pre-trained event image processing model is provided in the processing device, and the event image processing model (for the convenience of description, it may also be referred to as "processing model" below) is trained based on the training method of the processing model proposed in the present application. The trained event image processing model can use the event image processing method proposed in the embodiments of the present application to process the input event image, such as Figure 1 As shown, Figure 1 This is a schematic diagram of an optional application scenario of the event image processing device provided in the embodiment of the present application. First, the acquired event image is input into the processing device provided in the embodiment of the present application. The processing device can process the input event image based on the event image processing model and generate a sample color image corresponding to the event image; the target object in the sample color image is identified to obtain a target identification result. It can be understood that the accuracy of the target identification result output by the processing model is relatively high, that is, the processing model can have a good identification ability for the target object in the event image, and the performance is even better when the event images are input in batches and in large quantities.

[0083] After understanding the application scenarios of the processing device proposed in the present application, the event image processing method proposed in the embodiment of the present application is described next.

[0084] In the embodiments of the present application, the description will be made from the perspective of a processing device, which can be integrated into a computer device, such as a server. Figure 2 As shown, Figure 2is an optional flowchart of the method for processing event images provided in an embodiment of the present application. Figure 2 The method may include but is not limited to the following steps 101 to 106. When the processing device executes the method for processing event images, the specific process is as follows. It should be noted that the present embodiment Figure 2 The order of step 101 to step 106 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0085] Step 101, obtain an event image, and input the event image into a pre-trained event image processing model.

[0086] The following is a detailed description of step 101.

[0087] In some embodiments, Figure 3 As shown, Figure 3 is an optional schematic diagram of an event image processing model provided in an embodiment of the present application, Figure 3 The portion enclosed by the dotted line frame in the figure is used to represent the processing model. After the event image is acquired, the event image is input into the Figure 3 In the process shown, the event image is processed after the processing model is processed to generate a target color image corresponding to the event image.

[0088] The event image can be captured by an event camera. An event camera is a visual sensor that can sense and respond to changes in pixel brightness. Unlike traditional cameras, an event camera does not output complete image frames at a constant rate, but only outputs information related to local pixel-level brightness changes, which are called "events."

[0089] Alternatively, the event image can be extracted from a recorded video by a feature extractor. For example, a video object detection and classification network (such as SlowFast) can be used as a feature extractor to classify the actions in the video, extract key frames related to specific events, and stitch multiple events together to form an event image; alternatively, the event image can be obtained by feature extraction based on multimodal information (such as text and images); furthermore, in addition to being obtained based on personal processing by the user, the event image can also be obtained from an open database. That is, there are many ways to obtain event images, and the specific method of acquisition can be selected according to actual conditions, and the embodiments of the present application do not limit this.

[0090] It should be noted that the working principle of the event camera is that each pixel has an independent computing circuit to monitor changes in light intensity. Only when the brightness change of the pixel exceeds the preset threshold will an event be triggered. This asynchronous monitoring mechanism causes the camera to generate output only when there is a change, rather than taking a complete RGB image like a traditional camera. In addition, the output characteristics of the event camera are that the event camera only outputs information related to brightness changes, such as timestamps, pixel coordinates, and polarity (brightness increases or decreases), without background or other unchanged pixel information. This output method greatly reduces the redundancy of data. Therefore, limited by the working principle and output characteristics of the event camera, the number of event images is relatively small compared to the number of RGB images.

[0091] Furthermore, the event image is obtained by accumulating and summarizing events at multiple moments within a period of time, and an event stream is formed according to the time sequence of the received events. According to the different event types, the event stream further includes a positive event stream and a negative event stream. After capturing the positive and negative event streams, they can be converted into event images under the corresponding event types based on the following formula (1):

[0092]

[0093] Specifically, for a positive / negative event stream Where N is the number of events, and each event e k ∈ε can be represented by a quaternion (x i ,y i ,t i ,p i ) means, x i and i represents the spatial position, t i Indicates the timestamp, p i represents polarity; δ represents Kronecker function.

[0094] Further, according to The polarity p i (where polarity p i = +1 or -1) can be used to classify events as positive events and negative events And convert them into positive event images p t ∈R H×W and negative event image n t ∈R H ×W , where H and W represent the height and width of the event image under the corresponding event type respectively.

[0095] In some embodiments, extracting event features from an event image includes the following steps 201 to 202:

[0096] Step 201 , extracting features from a positive event image to obtain a plurality of positive event features, and extracting features from a negative event image to obtain a plurality of negative event features.

[0097] Step 202: fuse the positive event features and the negative event features to obtain event features.

[0098] Steps 201 to 202 are described in detail below.

[0099] In some embodiments, Figure 3 As shown, event streams of different event types are stacked to obtain event images, and the event images include positive event images and negative event images respectively; the positive event images and negative event images are both input into the trained processing model, and the processing model performs feature extraction processing on the positive event images and the negative event images respectively, and correspondingly obtains multiple positive event features and multiple negative event features; then, the extracted positive event features and multiple negative event features are fused using the cross-attention module in the processing model to obtain event features; so that the processing model can encode the event features afterwards, and the encoded results are input into the decoder set in the processing model for subsequent processing, thereby generating a target color image.

[0100] Furthermore, the positive event image and the negative event image are convolved (Conv) by the convolution layer set in the processing model to obtain the positive event feature F p ∈R C×H×W and negative event feature F n ∈R C×H×W , where C represents the number of channels, and C can usually be set to 32; of course, the number of channels can be set according to actual conditions, and the embodiments of the present application do not limit this.

[0101] Furthermore, the positive and negative event features are fused through the Cross-Attention Module set in the processing model. The Cross-Attention Module is an attention mechanism that is mainly used to process the dependency between two different sequences. The Cross-Attention Module processes the semantic relationship between two different sequences by calculating attention on them, so that one sequence can "pay attention" to another sequence, thereby capturing the relationship between the two sequences. Specifically, the working principle of the Cross-Attention Module is as follows:

[0102] (1) Input mapping: Map the input sequence into three matrices: query, key, and value. These matrices are obtained through linear transformation (such as fully connected layers) and are used for subsequent attention calculations.

[0103] (2) Calculate the attention weight: The attention weight is obtained by calculating the similarity (such as dot product) between the query matrix and the key matrix. The attention weight represents the degree of attention of each element in the query sequence to each element in the key sequence. Then, the attention weight is normalized to ensure that the sum of the attention weights of each query element is 1.

[0104] (3) Calculate the context vector: Use the normalized attention weights to perform weighted summation on the value matrix to obtain a context vector, which contains weighted information from another sequence related to the query sequence.

[0105] Exemplarily, the 1x1 convolutional layer in the cross attention module deployed in the processing device is used to respectively p and F n Perform mapping processing to obtain the first query Q p , first key K p and the first value V p ∈R C×(HW) , and the second query Q n , second key K n and the second value V n ∈R C×(HW) , then the fused event feature F can be obtained by the following formula (2): fuse :

[0106]

[0107] Among them, σ represents the sigmoid activation function, ☉ represents the inner product between two matrices, Indicates Q n The transpose of Indicates Q p The transpose of .

[0108] Step 102: extract event features from the event image, and input the event features into an encoder for encoding processing at multiple different scales, and output intermediate features at multiple different scales.

[0109] The following is a detailed description of step 102.

[0110] In some embodiments, after the event features are obtained based on the cross-attention module, the event features are input into the encoder of the processing model so that the encoder encodes the event features at different scales and obtains intermediate features at multiple different scales. By encoding the event features at different scales, multiple scale information in the image can be captured, including global structure and local details. This multi-scale fusion method can significantly improve the richness of features, thereby enabling the processing model to more comprehensively understand the information content in the event image, thereby improving the accuracy of the decoding results obtained by the subsequent decoder output.

[0111] Furthermore, if Figure 3 As shown, the encoder in the embodiment of the present application includes three encoding processing layers at different scales, which converts the event feature F fuse The event features are input into the encoder for feature encoding, where the encoder can be composed of basic convolution modules, each of which contains a 2D convolution, batch normalization (BN) and RELU activation function; after the event features are processed by convolution, regularization and activation at the encoding processing layer at different scales, three intermediate features are obtained accordingly: The distinction is and

[0112] Among them, the intermediate feature can be a numerical feature or a feature map; that is, the specific form of the intermediate feature can be determined according to the actual situation, and the embodiment of the present application does not limit this. It should also be noted that the number of encoding processing layers of the encoder can also be adaptively adjusted according to the actual situation, and the embodiment of the present application does not limit this.

[0113] Step 103, performing residual processing on the intermediate features output at the last scale to obtain the target semantic features, and inputting the target semantic features and each intermediate feature into a brightness decoder for decoding processing at the corresponding scale, and outputting brightness features at multiple different scales.

[0114] The following is a detailed description of step 103.

[0115] In some embodiments, after obtaining the intermediate features at each scale, the intermediate features obtained at the last scale are input into the residual module in the processing model for residual processing to extract high-level semantic features and obtain target semantic features. Among them, high-level semantic information refers to a deeper, more abstract and complex semantic representation that goes beyond low-level semantic information such as basic vocabulary, syntax and grammar. High-level semantic information not only includes the literal meaning in the text or image, but also covers logical reasoning, semantic association, emotional color, cultural background and other richer connotations; the processing model obtains intermediate features containing high-level semantic information through feature encoding, so as to have a deeper understanding of the event image.

[0116] Furthermore, the residual module introduces residual connections (also called "skip connections") to enable the input data to directly skip one or more layers and be added to the output data, thereby alleviating the gradient vanishing / exploding problem and improving the processing performance of the processing model on various tasks.

[0117] For example, Figure 3 As shown, the intermediate features obtained at the last scale are Input into the residual module to extract features and obtain target semantic features The target semantic features will be subsequently input into the decoder to participate in feature decoding.

[0118] In some embodiments, the target semantic feature and each intermediate feature are input into a brightness decoder for decoding processing at corresponding scales, and brightness features at multiple different scales are output, including the following steps 301 to 304:

[0119] Step 301 : Deconvolution is performed on the target semantic feature based on the brightness decoder to obtain an initial first brightness feature.

[0120] Step 302: Based on the first intermediate feature at the corresponding scale, the initial first brightness feature is decoded to obtain the first brightness feature.

[0121] Steps 301 to 302 are described in detail below.

[0122] In some embodiments, after obtaining the target semantic features and each intermediate feature, the target semantic features and each intermediate feature are input into a brightness decoder of the processing model; wherein the brightness decoder includes brightness decoding processing layers at different scales, and the number and scale of the brightness decoding processing layers correspond to the number and scale of the encoder processing layers, such as Figure 3 As shown, the encoder includes encoding processing layers at three scales, and the decoder also includes brightness decoding processing layers at three scales.

[0123] Among them, Figure 3As shown, for the encoder, from left to right are the encoding processing layer at the first scale, the encoding processing layer at the second scale, and the encoding processing layer at the third scale (the last scale), and the intermediate feature output by the encoding processing layer at the last scale is called the first intermediate feature; for the brightness / color decoder, from left to right are the brightness / color decoding processing layer at the third scale, the brightness / color decoding processing layer at the second scale, and the brightness / color decoding processing layer at the first scale. The processing layers at the same scale correspond one to one, and the higher the scale, the more abstract the features obtained by the corresponding processing.

[0124] Furthermore, the feature output by the deconvolution processing of the target semantic feature by the brightness decoding processing layer at the first scale is called the initial first brightness feature. Of course, the number of encoding / brightness decoding processing layers at each scale of the processing model can be adjusted according to actual conditions, and the embodiment of the present application does not limit this.

[0125] Furthermore, the brightness decoder is used to decode the information representing the brightness; a feature transfer module is also included between the encoder and the brightness decoder, and the feature transfer module is used to realize feature fusion and transmission across layers or time points, and to promote knowledge sharing and complementarity among multiple scales by integrating information at different time steps or different scales, so as to enhance the feature representation capability of the processing model, thereby helping to improve the accuracy and efficiency of the output results of subsequent decoders.

[0126] Next, combine Figure 3 Further explain how the feature transfer module and the brightness decoder cooperate to decode the intermediate features at each scale to obtain the brightness features at the corresponding scale:

[0127] First, the brightness decoding processing layer at the first scale of the decoder performs deconvolution on the target semantic features to obtain the initial first brightness feature Next, the initial first brightness feature is decoded using the first intermediate feature obtained after being processed by the feature transfer module to obtain the first brightness feature at the corresponding scale.

[0128] In some embodiments, based on the first intermediate feature at the corresponding scale, the initial first brightness feature is decoded to obtain the first brightness feature, including the following steps 401 to 403:

[0129] Step 401, concatenate the initial first brightness feature and the first intermediate feature at the corresponding scale, and process the concatenated result based on the attention module of the event image processing model to obtain the first sub-feature.

[0130] Step 402 , respectively add the results of performing global average pooling processing and activation processing on the first sub-feature, and calculate the XOR result of the added result and the first sub-feature to obtain an updated first sub-feature.

[0131] Step 403: superimpose the updated first sub-feature and the initial first brightness feature to obtain the first brightness feature.

[0132] Steps 401 to 403 are described in detail below.

[0133] In some embodiments, the initial brightness features at each scale and the intermediate features at the corresponding scale are processed based on the following equations (3) and (4) to obtain the brightness features at the corresponding scale:

[0134]

[0135] in, represents the initial brightness feature at the corresponding scale, i represents the corresponding scale; C 1×1 Represents a 1x1 convolution operation; represents the broadcast sum between matrices, GAP represents global average pooling, which is used to obtain global attention; ReLU is the ReLU activation operation.

[0136] In order to better understand step 401 to step 403, the processing of the initial first brightness feature and the first intermediate feature at the corresponding scale is taken as an example for explanation. After the initial first brightness feature and the first intermediate feature at the corresponding scale are obtained:

[0137] (1) Concatenate the initial first brightness feature and the first intermediate feature

[0138] (2) Use the attention module (FPM) in the processing model to process the concatenated result and obtain the first sub-feature F ′ ; Among them, the attention module processes the concatenated result by a convolution operation (C 1×1 ), regularization operation (BN) and activation operation (ReLU);

[0139] (3) After obtaining the first sub-feature F ′ After that, first of all, ′ Perform convolution to get C 1×1 (F ′ ); Then, C 1×1 (F ′ ) are input into two different processing layers, one of which processes C 1×1 (F ′ ) performs global average pooling and obtains GAP(C1×1 (F ′ )); Another processing layer for C 1×1 (F ′ ) is activated to obtain ReLU (C 1×1 (F ′ ));

[0140] (4) Then, the activated result can be convolved again to obtain C 1×1 (ReLU(C 1×1 (F ′ )0), for GAP(C 1×1 (F ′ )) and C 1×1 (ReLU(C 1×1 (F ′ )00 is added to activate (σ), and we get

[0141] (5) Calculate the XOR result of the added result and the first sub-feature (☉) to obtain the updated first sub-feature

[0142] (6) Superimpose the updated first sub-feature and the initial first brightness feature Get the first brightness feature

[0143] Step 303: perform deconvolution processing on the first brightness feature to obtain an initial second brightness feature.

[0144] Step 304: Based on the second intermediate feature at the corresponding scale, the initial second brightness feature is decoded to obtain a second brightness feature.

[0145] Steps 303 to 304 are described in detail below.

[0146] In some embodiments, after obtaining the first brightness feature, the brightness decoder passes it to the brightness decoding processing layer at the second scale for deconvolution processing to obtain an initial second brightness feature Next, based on the second intermediate feature at the corresponding scale, the initial second brightness feature is decoded to obtain the second brightness feature. The method for obtaining the second brightness feature is similar to step 401 to step 403, which will not be repeated here.

[0147] Furthermore, the embodiments of the present application show Figure 3 The third intermediate feature will also be output based on the third brightness decoding processing layer. The method for obtaining the third intermediate feature is similar to the method for obtaining the first intermediate feature and the second intermediate feature, and will not be repeated here.

[0148] Step 104: input the target semantic features and each brightness feature into a color decoder for decoding processing at corresponding scales, and output color features at multiple different scales.

[0149] Step 104 is described in detail below.

[0150] In some embodiments, the residual module processes the target semantic features Afterwards, in addition to In addition to being input to the brightness decoder, The color decoder is input into the color decoder so that the color decoder decodes the input content to obtain color features at different scales.

[0151] In some embodiments, the target semantic feature and each brightness feature are input into a color decoder for decoding processing at corresponding scales, and color features at multiple different scales are output, including the following steps 501 to 504:

[0152] Step 501 : Deconvolution is performed on the target semantic feature based on the color decoder to obtain an initial first color feature.

[0153] Step 502: Based on the first brightness feature at the corresponding scale, the initial first color feature is decoded to obtain the first color feature.

[0154] Step 503: perform deconvolution processing on the first color feature to obtain an initial second color feature.

[0155] Step 504: Based on the second brightness feature at the corresponding scale, the initial second color feature is decoded to obtain a second color feature.

[0156] Steps 501 to 504 are described in detail below.

[0157] In some embodiments, after obtaining the target semantic features and each brightness feature, the target semantic features and each brightness feature are input into the color decoder of the processing model; wherein the color decoder includes color decoding processing layers at different scales, and the number and scale of the color decoding processing layers correspond to the number and scale of the encoder processing layers, such as Figure 3 As shown, the encoder includes encoding processing layers at three scales, and the decoder also includes color decoding processing layers at three scales.

[0158] Furthermore, the color decoder is used to decode the information representing the color; next, combined with Figure 3 Further explain how the brightness decoder and the color decoder cooperate to decode the brightness features at each scale to obtain the color features at the corresponding scale:

[0159] First, the color decoding processing layer at the first scale of the decoder performs deconvolution on the target semantic features to obtain the initial first color feature Next, the initial first color feature is decoded based on the first brightness feature output by the brightness decoder to obtain the first color feature at the corresponding scale.

[0160] In some embodiments, after obtaining the first color feature, the color decoder passes it to the color decoding processing layer at the second scale for deconvolution processing to obtain an initial second color feature Next, based on the second brightness feature at the corresponding scale, the initial second color feature is decoded to obtain the second color feature. The method for obtaining the second color feature is similar to step 501 to step 502, and will not be repeated here.

[0161] Furthermore, the embodiments of the present application show Figure 3 A third brightness feature will also be output based on the third color decoding processing layer. The method for obtaining the third brightness feature is similar to the method for obtaining the first brightness feature and the second brightness feature, and will not be repeated here.

[0162] In some embodiments, based on the first brightness feature at the corresponding scale, the initial first color feature is decoded to obtain the first color feature, including the following steps 601 to 602:

[0163] Step 601 , performing a deconvolution process on the superimposed first brightness feature and the initial first color feature at the corresponding scale to obtain a deconvolution result.

[0164] Step 602: Regularize the deconvolution result to obtain a regularized result, and activate the regularized result to obtain a first color feature.

[0165] Steps 601 to 602 are described in detail below.

[0166] In some embodiments, the initial color features at each scale and the brightness features at the corresponding scale are processed by the following formula (5) to obtain the color features at the corresponding scale:

[0167]

[0168] Among them, DeConv represents the deconvolution operation, Represents the color features at different scales obtained by the color decoder.

[0169] In order to better understand step 601 to step 602, the processing of the initial first color feature and the first color feature at the corresponding scale is taken as an example for explanation. After the initial first color feature and the first brightness feature at the corresponding scale are obtained:

[0170] (1) Superimpose the initial first color feature and the first brightness feature And perform deconvolution processing (DeConv) on the superimposed result to obtain the deconvolution result

[0171] (2) Regularize the deconvolution result to obtain the regularized result

[0172] (3) Activate the regularized result to obtain the first brightness feature at the corresponding scale

[0173] Step 105 , concatenating brightness features and color features at a preset scale, and outputting a target color image corresponding to the event image.

[0174] The following is a detailed description of step 105.

[0175] In some embodiments, the preset scale is a first scale. That is, for a brightness decoder, a brightness image is obtained based on the brightness features output by the last brightness decoding processing layer (first scale); for a color decoder, a color image is obtained based on the color features output by the last color decoding processing layer (first scale); then, the brightness image and the color image are spliced ​​to obtain a target color image corresponding to the event image.

[0176] It can be understood that since color and brightness are two important and relatively independent attributes in an image, specifically, brightness reflects the degree of light and darkness of an image, while color is related to the specific wavelength of light reflected or emitted from the surface of an object, the processing of color and brightness is separated into two independent decoders, and based on the color decoder and the brightness decoder, a high-quality target color image that meets the processing requirements of the pre-trained model is generated with minimal computing resources.

[0177] Step 106, identifying the target object in the target color image to obtain a target recognition result.

[0178] Step 106 is described in detail below.

[0179] In some embodiments, after obtaining the target color image, the target object in the target color image is identified using a pre-trained model to obtain a target identification result corresponding to the input event image. It is understandable that the pre-trained model refers to a model that can identify the target object in the RGB image. The model is trained in advance based on a large number of sample RGB images, so it can have a good recognition ability for the target object in the RGB image. For example, the pre-trained model can be a visual processing model (Vision Transformer, VIT model); the embodiment of the present application converts the event image into an RGB image, and uses the trained model in the relevant technology to identify the target object in the target color image, so as to improve the recognition accuracy of the target object in the event image.

[0180] Among them, the target object (Object) refers to the specific entity or structure that the user or system is interested in. These target objects can be any form of physical objects, including but not limited to people, animals, vehicles, buildings, furniture, plants, tools, signs, symbols, texture areas, etc. The processing model can identify the object with the highest area in the target color image according to actual needs, or it can identify all objects appearing in the target color image and output all recognition results as target recognition results. It can be adaptively adjusted according to actual conditions, and the embodiments of this application are not limited.

[0181] It should be noted that the identification of the target object refers to the identification of its type. For example, the identification of the target object can be accurate to the level of "fish" or to the level of "tilapia". The specific method can be adjusted according to the actual situation, and different pre-training models can be selected accordingly. The embodiment of the present application does not limit the accuracy of the target object identification.

[0182] Furthermore, different types of pre-trained models can be selected according to actual needs and connected to the processing model in the embodiment of the present application to complete the recognition processing of the target object in the target color image. Among them, different types of pre-trained models refer to models trained based on different tasks and data types, and different tasks refer to different emphases of the model on the recognition of RGB images. For example, pre-trained model a is suitable for the recognition of plant target objects, and pre-trained model b is suitable for the recognition of animal target objects.

[0183] like Figure 4 As shown, Figure 4 is an optional flowchart of a training method for a processing model provided in an embodiment of the present application (for ease of description, hereinafter referred to as “training method”). Figure 4The method in the embodiment may include but is not limited to the following steps 701 to 706. When the processing model is trained based on the training method, the specific process is as follows. It should be noted that the present embodiment Figure 3 The order of step 701 to step 706 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0184] Step 701, obtaining a sample event image, and inputting the sample event image into an initial event image processing model;

[0185] Step 702: extract sample event features from the sample event image, and input the sample event features into an encoder for encoding processing at multiple different scales, and output sample intermediate features at multiple different scales.

[0186] Step 703, performing residual processing on the sample intermediate features output by the last convolutional layer to obtain sample semantic features, and inputting the sample semantic features and each sample intermediate feature into a brightness decoder for decoding processing at the corresponding scale, and outputting sample brightness features at multiple different scales.

[0187] Step 704: input the sample semantic features and each sample brightness feature into a color decoder for decoding processing at a corresponding scale, and output sample color features at multiple different scales.

[0188] Step 705 , splicing the sample brightness features and the sample color features at a preset scale, outputting a sample color image corresponding to the sample event image, and identifying the sample object in the sample color image to obtain a sample identification result.

[0189] Steps 701 to 705 are described in detail below.

[0190] In some embodiments, in order to better use the processing model proposed in the embodiments of the present application in practical applications, before the event image is input into the processing model, the pre-built initial processing model is first trained. The method of obtaining the sample event image for training the processing model is similar to step 101, and the method of the processing model processing the sample event image and outputting the sample recognition result is similar to steps 102 to 105, which will not be repeated here.

[0191] In some embodiments, splicing the sample brightness features and the sample color features at a preset scale and outputting the sample color image corresponding to the sample event image includes the following steps 801 to 802:

[0192] Step 801 , concatenate sample brightness features and sample color features at a preset scale to obtain a sample color image in a LAB color space.

[0193] Step 802: Perform nonlinear conversion processing on the sample color image to obtain an updated sample color image in the RGB color space.

[0194] In some embodiments, the preset scale is the first scale. That is, for a brightness decoder, a sample brightness image is obtained based on the sample brightness features output by the last brightness decoding processing layer (first scale); for a color decoder, a sample color image is obtained based on the sample color features output by the last color decoding processing layer (first scale); then, the sample brightness image and the sample color image are spliced ​​to obtain a sample color image in the LAB color space corresponding to the sample event image.

[0195] Among them, the LAB color space is a color space based on human visual perception, with a wide color gamut, which contains all the colors that the human eye can see. The LAB color space consists of three components:

[0196] (1) L*: stands for luminance. Its value range is generally 0 to 100, where 0 represents black and 100 represents white. The L* component is closely related to the brightness perception of human vision and can better reflect the changes in color brightness.

[0197] (2) a*: represents the component from green to red. Its value range is usually -128 to +127 (or -100 to +100 in some cases), where negative values ​​represent green and positive values ​​represent red. By adjusting the a* component, the color can be adjusted in the red-green direction.

[0198] (3) b*: represents the component from blue to yellow. Its value range is also -128 to +127 (or -100 to +100 in some cases), where negative values ​​represent blue and positive values ​​represent yellow. By adjusting the b* component, the color can be adjusted in the yellow-blue direction.

[0199] Furthermore, in order to improve the recognition accuracy of sample objects in event images, the images in the LAB color space can be converted into sample color images in the RGB color space to improve the accuracy of sample recognition results obtained during the training process, thereby improving the performance of the processing model obtained through training.

[0200] Among them, RGB color space is a color standard, which obtains the corresponding image by mixing red, green, and blue. RGB color space can describe and generate more than 16 million colors in digital form (for the common 8-bit / channel case, that is, 2563 combinations), that is, RGB color space can perceive almost all colors. RGB color space consists of three components:

[0201] (1) Red: In the RGB color space, the red component is used to represent the intensity of red in the RGB image. When the value of the red component is higher, the image will present a stronger red tone;

[0202] (2) Green: The green component represents the intensity of green in the RGB image. A higher green value will make the image tend to be green.

[0203] (3) Blue: The blue component represents the intensity of blue in the RGB image. Increasing the blue value will make the image appear darker blue.

[0204] Step 706, a sample loss value is calculated based on the sample recognition result, and model parameters of the event image processing model are adjusted based on the sample loss value to obtain a trained event image processing model.

[0205] Step 706 is described in detail below.

[0206] In some embodiments, in order to make the final trained processing model have better image feature processing capabilities, it is also necessary to calculate the sample loss value based on the output sample recognition results, so that the model parameters in the corresponding processing model can be adjusted based on the sample loss value to obtain the processing model after training.

[0207] Among them, model parameters include model learning parameters, model simulation parameters, and other model parameters. Furthermore, model learning parameters include but are not limited to hyperparameters, weight parameters, and bias parameters; hyperparameters are used to control the learning process and affect the capacity, regularization, and optimization strategies of the model, including but not limited to learning rate, batch size, number of layers, activation function, etc.; weight parameters are used to characterize the intensity or importance assigned to different features or inputs, and different weight parameters can be set for different processing layers; bias parameters are used to explain any systematic errors or differences between predicted values ​​and true values ​​to help the processing model capture the overall trend or deviation in the data.

[0208] Furthermore, the model simulation parameters include but are not limited to model geometric parameters and physical and chemical parameters. Model geometric parameters are used to characterize the boundary position of the processing model, the thickness of the simulation unit, etc.; physical and chemical parameters: such as temperature parameters, porosity, etc.; external action parameters: such as initial conditions, boundary conditions, etc. Other model parameters include other parameters that the user or system may be concerned about, such as character portrait parameters, object feature parameters, etc. It should be noted that the model parameters can be adaptively adjusted according to actual conditions, and the embodiments of the present application do not limit this.

[0209] In some embodiments, calculating the sample loss value based on the sample identification result includes the following steps 901 to 906:

[0210] Step 901: Acquire a sample verification image corresponding to a sample event image, wherein the sample event image and the sample verification image include the same sample object.

[0211] Step 902: Identify the sample object in the sample verification image to obtain a sample verification result, and compare the difference between the sample identification result and the sample verification result to obtain a cross entropy loss value.

[0212] Step 903: Perform a feature extraction operation on the sample color image to obtain a first semantic feature, and perform a feature extraction operation on the sample verification image to obtain a second semantic feature.

[0213] Step 904 : compare the difference between the first semantic feature and the second semantic feature to obtain a semantic alignment loss value.

[0214] Step 905: Perform logarithmic operation on the first semantic feature and the second semantic feature to obtain a contrastive learning loss value.

[0215] Step 906: superimpose the cross entropy loss value, the semantic alignment loss value, and the contrastive learning loss value to obtain a sample loss value.

[0216] In some embodiments, in order to calculate the sample loss value, it is also necessary to obtain a sample verification image corresponding to the sample event image, wherein the sample verification image is used to represent the real image corresponding to the sample event image in the RGB color space. In addition, the sample event image and the sample verification image include the same sample object. For example, the sample event image and the sample verification image also include "a fish", but "a fish" has different image representations in the sample event image and the sample verification image.

[0217] Furthermore, in order to better train the processing model, it is also necessary to connect the pre-trained models of any type according to the actual situation so that the pre-trained models can recognize the sample objects in the sample color images.

[0218] Furthermore, if Figure 5 As shown, Figure 5 : is an optional processing diagram of the training method of the processing model provided in the embodiment of the present application. Specifically, the sample object in the sample verification image is also identified to obtain a sample verification result; a cross-entropy loss value L representing the difference in recognition results between the sample recognition result and the sample verification result is calculated based on the cross-entropy loss. cls Alternatively, directly set the corresponding true label for the sample object in the sample color image. The true label is used to characterize the true category of the sample object. In this way, the cross entropy loss value L can be calculated based on the sample recognition result and the true label.cls .

[0219] Among them, the cross entropy loss is used to measure the difference between the probability distribution predicted by the model and the probability distribution of the true label. The cross entropy loss is based on the concept of cross entropy in information theory. The model parameters are optimized by minimizing the cross entropy so that the probability distribution predicted by the model is as close as possible to the probability distribution of the true label.

[0220] Furthermore, the image encoder of the CLIP module is used to extract the semantic features of the sample color image and the sample verification image to obtain the first semantic feature and the second semantic feature The mean square error (MSE) is used to align the features between the two, and the semantic alignment loss value L is obtained. mse Among them, MSE is used to measure the difference between the model prediction results and the actual results. The smaller the MSE value, the higher the prediction accuracy of the model, that is, the closer the model's prediction value is to the actual value.

[0221] Among them, the sample color image and the sample verification image are both images containing semantic information. The core idea of ​​the CLIP module is to map the image and descriptive text into the same vector space through contrastive learning, thereby achieving cross-modal understanding and mutual correlation between images and texts.

[0222] Furthermore, the contrastive learning loss value L is calculated using the InfoNCE loss function of the following formula (6): con :

[0223]

[0224] Among them, τ represents the temperature hyperparameter; log is the logarithmic operation processing.

[0225] Furthermore, after calculating the cross entropy loss value, semantic alignment loss value, and contrastive learning loss value, these calculation results are superimposed by the following formula (7) to obtain the sample loss value L total :

[0226] L total =L cls +L mse +L con (7)

[0227] Furthermore, through the sample loss value L total Guide the training of the processing model to pass L totalTo adjust the model parameters to achieve the training purpose. After the processing model is trained, the CLIP module can be removed, leaving only the overall model consisting of the processing model and the pre-trained model. The overall model can be fine-tuned for parameters in downstream tasks without consuming a large amount of computing resources to regenerate the overall model that can recognize event images. This can meet the needs of downstream tasks while avoiding the waste of computing resources.

[0228] Among them, downstream tasks usually refer to tasks for further training or fine-tuning for specific applications or problems after the overall model training is completed. These tasks are designed to use the knowledge and representation capabilities in the pre-trained model to solve specific problems in practice. Taking the ViT model as an example, downstream tasks can include but are not limited to the following:

[0229] (1) Image classification: In image classification, the model needs to identify and classify objects or scenes in the input image;

[0230] (2) Object detection: The object detection task requires the model to locate and identify multiple objects in an image, mark each detected object, and give the object's category label;

[0231] (3) Semantic segmentation: Semantic segmentation is a pixel-level classification task that requires the model to classify each pixel in the image to distinguish different objects or regions.

[0232] (4) Instance segmentation: Instance segmentation is a combination of object detection and semantic segmentation, which requires the overall model to locate and identify multiple objects in the image and draw a separate bounding box or mask for each object;

[0233] It should be noted that the selection and definition of downstream tasks depends on the specific application scenarios and requirements. In practical applications, the downstream tasks can be adaptively adjusted according to actual conditions, and the embodiments of the present application do not limit this.

[0234] like Figure 6 As shown, Figure 6 This is an optional schematic diagram of comparing sample images and verification images provided in an embodiment of the present application. The first horizontal row is sample event images, the second horizontal row is sample color images output after being processed by the processing model, and the third horizontal row is sample verification images representing the real situation corresponding to the sample color images. It can be found that the sample color images and sample verification images processed by the embodiment of the present application have a high degree of similarity, so the pre-trained model connected to the processing model can easily realize the recognition of sample objects.

[0235] It can be understood that in the case of scarcity of event image training samples, the embodiments of the present application can still achieve high-accuracy recognition of target objects in event images by training the processing model, and there is no need to retrain the pre-trained model in the related technology based on the event image. While greatly saving the consumption of computing power resources, it improves the generalization ability of the trained processing model, thereby improving the subsequent recognition accuracy of samples / target objects in event images.

[0236] like Figure 7 As shown, Figure 7 This is an optional functional module diagram of the event image processing device provided in an embodiment of the present application, and the device includes:

[0237] The acquisition module is used to acquire event images and input the event images into a pre-trained event image processing model.

[0238] The feature extraction module is used to extract event features from the event image, and input the event features into the encoder for encoding processing at multiple different scales, and output intermediate features at multiple different scales.

[0239] The brightness feature determination module is used to perform residual processing on the intermediate features output at the last scale to obtain the target semantic features, and input the target semantic features and each intermediate feature into the brightness decoder for decoding processing at the corresponding scale, and output brightness features at multiple different scales.

[0240] The color feature determination module is used to input the target semantic features and each brightness feature into the color decoder for decoding processing at corresponding scales, and output color features at multiple different scales.

[0241] The output module is used to splice the brightness features and color features at a preset scale and output the target color image corresponding to the event image.

[0242] The recognition module is used to recognize the target object in the target color image and obtain the target recognition result.

[0243] The specific implementation of the event image processing device is basically the same as the specific implementation of the above-mentioned event image processing method, and will not be repeated here.

[0244] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned event image processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0245] like Figure 8 As shown, Figure 8: is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, the electronic device includes:

[0246] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0247] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the event image processing method of the embodiment of the present application;

[0248] Input / output interface 1103, used to implement information input and output;

[0249] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0250] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0251] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0252] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned event image processing method is implemented.

[0253] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0254] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0255] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0256] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0257] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0258] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0259] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0260] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0261] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0262] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0263] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0264] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for processing event images, characterized in that: The method comprises: Acquire an event image, and input the event image into a pre-trained event image processing model; Extracting event features from the event image, and inputting the event features into an encoder for encoding processing at multiple different scales, and outputting intermediate features at multiple different scales; Performing residual processing on the intermediate features output at the last scale to obtain target semantic features, and inputting the target semantic features and each of the intermediate features into a brightness decoder for decoding processing at corresponding scales, and outputting brightness features at multiple different scales; Inputting the target semantic feature and each of the brightness features into a color decoder for decoding processing at corresponding scales, and outputting color features at multiple different scales; splicing the brightness features and the color features at a preset scale, and outputting a target color image corresponding to the event image; The target object in the target color image is identified to obtain a target identification result.

2. The method according to claim 1, characterized in that: The event images include positive event images and negative event images; The extracting event features from the event image comprises: Performing feature extraction on the positive event image to obtain a plurality of positive event features, and performing feature extraction on the negative event image to obtain a plurality of negative event features; The positive event feature and the negative event feature are fused to obtain the event feature.

3. The method according to claim 1, characterized in that The intermediate features include a first intermediate feature and a second intermediate feature; The target semantic feature and each of the intermediate features are input into a brightness decoder for decoding at a corresponding scale, and brightness features at multiple different scales are output, including: Performing deconvolution processing on the target semantic feature based on the brightness decoder to obtain an initial first brightness feature; Based on the first intermediate feature at the corresponding scale, decoding the initial first brightness feature to obtain a first brightness feature; Performing deconvolution processing on the first brightness feature to obtain an initial second brightness feature; Based on the second intermediate feature at the corresponding scale, the initial second brightness feature is decoded to obtain a second brightness feature.

4. The method according to claim 3, characterized in that: The decoding process of the initial first brightness feature based on the first intermediate feature at the corresponding scale to obtain the first brightness feature includes: splicing the initial first brightness feature and the first intermediate feature at the corresponding scale, and processing the spliced ​​result based on the attention module of the event image processing model to obtain a first sub-feature; Adding results of performing global average pooling processing and activation processing on the first sub-feature respectively, and calculating the same or the same result of the added result and the first sub-feature, to obtain an updated first sub-feature; The updated first sub-feature and the initial first brightness feature are superimposed to obtain a first brightness feature.

5. The method according to claim 3, characterized in that: The target semantic feature and each brightness feature are input into a color decoder for decoding at a corresponding scale, and color features at multiple different scales are output, including: Performing deconvolution processing on the target semantic feature based on the color decoder to obtain an initial first color feature; Based on the first brightness feature at the corresponding scale, decoding the initial first color feature to obtain a first color feature; Performing a deconvolution process on the first color feature to obtain an initial second color feature; Based on the second brightness feature at the corresponding scale, the initial second color feature is decoded to obtain a second color feature.

6. The method according to claim 5, characterized in that The decoding process of the initial first color feature based on the first brightness feature at the corresponding scale to obtain the first color feature includes: Performing a deconvolution process on the first brightness feature and the initial first color feature superimposed at the corresponding scale to obtain a deconvolution result; The deconvolution result is regularized to obtain a regularized result, and the regularized result is activated to obtain a first color feature.

7. A training method for a processing model, characterized in that: The method comprises: Acquire a sample event image, and input the sample event image into an initial event image processing model; Extracting sample event features from the sample event image, and inputting the sample event features into an encoder for encoding processing at multiple different scales, and outputting sample intermediate features at multiple different scales; Performing residual processing on the sample intermediate features output by the last convolutional layer to obtain sample semantic features, and inputting the sample semantic features and each of the sample intermediate features into a brightness decoder for decoding processing at corresponding scales, and outputting sample brightness features at multiple different scales; Inputting the sample semantic features and each of the sample brightness features into a color decoder for decoding processing at corresponding scales, and outputting sample color features at multiple different scales; splicing the sample brightness feature and the sample color feature at a preset scale, outputting a sample color image corresponding to the sample event image, and identifying the sample object in the sample color image to obtain a sample identification result; A sample loss value is calculated based on the sample recognition result, and model parameters of the event image processing model are adjusted based on the sample loss value to obtain the trained event image processing model.

8. The method according to claim 7, characterized in that The step of splicing the sample brightness features and the sample color features at each scale and outputting a sample color image corresponding to the sample event image includes: Splicing the sample brightness features and the sample color features at each scale to obtain a sample color image in the LAB color space; The sample color image is subjected to nonlinear conversion processing to obtain the updated sample color image in RGB color space.

9. The method according to claim 7, characterized in that: The calculating the sample loss value based on the sample recognition result includes: Acquire a sample verification image corresponding to the sample event image, wherein the sample event image and the sample verification image include the same sample object; Identify the sample object in the sample verification image to obtain a sample verification result, and compare the difference between the sample identification result and the sample verification result to obtain a cross entropy loss value; Performing a feature extraction operation on the sample color image to obtain a first semantic feature, and performing a feature extraction operation on the sample verification image to obtain a second semantic feature; Comparing the difference between the first semantic feature and the second semantic feature to obtain a semantic alignment loss value; Performing logarithmic operation on the first semantic feature and the second semantic feature to obtain a contrastive learning loss value; The cross entropy loss value, the semantic alignment loss value, and the contrastive learning loss value are superimposed to obtain the sample loss value.

10. An event image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring event images and inputting the event images into a pre-trained event image processing model; A feature extraction module, used for extracting event features from the event image, and inputting the event features into an encoder for encoding processing at multiple different scales, and outputting intermediate features at multiple different scales; A brightness feature determination module is used to perform residual processing on the intermediate features output at the last scale to obtain a target semantic feature, and input the target semantic feature and each of the intermediate features into a brightness decoder for decoding processing at a corresponding scale, and output brightness features at multiple different scales; A color feature determination module, used for inputting the target semantic feature and each of the brightness features into a color decoder for decoding processing at a corresponding scale, and outputting color features at multiple different scales; An output module, used for splicing the brightness features and the color features at a preset scale, and outputting a target color image corresponding to the event image; The recognition module is used to recognize the target object in the target color image to obtain a target recognition result.

11. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method for processing event images described in any one of claims 1 to 6 or the method for training a processing model described in any one of claims 7 to 9 when executing the computer program.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method for processing event images described in any one of claims 1 to 6 or the method for training a processing model described in any one of claims 7 to 9.

Images