Data processing method and apparatus

Abstract

A data processing method, which is applied to image processing. The method comprises: acquiring first data collected by an image sensor; on the basis of the first data, obtaining spectral information by means of a neural network model, wherein the neural network model comprises an attention module, and the attention module is used for determining an attention matrix on the basis of input data; and executing an attention operation on the basis of the attention matrix, wherein the attention matrix is obtained by means of performing a first merge operation on correlation information of different channels of the input data and correlation information of the channels themselves. The present application can merge the degree of association between different channels and the degree of association of the channels themselves, such that an attention matrix can simultaneously model the correlation and specificity between different channels, thereby improving the accuracy of reconstruction of a spectral signal.

Description

A data processing method and device thereof

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 5, 2023, with application number 202311138944.0 and application name “A data processing method and device thereof”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and in particular to a data processing method and device thereof. Background Art

[0003] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0004] Widely used cameras and smartphones today are equipped with image sensors. Common image sensors use a color filter array (CFA), also known as a Bayer filter, arranged above the photosensitive area. A common CFA arrangement consists of a 2×2 pixel unit, forming a circular unit. This unit contains the three primary colors: red (R), green (G), and blue (B), so color image sensors are often referred to as RGB image sensors.

[0005] RGB sensors are designed to directly simulate the response of the three types of cone cells (L / M / S) of the human eye to the spectrum. This response is a sampling of the complete spectral information, which cannot fully capture the spectral information and easily causes metamerism problems. As a result, the color reproduction performance of existing RGB cameras is limited in some difficult scenes, such as scenes containing large areas of pure color. Compared with RGB color sensors, multispectral sensors are designed to capture the original spectral signal more completely. The multispectral sensor consists of a 4×4 sixteen pixel unit above the gray photosensitive area, where each pixel corresponds to a color filter (or color filter coating). Through this design, the sensor's spectral resolution is increased from 3 dimensions to 16 dimensions, which can capture the spectral signal more completely.

[0006] However, due to constraints on filter materials, manufacturing processes, and cost, the raw signals (RAW signals) collected by image sensors exhibit aliasing between spectral bands. Therefore, the RAW signals must be analyzed and reconstructed to obtain spectral information. Existing implementations suffer from poor spectral reconstruction accuracy.

[0007] Summary of the Invention

[0008] This application provides a data processing method that can improve the accuracy of spectral signal reconstruction.

[0009] In a first aspect, the present application provides an image processing method, comprising: obtaining first data collected by an image sensor; obtaining spectral information based on the first data through a neural network model; wherein the neural network model includes an attention module, the attention module is used to determine an attention matrix based on the input data, and perform an attention operation based on the attention matrix; the attention matrix is ​​obtained by performing a first fusion operation on the correlation information between different channels of the input data and the correlation information of the channel itself.

[0010] In some implementations, when the sizes of the elements in the attention matrix are close (for example, they are all very small or very large), there is no obvious distinction between different channels, which will lead to poor attention operation effects. Especially in the spectral information reconstruction scenario, each channel can be considered to have a specific physical meaning. Therefore, the attention matrix needs to express the distinction between different channels as much as possible to improve the reconstruction accuracy of the spectral information.

[0011] In an embodiment of the present application, when calculating the attention matrix, the attention module can fuse the correlation between different channels and the correlation between the channels themselves, so that the attention matrix can simultaneously model the correlation and specificity between different channels, thereby improving the accuracy of spectral signal reconstruction.

[0012] Among them, the correlation information of the channel itself can be understood as the information in each channel that is different from other channels, that is, specific information. This information can be obtained not through interaction with other channels, but through interactive operations on the data within the channel (such as dimensionality reduction, convolution, etc.). This information can reflect the distinction between channels.

[0013] Among them, the neural network model can include a feature extraction network (such as an encoder and a decoder) and a downstream task network (such as a mapping network). The attention module can be a module in the feature extraction network used to perform attention operations (for example, the module is an attention head). The feature extraction network can also include other modules, such as addition and normalization layers, and feedforward layers. The feature representation obtained by the feature extraction network can be input into the downstream task network based on the input data (such as data collected by the image sensor) to obtain output (such as spectral information).

[0014] In one possible implementation, the attention module is specifically used to perform a linear transformation on the input data to obtain a K matrix, a Q matrix and a V matrix, and determine the attention matrix based on the K matrix, the Q matrix and the V matrix; the correlation information with the channel itself is determined based on the V matrix, and the correlation information between different channels is determined based on the K matrix and the Q matrix.

[0015] In one possible implementation, the correlation information of the channel itself is determined in the following manner: performing a second fusion operation on the data within each channel in the V matrix to obtain reduced-dimensional data; performing convolution on the reduced-dimensional data, and converting the convolution result into a diagonal matrix; the diagonal matrix is ​​used as the correlation information of the channel itself.

[0016] In a possible implementation, the first fusion operation is matrix addition.

[0017] In a possible implementation, the first fusion operation is a weighted summation; the weights corresponding to the correlation information between different channels and the weights corresponding to the correlation information of the channel itself are learnable parameters.

[0018] In a possible implementation, the second fusion operation is a global average pooling or convolution operation.

[0019] In one possible implementation, the attention module belongs to at least one attention head in the neural network model.

[0020] In one possible implementation, the linear transformation of the input data to obtain a K matrix, a Q matrix, and a V matrix includes: performing a convolution operation on the data of the first channel included in the input data according to a first convolution kernel to obtain a first convolution result; performing a convolution operation on the data of the first channel included in the input data according to a second convolution kernel to obtain a second convolution result; the first convolution result and the second convolution result are used to splice to obtain one of the K matrix, the Q matrix, and the V matrix.

[0021] In one possible implementation, a convolution kernel can be applied to a channel of the input data to obtain the values ​​of the K, Q, or V matrices for the corresponding channel. In other words, when calculating the K, Q, or V matrices, at least one element in the matrix is ​​calculated using a single convolution kernel within a single channel of the input data, rather than through interactions between channels. This projection method acts independently on each channel of the spectral feature, preserving the specificity of the spectral information token.

[0022] In a possible implementation, the first convolution kernel and the second convolution kernel are the same.

[0023] In a possible implementation, the spectral information includes multiple response values ​​in a narrowband spectral response space.

[0024] In a second aspect, the present application provides a data processing device, comprising:

[0025] An acquisition module, configured to acquire first data collected by an image sensor;

[0026] A processing module is used to obtain spectral information based on the first data through a neural network model; wherein the neural network model includes an attention module, and the attention module is used to determine an attention matrix based on the input data and perform an attention operation based on the attention matrix; the attention matrix is ​​obtained by performing a first fusion operation on the correlation information between different channels of the input data and the correlation information of the channel itself.

[0027] In one possible implementation, the attention module is specifically used to perform a linear transformation on the input data to obtain a K matrix, a Q matrix and a V matrix, and determine the attention matrix based on the K matrix, the Q matrix and the V matrix; the correlation information with the channel itself is determined based on the V matrix, and the correlation information between different channels is determined based on the K matrix and the Q matrix.

[0028] In a possible implementation, the correlation information of the channel itself is determined in the following manner:

[0029] Performing a second fusion operation on the data within each channel of the V matrix to obtain dimensionally reduced data;

[0030] Convolution is performed on the reduced-dimensional data, and the convolution result is converted into a diagonal matrix; the diagonal matrix is ​​used as the correlation information of the channel itself.

[0031] In a possible implementation, the first fusion operation is matrix addition.

[0032] In a possible implementation, the first fusion operation is a weighted summation; the weights corresponding to the correlation information between different channels and the weights corresponding to the correlation information of the channel itself are learnable parameters.

[0033] In a possible implementation, the second fusion operation is a global average pooling or convolution operation.

[0034] In one possible implementation, the attention module belongs to at least one attention head in the neural network model.

[0035] In a possible implementation, the processing module is specifically configured to:

[0036] Performing a convolution operation on the data of the first channel included in the input data according to the first convolution kernel to obtain a first convolution result;

[0037] Performing a convolution operation on the data of the first channel included in the input data according to the second convolution kernel to obtain a second convolution result;

[0038] The first convolution result and the second convolution result are used to splice to obtain one of the K matrix, the Q matrix and the V matrix.

[0039] In a possible implementation, the first convolution kernel and the second convolution kernel are the same.

[0040] In a possible implementation, the spectral information includes multiple response values ​​in a narrowband spectral response space.

[0041] In a third aspect, an embodiment of the present application provides a data processing device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to perform the first aspect and any optional method thereof.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method thereof.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.

[0044] In a sixth aspect, the present application provides a chip system, which includes a processor for supporting the execution of a data processing device to implement the functions involved in the above aspects, such as sending or processing the data involved in the above methods; or information. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the data processing device. The chip system can be composed of a chip or can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of the structure of the artificial intelligence main framework;

[0046] FIG2 is a schematic diagram of the application system framework of the present application;

[0047] Figures 3 to 5 are schematic diagrams of a network structure;

[0048] FIG6 is a flowchart of a data processing method provided in an embodiment of the present application;

[0049] Figure 7 is a schematic diagram of data collected by the image sensor;

[0050] Figure 8 is a schematic diagram of data collected by the image sensor;

[0051] Figures 9 and 10 are schematic diagrams of a network structure;

[0052] 11A to 11C are schematic diagrams of a process of a data processing method provided in an embodiment of the present application;

[0053] FIG11D is a schematic diagram of an effect of the present application;

[0054] FIG12 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0055] FIG13 is a schematic diagram of a structure of an execution device provided in an embodiment of the present application;

[0056] FIG14 is a schematic diagram of a structure of a training device provided in an embodiment of the present application;

[0057] FIG15 is a schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0059] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0060] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0061] As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation, not as terms of degree, and are intended to take into account the inherent deviations in measurements or calculations that one of ordinary skill in the art would recognize. Furthermore, the use of "may" when describing embodiments of the present application refers to "one or more possible embodiments." As used herein, the terms "use," "using," and "used" may be considered synonymous with the terms "utilize," "utilizing," and "utilized," respectively. Additionally, the term "exemplary" is intended to refer to an example or illustration.

[0062] First, let's describe the overall workflow of an AI system. See Figure 1, which shows a schematic diagram of the AI ​​framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0063] (1) Infrastructure

[0064] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0065] (2) Data

[0066] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0067] (3) Data processing

[0068] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0069] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0070] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0071] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0072] (4) General ability

[0073] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0074] (5) Smart products and industry applications

[0075] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0076] It should be understood that the steps related to the model reasoning process in the embodiments of this application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and server is not limited to the processor-memory architecture described above. The system architecture provided by the embodiments of this application is described in detail below with reference to Figure 2.

[0077] FIG2 is a schematic diagram of the system architecture provided by an embodiment of the present application. As shown in FIG2 , the system architecture 500 includes an execution device 510 , a training device 520 , a database 530 , a client device 540 , a data storage system 550 , and a data acquisition system 560 .

[0078] The execution device 510 includes a calculation module 511, an I / O interface 512, a pre-processing module 513, and a post-processing module 514. The calculation module 511 may include the target model / rule 501, and the pre-processing module 513 and the post-processing module 514 are optional.

[0079] The execution device 510 may be a terminal device or a server that runs the above-mentioned image processing application.

[0080] Data acquisition device 560 is used to collect training samples. Training samples can include data collected by an image sensor (or the image sensor's response values) and corresponding spectral information (or multiple response values ​​in a narrowband spectral response space). After collecting training samples, data acquisition device 560 stores them in database 530.

[0081] The training device 520 can train the neural network to be trained (such as the neural network model in the embodiment of the present application (such as including an encoder, a decoder, etc.)) based on the training samples maintained in the database 530 to obtain the target model / rule 501.

[0082] It should be understood that the training device 520 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model based on the pre-training.

[0083] It should be noted that, in actual applications, the training samples maintained in the database 530 may not all be collected by the data acquisition device 560, but may also be received from other devices. It should also be noted that the training device 520 may not train the target model / rule 501 entirely based on the training samples maintained in the database 530, but may also obtain training samples from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0084] The target model / rule 501 obtained through training with the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in FIG2 . The execution device 510 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR) device, an in-vehicle terminal, etc., or a server, etc.

[0085] Specifically, the training device 520 may transfer the trained model to the execution device 510 .

[0086] In FIG. 2 , the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with an external device. A user can input data to the I / O interface 512 through a client device 540 .

[0087] Preprocessing module 513 and preprocessing module 514 are used to preprocess the input data received by I / O interface 512. It should be understood that preprocessing module 513 and preprocessing module 514 may be absent or only one preprocessing module may be present. If preprocessing module 513 and preprocessing module 514 are absent, computing module 511 may be used directly to process the input data.

[0088] When the execution device 510 preprocesses the input data, or when the computing module 511 of the execution device 510 performs calculations and other related processing, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 550.

[0089] Finally, the I / O interface 512 provides the processed results to the client device 540 and thus to the user.

[0090] In the scenario shown in FIG. 2 , the user can manually input data, and this "manual input data" can be operated through the interface provided by I / O interface 512. Alternatively, client device 540 can automatically send input data to I / O interface 512. If user authorization is required for client device 540 to automatically send input data, the user can set the corresponding permissions in client device 540. The user can view the output of execution device 510 on client device 540, which can be presented in a display, sound, action, or other specific form. Client device 540 can also serve as a data acquisition terminal, collecting input data and output results from I / O interface 512 as new sample data and storing them in database 530. Of course, collection can also be performed without client device 540, with I / O interface 512 directly storing the input data and output results from I / O interface 512 as new sample data in database 530.

[0091] It is worth noting that FIG2 is merely a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in FIG2 , the data storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the execution device 510 can be deployed in the client device 540.

[0092] From the inference side of the model:

[0093] In the embodiment of the present application, the computing module 511 of the above-mentioned execution device 510 can obtain the code stored in the data storage system 550 to implement the steps related to the model reasoning process in the embodiment of the present application.

[0094] In an embodiment of the present application, the computing module 511 of the execution device 510 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0095] Specifically, the computing module 511 of the execution device 510 can be a hardware system with an execution instruction function, and the steps related to the model reasoning process provided in the embodiment of the present application can be software codes stored in the memory. The computing module 511 of the execution device 510 can obtain the software code from the memory and execute the obtained software code to implement the steps related to the model reasoning process provided in the embodiment of the present application.

[0096] It should be understood that the computing module 511 of the execution device 510 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to the model reasoning process provided in the embodiment of the present application can also be implemented by the hardware system that does not have the function of executing instructions in the computing module 511 of the execution device 510, which is not limited here.

[0097] From the training side of the model:

[0098] In an embodiment of the present application, the above-mentioned training device 520 can obtain the code stored in the memory (not shown in Figure 2, which can be integrated into the training device 520 or deployed separately from the training device 520) to implement the steps related to model training in the embodiment of the present application.

[0099] In an embodiment of the present application, the training device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0100] It should be understood that the training device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to model training provided in the embodiments of the present application can also be implemented by the hardware system in the training device 520 that does not have the function of executing instructions, which is not limited here.

[0101] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0102] (1) Neural Network

[0103] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0104] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0105] (2) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. The convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way of extracting features is independent of position. The convolution kernel can be formalized as a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0106] CNN is a very common neural network. The following, combined with Figure 3, focuses on a detailed description of its structure. As mentioned in the previous basic concepts, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture uses machine learning algorithms to perform multiple levels of learning at different levels of abstraction. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which individual neurons respond to input images.

[0107] As shown in FIG3 , a convolutional neural network (CNN) 200 may include an input layer 210 , a convolutional layer / pooling layer 220 (wherein the pooling layer is optional), and a fully connected layer 230 .

[0108] Convolutional layer / pooling layer 220:

[0109] Convolutional layer:

[0110] As shown in FIG3 , the convolution layer / pooling layer 220 may include layers 221-226. For example, in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer. In another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layers 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can serve as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0111] The following will take convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0112] Convolution layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from an input image matrix. A convolution operator is essentially a weight matrix, which is typically predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), thereby extracting specific features from the image. The size of the weight matrix should be related to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolution output with a single depth dimension. However, in most cases, a single weight matrix is ​​not used. Instead, multiple weight matrices of the same size (rows × columns) are applied, i.e., multiple homogeneous matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolved image, where the dimension is determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract edge information, another weight matrix can be used to extract specific colors, and yet another weight matrix can be used to blur unwanted noise in the image. The weight matrices have the same size (rows × columns), and the feature maps extracted by these weight matrices are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0113] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0114] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0115] Pooling layer:

[0116] Because it's often necessary to reduce the number of training parameters, pooling layers are often periodically introduced after convolutional layers. For example, in layers 221-226, as shown in Figure 3 , layers 220 can be composed of one convolutional layer followed by one pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of a pooling layer is to reduce the spatial size of an image. Pooling layers can include average pooling and / or max pooling operators, which are used to downsample the input image to produce a smaller image. The average pooling operator calculates the average value of pixel values ​​within a specific range as the average pooling result. The max pooling operator takes the pixel with the largest value within a specific range as the max pooling result. Furthermore, just as the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The output image size after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding subregion of the image input to the pooling layer.

[0117] Fully connected layer 230:

[0118] After being processed by the convolutional layer / pooling layer 220, the convolutional neural network 200 is still insufficient to output the required output information. As previously mentioned, the convolutional layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 200 needs to use the fully connected layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the fully connected layer 230 may include multiple hidden layers (such as 231, 232 to 23n as shown in Figure 3). The parameters contained in these multiple hidden layers can be pre-trained based on relevant training data for a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0119] After the multiple hidden layers in the fully connected layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240. The output layer 240 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (as shown in Figure 3, the propagation from 210 to 240 is forward propagation) is completed, the back propagation (as shown in Figure 3, the propagation from 240 to 210 is back propagation) will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0120] It should be noted that the convolutional neural network 200 shown in Figure 3 is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, including only a part of the network structure shown in Figure 3. For example, the convolutional neural network used in the embodiment of the present application can only include an input layer 210, a convolution layer / pooling layer 220 and an output layer 240.

[0121] It should be noted that the convolutional neural network 100 shown in Figure 3 is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models. For example, as shown in Figure 4, multiple convolutional layers / pooling layers are run in parallel, and the extracted features are input into the fully connected layer 230 for processing.

[0122] (3) Deep Neural Networks

[0123] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. Based on the position of different layers in DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0124] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

[0125] It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0126] (4) Loss function

[0127] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0128] (5) Backpropagation algorithm

[0129] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the optimal super-resolution model parameters, such as the weight matrix.

[0130] (6) Multispectral: refers to a spectrum detection technology that can simultaneously acquire multiple optical spectrum bands (usually more than 3) and can expand to infrared light and ultraviolet light based on visible light.

[0131] (7) Aliasing: This refers to the overlapping of wavelengths covered by various spectral response channels. This weakens the orthogonality between spectral RAW signals and increases the difficulty of signal reconstruction.

[0132] (8) Attention mechanism: It is a technique in artificial neural networks that mimics cognitive attention. This mechanism can increase the weight of certain parts of the neural network input data while reducing the weight of other parts, thereby focusing the network's attention on the most important small part of the data.

[0133] (9) Feature token: It is an expression in the attention mechanism that refers to a part of independent feature information. For example, the feature embedding of each word in a sentence can be regarded as a token, and the feature information of different channels in a color image can also be regarded as a token.

[0134] (10) Feature similarity and particularity: For a set of feature markers, similarity describes the distance between each pair of features. The closer the distance, the more similar they are. Specificity describes the degree of independence of each token. If there are two very similar tokens, the specificity of the token is considered to be weak.

[0135] (11) Image signal processor (ISP): This processor processes the image signal output by the image sensor. It plays a central role in the camera system and is a crucial component of the camera. Its main functions include de-mosaicing, automatic exposure, automatic white balance, lens shading removal, gamma correction, color space conversion, dynamic range correction, and image cropping.

[0136] (12) Spatial / Spectral Resolution: An image is represented by H×W×C, where H×W represents the spatial resolution of the image and C represents the spectral resolution of the image. For example, a common three-channel color image has a spectral resolution of 3, while for a multispectral image, the spectral resolution C is usually greater than 3.

[0137] (13) Transformer layer

[0138] The neural network includes an embedding layer and at least one transformer layer, and the at least one transformer layer can be N transformer layers (N is an integer greater than 0), wherein each transformer layer includes an attention layer, an add&norm layer, a feed forward layer, and an add&norm layer that are adjacent in sequence. In the embedding layer, the current input is embedded to obtain multiple embedding vectors; in the attention layer, P input vectors are obtained from the previous layer of the first transformer layer, and with any first input vector among the P input vectors as the center, based on the correlation between each input vector within a preset attention window and the first input vector, the intermediate vector corresponding to the first input vector is obtained, thereby determining the P intermediate vectors corresponding to the P input vectors; in the pooling layer, the P intermediate vectors are merged into Q output vectors, wherein the multiple output vectors obtained by the last transformer layer in the transformer layer are used as feature representations of the current input.

[0139] Referring to Figure 5, Figure 5 is a schematic diagram of the structure of a transformer layer, wherein the transformer layer may include a multi-head attention layer (or simply referred to as an attention layer), an add and normalization (add&norm) layer, a feed forward net (FFN), and an add and normalization layer adjacent to each other.

[0140] Among them, the multi-head attention layer obtains N input vectors Xl from the layer above it, and the N input vectors Xl can be expressed as a matrix X. The multi-head attention layer adopts a self-attention mechanism to transform each vector based on the correlation between vectors to obtain N output vectors, which can be expressed as a matrix Y. It can be understood that when the multi-head attention layer is a layer directly connected to the embedding layer, such as the transformer layer directly connected to the embedding layer in Figure 5, the input vector it obtains is the embedding vector output by the embedding layer; when the multi-head attention layer is a multi-head attention layer included in a subsequent transformer layer, such as the multi-head attention layer included in a transformer layer directly connected to the previous transformer layer in Figure 5, the input vector it obtains is the output vector of the previous transformer layer. The multi-head attention layer can include multiple attention heads.

[0141] (14) Attention mechanism

[0142] The attention mechanism mimics the internal process of biological observation behavior, namely, a mechanism that aligns internal experience and external sensations to increase the observation precision of certain areas. It can quickly filter out high-value information from a large amount of information using limited attention resources. The attention mechanism can quickly extract important features from sparse data and is therefore widely used in natural language processing tasks, especially machine translation. The self-attention mechanism is an improvement on the attention mechanism, which reduces dependence on external information and is better at capturing the internal correlation of data or features. The essential idea of ​​the attention mechanism can be rewritten as the following formula:

[0143] Here, Lx = ||Source|| represents the length of the Source. This formula implies that the elements in the Source are imagined to consist of a series of data pairs. Given a Query element in the target, the similarity or correlation between the Query and each Key is calculated to obtain the weight coefficient for each Key's corresponding Value. The weighted sum of the Values ​​is then taken to obtain the final Attention value. Essentially, the Attention mechanism performs a weighted sum of the Values ​​of the Source elements, with the Query and Key used to calculate the weight coefficient for the corresponding Value. Conceptually, Attention can be understood as selectively filtering out a small amount of important information from a large amount of information and focusing on this important information, while ignoring the majority of less important information. This focusing process is reflected in the calculation of the weight coefficients: the larger the weight, the more focus is placed on the corresponding Value. In other words, the weight represents the importance of the information, while the Value represents the corresponding information. The self-attention mechanism can be understood as internal attention. The attention mechanism occurs between the Query element of the Target and all elements of the Source. The self-attention mechanism refers to the attention mechanism that occurs between the internal elements of the Source or the internal elements of the Target. It can also be understood as the attention calculation mechanism in the special case of Target = Source. The specific calculation process is the same, only the calculation object has changed.

[0144] Widely used cameras and smartphones today are equipped with image sensors. Common image sensors use a color filter array (CFA), also known as a Bayer filter, arranged above the photosensitive area. A common CFA arrangement consists of a 2×2 pixel unit, forming a circular unit. This unit contains the three primary colors: red (R), green (G), and blue (B), so color image sensors are often referred to as RGB image sensors.

[0145] RGB sensors are designed to directly simulate the response of the three types of cone cells (L / M / S) of the human eye to the spectrum. This response is a sampling of the complete spectral information, which cannot fully capture the spectral information and easily causes metamerism problems. As a result, the color reproduction performance of existing RGB cameras is limited in some difficult scenes, such as scenes containing large areas of pure color. Compared with RGB color sensors, multispectral sensors are designed to capture the original spectral signal more completely. The multispectral sensor consists of a 4×4 sixteen pixel unit above the gray photosensitive area, where each pixel corresponds to a color filter (or color filter coating). Through this design, the sensor's spectral resolution is increased from 3 dimensions to 16 dimensions, which can capture the spectral signal more completely.

[0146] However, due to constraints on filter materials, manufacturing processes, and cost, the raw signals (RAW signals) collected by image sensors exhibit aliasing between spectral bands. Therefore, the RAW signals must be analyzed and reconstructed to obtain spectral information. Existing implementations suffer from poor spectral reconstruction accuracy.

[0147] In order to solve the above problems, the present invention provides a data processing method. The data processing method of the present invention is described in detail below with reference to the accompanying drawings.

[0148] Referring to Figure 6, Figure 6 is a flow chart of a data processing method provided in an embodiment of the present application. As shown in Figure 6, a data processing method provided in an embodiment of the present application may include steps 601 to 602, and these steps are described in detail below.

[0149] 601. Acquire first data collected by an image sensor.

[0150] The first data may be a response value of an image sensor.

[0151] For example, referring to FIG. 7 , FIG. 7 is a schematic diagram of response values ​​collected by the RGB sensor.

[0152] For example, referring to FIG8 , FIG8 is a schematic diagram of a multi-spectral sensor and the response values ​​collected by the sensor.

[0153] Due to constraints on filter materials, manufacturing processes, and costs, raw images captured by image sensors (e.g., the first data in the present embodiment) often exhibit aliasing between spectral bands. Therefore, spectral information must be obtained through analysis and reconstruction. The present embodiment parses and reconstructs the first data to obtain spectral information.

[0154] In one possible implementation, the spectral information includes multiple response values ​​in a narrowband spectral response space. For example, the spectral information may include the response of each channel to a narrowband (eg, a resolution of 10 nm, 5 nm, etc.).

[0155] 602. Obtain spectral information based on the first data through a neural network model; wherein the neural network model includes an attention module, the attention module is used to determine an attention matrix based on the input data, and perform an attention operation based on the attention matrix; the attention matrix is ​​obtained by performing a first fusion operation on correlation information between different channels of the input data and correlation information of the channel itself.

[0156] In one possible implementation, the network used to parse and reconstruct spectral information can be the neural network model in the embodiment of the present application, which may include an embedding layer, an encoder, and a decoder (optionally, it may also include residual connections between modules).

[0157] Refer to Figure 9, which shows a schematic diagram of the neural network model. The overall network structure is a multi-stage U-shaped structure. Thanks to upsampling and downsampling, this structure can capture multi-scale spatial information while reducing the computational burden. Feature embedding and mapping: RGB color images (or multispectral images (MSI)) are input, image features are extracted through convolution operations, spectral information is reconstructed based on residual learning, and finally the spectral features are remapped to hyperspectral signal (HSI) output.

[0158] In one possible implementation, the neural network model may include a module for performing attention operations (that is, the attention module in the embodiment of the present application), which may belong to one or more of the at least one attention head included in the neural network model.

[0159] For example, the main body of the encoder and decoder in the neural network model can be composed of a hybrid spectral-wise attention block (HSAB) as shown in Figure 9, and there can be a residual connection between the encoder and decoder to reduce information loss.

[0160] For example, referring to Figure 10 , HSAB can be composed of layer normalization (Layer Norm), a plausible spectral-wise self-attention module (PSA), and a feed-forward network (FFN). The FFN is implemented by a 3x3 depthwise separable convolution, and the PSA includes a spectral re-calibration (SRC) module and a token independent mapping (TIM) module.

[0161] Among them, the attention module may belong to PSA (for example, a head in PSA, and the PSA shown in Figure 10 is a schematic diagram of a head).

[0162] In one possible implementation, the attention module is specifically used to perform a linear transformation on the input data (optionally, it can be a fusion result of the input data and position encoding) (for example, a linear transformation is performed by the TIM module shown in Figure 10) to obtain a K matrix, a Q matrix and a V matrix, and determine the attention matrix based on the K matrix, the Q matrix and the V matrix.

[0163] In the existing implementation, the first transformation matrix Q, the second transformation matrix K and the third transformation matrix V are used to transform N input vectors respectively.<X1,X2,…,XN> Each input vector Xi is transformed to obtain the first intermediate vector (q vector), the second intermediate vector (k vector) and the third intermediate vector (v vector) corresponding to each input vector. Operationally, the first transformation matrix Q, the second transformation matrix K and the third transformation matrix V can be used to perform linear transformation on the input matrix X composed of N input vectors, respectively, to obtain the Q matrix, K matrix and V matrix of the input matrix, and then the matrix is ​​split respectively to obtain the q vector, k vector and v vector corresponding to each input vector. For any i-th input vector Xi among the N input vectors, based on the dot product operation of the first intermediate vector (q vector, qi) corresponding to the i-th input vector and the second intermediate vector (k vector, kj) corresponding to each input vector Xj, the correlation between the i-th input vector Xi and each input vector Xj is determined. Although the dot product result of qi and kj can also be directly determined as the correlation, it is more classic to first divide the dot product result by a constant, and then perform a softmax operation, and use the result of the operation as the correlation between the input vector Xi and Xj. The correlation between each input vector can constitute an attention matrix, that is:

[0164] Therefore, the correlation degrees αi,j between the i-th input vector Xi and each input vector Xj can be used as weight factors to perform a weighted combination on the third intermediate vectors (v vectors, vj) corresponding to each input vector Xj (this weighted combination operation is equivalent to the attention operation based on the attention matrix) to obtain the i-th combination vector Ci corresponding to the i-th input vector Xi:

[0165] Thus, we can get the vector sequence of N combination vectors corresponding to N input vectors<C1,C2,…,CN> , or matrix C. Based on this combination vector sequence, N output vectors can be obtained. Specifically, in one embodiment, the vector sequence of N combination vectors can be directly used as N output vectors, that is, Yi=Ci. At this time, the output matrix Y is the combination vector matrix C, which can also be written as:

[0166] However, in the above implementation, only the correlation between different channels is considered during the calculation of the attention matrix. In some implementations, when the sizes of the elements in the attention matrix are close (for example, they are all very small or very large), there is no obvious distinction between different channels, which will lead to poor attention operation effect. Especially in the reconstruction scenario of spectral information, each channel can be considered to have a specific physical meaning. Therefore, the attention matrix needs to express the distinction between different channels as much as possible to improve the reconstruction accuracy of spectral information.

[0167] In an embodiment of the present application, when calculating the attention matrix, the attention module can fuse the correlation between different channels and the correlation between the channels themselves, so that the attention matrix can model the discrimination between different channels.

[0168] In a possible implementation, the correlation information between different channels may be determined based on the K matrix and the Q matrix.

[0169] In a possible implementation, the correlation information with the channel itself may be determined according to the V matrix.

[0170] In one possible implementation, the correlation information of the channel itself is determined in the following manner: performing a second fusion operation on the data within each channel in the V matrix to obtain reduced-dimensional data; performing convolution on the reduced-dimensional data, and converting the convolution result into a diagonal matrix; the diagonal matrix is ​​used as the correlation information of the channel itself.

[0171] In a possible implementation, the second fusion operation is a global average pooling or convolution operation.

[0172] For example, as shown in Figure 11A, global average pooling (GAP) or other alternative operations such as convolution can be used to reduce the dimension of H×W×C features to 1×C and extract spectral dimension features; then one-dimensional convolution is used to map the 1×C features into a specific attention weight vector 1×C, where, optionally, the weight of the one-dimensional convolution can be shared among each head branch.

[0173] As shown in Figure 11B, the obtained specific attention weight vector is diagonally embedded (DE) to obtain a C×C matrix. This diagonal matrix is ​​used to fuse the correlation information of the channel itself with the correlation information between channels obtained based on the Q matrix and the K matrix to obtain the attention matrix.

[0174] In a possible implementation, the correlation information of the channel itself and the correlation information between channels obtained based on the Q matrix and the K matrix may be fused through a first fusion operation.

[0175] In a possible implementation, the first fusion operation is matrix addition.

[0176] In a possible implementation, the first fusion operation is a weighted summation; the weights corresponding to the correlation information between different channels and the weights corresponding to the correlation information of the channel itself are learnable parameters.

[0177] For example, the above process can be expressed by the following formula:

[0178] Where X represents the input feature, Q / K / V represent the query / key / value in the attention mechanism, respectively, which are obtained by linear transformation of X, and σ1 and σ2 are learnable weighting parameters.

[0179] In addition, in order to further improve the discrimination between different channels, the present application can interactively fuse the input data (or the fusion of input data and other data such as position features) within the channel during the calculation of the K matrix, Q matrix and V matrix to obtain the numerical value of the K matrix, Q matrix or V matrix in the corresponding channel.

[0180] In one possible implementation, a convolution kernel can be applied to a channel of the input data to obtain the values ​​of the K, Q, or V matrices for the corresponding channel. In other words, when calculating the K, Q, or V matrices, at least one element in the matrix is ​​calculated using a single convolution kernel within a single channel of the input data, rather than through interactions between channels. This projection method acts independently on each channel of the spectral feature, preserving the specificity of the spectral information token.

[0181] For example, a convolution operation can be performed on the data of the first channel included in the input data according to a first convolution kernel to obtain a first convolution result; a convolution operation can be performed on the data of the first channel included in the input data according to a second convolution kernel to obtain a second convolution result; the first convolution result and the second convolution result are used to splice to obtain one of the K matrix, the Q matrix and the V matrix. For example, if the above steps are used when calculating the K matrix, the first convolution result and the second convolution result are used to splice to obtain the K matrix (of course, the spliced ​​object can also include other convolution calculation results); for example, if the above steps are used when calculating the Q matrix, the first convolution result and the second convolution result are used to splice to obtain the Q matrix (of course, the spliced ​​object can also include other convolution calculation results); for example, if the above steps are used when calculating the V matrix, the first convolution result and the second convolution result are used to splice to obtain the V matrix (of course, the spliced ​​object can also include other convolution calculation results).

[0182] In one possible implementation, for the same Q matrix, or the same K matrix, or the same V matrix, the same convolution kernel can be used when calculating the values ​​of at least two different channels. For example, according to a first convolution kernel, a convolution operation can be performed on the data of the first channel included in the input data to obtain a first convolution result; according to a second convolution kernel, a convolution operation can be performed on the data of the first channel included in the input data to obtain a second convolution result, and the first convolution kernel and the second convolution kernel are the same.

[0183] The above process of calculating the K matrix, the Q matrix or the V matrix may be performed by the TIM module in the structure shown in FIG10 .

[0184] As shown in Figure 11C, for one branch (Head) in the multi-head attention mechanism, a convolution kernel that implements linear mapping can be learned; the learned convolution kernel is then copied and broadcasted inside the head, that is, the convolution kernel is shared and acts on each channel of the spectral feature.

[0185] The above is a description of the processing of an attention head. In the MHA architecture, the MHA layer maintains m sets of transformation matrices. Each set of transformation matrices includes the aforementioned Q matrix, K matrix, and V matrix, so that the above operations can be performed in parallel to obtain m combination vector sequences (i.e., m matrices C). Each vector sequence includes N combination vectors obtained based on a set of transformation matrices. In this case, the MHA layer splices the obtained m combination vector sequences to obtain a spliced ​​matrix. Splitting the output matrix Y corresponds to N output vectors.<Y1,Y2,…,YN> Through the above operation process, the MHA layer performs transformation operations based on the correlation between N input vectors to obtain N output vectors.

[0186] Next, the beneficial effects of the embodiments of the present application are introduced with reference to experiments.

[0187] This solution improves the performance of spectral signal reconstruction by modeling spectral specificity through global average pooling (GAP) and 1D convolution. As shown in Figure 11D, from the visualization results of the similarity of attention features, it can be seen that compared with the industry's representative algorithm (MST++), the embodiment of the present application significantly improves the specificity of spectral features. As shown in Table 1, when the SRC structure is removed from the network model, the mean relative absolute error (MRAE) and the root mean square error (RMSE) both increase significantly, proving the effectiveness of the SRC structure.

[0188] Table 1

[0189] As shown in Table 2, when the TIM structure is removed from the network model, the mean relative absolute error (MRAE) and root mean square error (RMSE) both increase significantly, which proves the effectiveness of the TIM structure.

[0190] Table 2

[0191] This solution is relatively low in complexity, enabling end-to-end joint optimization across its modules. Based on the proposed spectral remodulation and label-independent mapping model structure, it simultaneously models the correlation and specificity of spectral features. It achieves leading results in spectral signal reconstruction on public datasets, achieving significantly lower reconstruction error compared to a range of existing algorithms and MST++, the winning solution in the NTIRE 2022 Spectral Reconstruction Challenge, as shown in Table 3.

[0192] Table 3

[0193] In addition, referring to FIG. 12 , FIG. 12 is a schematic diagram of the structure of a data processing device 1200 provided in an embodiment of the present application, including:

[0194] An acquisition module 1201 is configured to acquire first data collected by an image sensor;

[0195] For a detailed description of the acquisition module 1201 , please refer to the introduction of step 601 in the above embodiment, which will not be repeated here.

[0196] The processing module 1202 is used to obtain spectral information based on the first data through a neural network model; wherein the neural network model includes an attention module, and the attention module is used to determine an attention matrix based on the input data and perform an attention operation based on the attention matrix; the attention matrix is ​​obtained by performing a first fusion operation on the correlation information between different channels of the input data and the correlation information of the channel itself.

[0197] For a detailed description of the processing module 1202 , please refer to the introduction of step 602 in the above embodiment, which will not be repeated here.

[0198] In one possible implementation, the attention module is specifically used to perform a linear transformation on the input data to obtain a K matrix, a Q matrix and a V matrix, and determine the attention matrix based on the K matrix, the Q matrix and the V matrix; the correlation information with the channel itself is determined based on the V matrix, and the correlation information between different channels is determined based on the K matrix and the Q matrix.

[0199] In a possible implementation, the correlation information of the channel itself is determined in the following manner:

[0200] Performing a second fusion operation on the data within each channel of the V matrix to obtain dimensionally reduced data;

[0201] Convolution is performed on the reduced-dimensional data, and the convolution result is converted into a diagonal matrix; the diagonal matrix is ​​used as the correlation information of the channel itself.

[0202] In a possible implementation, the first fusion operation is matrix addition.

[0203] In a possible implementation, the first fusion operation is a weighted summation; the weights corresponding to the correlation information between different channels and the weights corresponding to the correlation information of the channel itself are learnable parameters.

[0204] In a possible implementation, the second fusion operation is a global average pooling or convolution operation.

[0205] In one possible implementation, the attention module belongs to at least one attention head in the neural network model.

[0206] In a possible implementation, the processing module 1202 is specifically configured to:

[0207] Performing a convolution operation on the data of the first channel included in the input data according to the first convolution kernel to obtain a first convolution result;

[0208] Performing a convolution operation on the data of the first channel included in the input data according to the second convolution kernel to obtain a second convolution result;

[0209] The first convolution result and the second convolution result are used to splice to obtain one of the K matrix, the Q matrix and the V matrix.

[0210] In a possible implementation, the first convolution kernel and the second convolution kernel are the same.

[0211] In a possible implementation, the spectral information includes multiple response values ​​in a narrowband spectral response space.

[0212] Next, an execution device provided in an embodiment of the present application is introduced. Please refer to Figure 13. Figure 13 is a structural diagram of an execution device provided in an embodiment of the present application. The execution device 1300 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop computer, a smart wearable device, a monitoring data processing device or a server, etc., which is not limited here. Specifically, the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303 and a memory 1304 (wherein the number of processors 1303 in the execution device 1300 can be one or more, and Figure 13 takes one processor as an example), wherein the processor 1303 may include an application processor 13031 and a communication processor 13032. In some embodiments of the present application, the receiver 1301, the transmitter 1302, the processor 1303 and the memory 1304 may be connected via a bus or other means.

[0213] Memory 1304 may include read-only memory and random access memory, and provides instructions and data to processor 1303. A portion of memory 1304 may also include non-volatile random access memory (NVRAM). Memory 1304 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0214] Processor 1303 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0215] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1303. Processor 1303 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1303. The above processor 1303 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1303 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1304. Processor 1303 reads information from memory 1304 and, in conjunction with its hardware, completes the steps involved in the model inference process in the above method.

[0216] Receiver 1301 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1302 can be used to output digital or character information through the first interface. Transmitter 1302 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1302 can also include a display device such as a display screen.

[0217] The present application also provides a training device. Please refer to FIG. 14 , which is a schematic diagram of a structure of a training device provided by an embodiment of the present application. Specifically, the training device 1400 is implemented by one or more servers. The training device 1400 may vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1414 (e.g., one or more processors), a memory 1432, and one or more storage media 1430 (e.g., one or more mass storage devices) storing application programs 1442 or data 1444. The memory 1432 and storage medium 1430 may be either ephemeral or persistent storage. The program stored in the storage medium 1430 may include one or more modules (not shown), each of which may include a series of instruction operations on the training device. Furthermore, the CPU 1414 may be configured to communicate with the storage medium 1430 to execute the series of instruction operations in the storage medium 1430 on the training device 1400.

[0218] The training device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458; or, one or more operating systems 1441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0219] In the embodiment of the present application, the central processing unit 1414 is used to execute actions related to model training in the above embodiments.

[0220] An embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0221] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0222] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0223] Specifically, see Figure 15 , which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 1500. NPU 1500 is mounted on the host CPU (host CPU) as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1503, which is controlled by controller 1504 to extract matrix data from memory and perform multiplication operations.

[0224] In some implementations, arithmetic circuit 1503 includes multiple processing units (PEs). In some implementations, arithmetic circuit 1503 is a two-dimensional systolic array. Arithmetic circuit 1503 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1503 is a general-purpose matrix processor.

[0225] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1502 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1501 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1508.

[0226] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 through the Direct Memory Access Controller (DMAC) 1505. Input data is also transferred to unified memory 1506 through the DMAC.

[0227] BIU stands for Bus Interface Unit 1510 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1509 .

[0228] The bus interface unit 1510 (BIU) is used for the instruction fetch memory 1509 to obtain instructions from the external memory, and is also used for the storage unit access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0229] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1506 or transfer weight data to the weight memory 1502 or transfer input data to the input memory 1501.

[0230] The vector calculation unit 1507 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0231] In some implementations, the vector calculation unit 1507 can store the processed output vector in the unified memory 1506. For example, the vector calculation unit 1507 can apply a linear function or a nonlinear function to the output of the operation circuit 1503, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1507 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1503, for example, for use in subsequent layers in a neural network.

[0232] An instruction fetch buffer 1509 connected to the controller 1504 is used to store instructions used by the controller 1504;

[0233] Unified memory 1506, input memory 1501, weight memory 1502, and instruction fetch memory 1509 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0234] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0235] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0236] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0237] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0238] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A data processing method, characterized in that: The method comprises: Acquire first data collected by the image sensor; According to the first data, spectral information is obtained through a neural network model; wherein, The neural network model includes an attention module, which is used to determine an attention matrix based on input data and perform attention operations based on the attention matrix; the attention matrix is ​​obtained by performing a first fusion operation on the correlation information between different channels of the input data and the correlation information of the channel itself.

2. The method according to claim 1, characterized in that The attention module is specifically used to perform a linear transformation on the input data to obtain a K matrix, a Q matrix and a V matrix, and determine the attention matrix based on the K matrix, the Q matrix and the V matrix; the correlation information with the channel itself is determined based on the V matrix, and the correlation information between different channels is determined based on the K matrix and the Q matrix.

3. The method according to claim 1 or 2, characterized in that: The correlation information of the channel itself is determined in the following manner: Perform a second fusion operation on the data in each channel of the V matrix to obtain data after dimensionality reduction; Convolution is performed on the reduced-dimensional data, and the convolution result is converted into a diagonal matrix; the diagonal matrix is ​​used as the correlation information of the channel itself.

4. The method according to any one of claims 1 to 3, characterized in that: The first fusion operation is matrix addition.

5. The method according to any one of claims 1 to 4, characterized in that: The first fusion operation is a weighted summation; the weights corresponding to the correlation information between different channels and the weights corresponding to the correlation information of the channel itself are learnable parameters.

6. The method according to any one of claims 2 to 5, characterized in that: The second fusion operation is a global average pooling or a convolution operation.

7. The method according to any one of claims 1 to 6, characterized in that: The attention module belongs to at least one attention head in the neural network model.

8. The method according to any one of claims 2 to 7, characterized in that: The linear transformation of the input data to obtain the K matrix, Q matrix and V matrix includes: According to the first convolution kernel, a convolution operation is performed on the data of the first channel included in the input data to obtain a first convolution result; According to the second convolution kernel, a convolution operation is performed on the data of the first channel included in the input data to obtain a second convolution result; The first convolution result and the second convolution result are used to concatenate to obtain one of the K matrix, the Q matrix and the V matrix.

9. The method according to claim 8, characterized in that The first convolution kernel and the second convolution kernel are the same.

10. The method according to any one of claims 1 to 9, characterized in that: The spectral information includes a plurality of response values ​​in a narrowband spectral response space.

11. A data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring first data collected by the image sensor; A processing module is used to obtain spectral information through a neural network model according to the first data; wherein the neural network model includes an attention module, and the attention module is used to determine an attention matrix according to the input data, and perform attention according to the attention matrix. Attention operation; the attention matrix is ​​obtained by performing a first fusion operation on the correlation information between different channels of the input data and the correlation information of the channel itself.

12. The device according to claim 11, characterized in that The attention module is specifically used to perform a linear transformation on the input data to obtain a K matrix, a Q matrix and a V matrix, and determine the attention matrix based on the K matrix, the Q matrix and the V matrix; the correlation information with the channel itself is determined based on the V matrix, and the correlation information between different channels is determined based on the K matrix and the Q matrix.

13. The device according to claim 11 or 12, characterized in that The correlation information of the channel itself is determined in the following manner: Perform a second fusion operation on the data in each channel of the V matrix to obtain data after dimensionality reduction; Convolution is performed on the reduced-dimensional data, and the convolution result is converted into a diagonal matrix; the diagonal matrix is ​​used as the correlation information of the channel itself.

14. The device according to any one of claims 11 to 13, characterized in that: The first fusion operation is matrix addition.

15. The device according to any one of claims 11 to 14, characterized in that: The first fusion operation is a weighted summation; the weights corresponding to the correlation information between different channels and the weights corresponding to the correlation information of the channel itself are learnable parameters.

16. The device according to any one of claims 12 to 15, characterized in that The second fusion operation is a global average pooling or a convolution operation.

17. The device according to any one of claims 11 to 16, characterized in that The attention module belongs to at least one attention head in the neural network model.

18. The device according to any one of claims 12 to 17, characterized in that The processing module is specifically used for: According to the first convolution kernel, a convolution operation is performed on the data of the first channel included in the input data to obtain a first convolution result; According to the second convolution kernel, a convolution operation is performed on the data of the first channel included in the input data to obtain a second convolution result; The first convolution result and the second convolution result are used to concatenate to obtain one of the K matrix, the Q matrix and the V matrix.

19. The device according to claim 18, characterized in that The first convolution kernel and the second convolution kernel are the same.

20. The device according to any one of claims 11 to 19, characterized in that The spectral information includes a plurality of response values ​​in a narrowband spectral response space.

21. A computer storage medium, characterized in that The computer storage medium stores one or more instructions which, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 10.

22. A computer program product, characterized in that The method comprises computer-readable instructions, and when the computer-readable instructions are executed on a computer device, the computer device is caused to execute the method according to any one of claims 1 to 10.

23. A system comprising at least one processor and at least one memory; the processor and the memory are connected via a communication bus and communicate with each other; The at least one memory is used to store code; The at least one processor is configured to execute the code to perform the method according to any one of claims 1 to 10.

24. A chip, comprising a processor, characterized in that: The processor is used to support a data processing device to implement the method according to any one of claims 1 to 10.

Images