Multimodal data processing method, device, equipment and storage medium

Abstract

The present application discloses a multimodal data processing method, apparatus, device and storage medium, which includes: obtaining different optical modal information of a target object and producing a multimodal data set; constructing a multimodal fusion network model; the multimodal fusion network model includes a modal feature extraction network for extracting features of each modal, a modal feature fusion network for merging features of each modal, and a decision network for performing classification or regression tasks on the merged target features; training the multimodal fusion network model using a multimodal data set; obtaining different optical modal information of the object to be tested and inputting it into the trained multimodal fusion network model, and outputting classification results or regression results. In this way, rich features of each optical modality of the object and the intrinsic relationship between different optical modalities can be obtained, classification or regression tasks can be completed efficiently, thereby promoting the development of multimodal artificial intelligence information extraction and fusion and enhancing competitiveness.

Description

Technical Field

[0001] The present invention relates to the field of multimodal information processing, and in particular to a multimodal data processing method, apparatus, device and storage medium. Background Art

[0002] People experience the world multimodally. For AI to better understand the world around them, it needs to be able to interpret and reason about multimodal information. In multimodal machine learning, because different modalities may contain complementary information, using data from multiple modalities allows the model to make more robust predictions. Furthermore, multimodal systems can still function even when data from one modality is missing. In recent years, multimodal machine learning has rapidly developed, encompassing fields such as audio-visual speech recognition, multimodal emotion recognition, medical image analysis, and multimedia event detection.

[0003] While academia and industry have made significant progress in multimodal fusion, current research primarily focuses on images, speech, and text. Multimodal datasets and multimodal data fusion related to optical modalities, such as polarization and frequency, are less studied. However, the rich target and environmental characteristics of optical modalities are of great significance in fields such as target recognition, security, and biomedicine. Summary of the Invention

[0004] In view of this, the present invention aims to provide a multimodal data processing method, apparatus, device, and storage medium that can obtain the rich features of each optical mode of an object and the inherent relationship between different optical modes, efficiently complete classification or regression tasks, and further promote the development of multimodal information extraction and fusion. The specific scheme is as follows:

[0005] A multimodal data processing method, comprising:

[0006] Obtain different optical modal information of the target object and create a multimodal dataset;

[0007] Constructing a multimodal fusion network model; the multimodal fusion network model includes a modal feature extraction network for extracting features of each modality, a modal feature fusion network for merging features of each modality, and a decision network for performing classification tasks or regression tasks on the merged target features;

[0008] Training the multimodal fusion network model using the multimodal dataset;

[0009] The different optical modal information of the object to be measured is obtained and input into the trained multimodal fusion network model, and the classification result or regression result is output.

[0010] Preferably, in the multimodal data processing method provided in the embodiment of the present invention, the step of obtaining different optical modal information of the target object includes:

[0011] Obtain information of at least two of the three different modes of intensity, polarization, and frequency of the target object.

[0012] Preferably, in the multimodal data processing method provided in the embodiment of the present invention, obtaining information of three different modes of intensity, polarization, and frequency of the target object includes:

[0013] The reflected light from the target object is split into a first light beam and a second light beam by a light splitting system; the first light beam is transmitted to an optical micropolarizer system; and the second light beam is transmitted to a Fourier 4f system;

[0014] acquiring intensity information and polarization information through the optical micropolarizer system;

[0015] At the same time, frequency information is acquired through the Fourier 4f system.

[0016] Preferably, in the above-mentioned multimodal data processing method provided by an embodiment of the present invention, the optical micropolarizer system includes a first convex lens, a micropolarizer and a first detector; wherein,

[0017] The first convex lens is used to converge the first light beam onto the micro-polarizer;

[0018] The micro-polarizer is used to simultaneously collect intensity information and polarization information;

[0019] The first detector is used to convert the intensity information and polarization information collected by the micro-polarizer into two-dimensional matrix data.

[0020] Preferably, in the multimodal data processing method provided in the embodiment of the present invention, each pixel unit in the micro-polarizer includes two anti-reflection sub-units for collecting intensity information and two linear polarization sub-units for collecting polarization information;

[0021] The two linear polarization sub-units are distributed diagonally; the two anti-reflection sub-units are distributed diagonally.

[0022] Preferably, in the above-mentioned multimodal data processing method provided by an embodiment of the present invention, the Fourier 4f system includes a second convex lens, a third convex lens, a diffraction screen and a second detector; wherein,

[0023] The second convex lens is located between the target object and the light splitting system, and is used to converge the reflected light from the target object to obtain parallel light and transmit it to the light splitting system;

[0024] The diffraction screen is located between the light splitting system and the third convex lens, and is used to diffract the second light beam to obtain diffracted light;

[0025] The third convex lens is used to converge the diffracted light onto the second detector;

[0026] The second detector is used to collect spectrum signals.

[0027] Preferably, in the above-mentioned multimodal data processing method provided by an embodiment of the present invention, the modal feature extraction network includes multiple modal feature extraction sub-networks; each of the modal feature extraction sub-networks corresponds one-to-one to each modality.

[0028] Preferably, in the above-mentioned multimodal data processing method provided by an embodiment of the present invention, the input of each modal feature extraction sub-network is a multimodal data set in the form of a two-dimensional matrix, and the output is a modal embedding vector;

[0029] The input of the modal feature fusion network is the modal embedding vector, and the output is the fused modality obtained by calculating the triple Cartesian product.

[0030] The input of the decision network is the fusion modality, and the output is the result after completing the classification task or regression task.

[0031] An embodiment of the present invention further provides a multimodal data processing device, comprising:

[0032] The multimodal information acquisition module is used to obtain different optical modal information of the target object; it is also used to obtain different optical modal information of the object to be measured;

[0033] A data set production module is used to produce a multimodal data set based on different optical modal information of the target object;

[0034] A model construction module for constructing a multimodal fusion network model; the multimodal fusion network model includes a modal feature extraction network for extracting features of each modality, a modal feature fusion network for merging features of each modality, and a decision network for performing classification or regression tasks on the merged target features;

[0035] A model training module, configured to train the multimodal fusion network model using the multimodal dataset;

[0036] The model inference module is used to input different optical modal information of the object to be measured into the trained multimodal fusion network model and output classification results or regression results.

[0037] Preferably, in the above-mentioned multimodal data processing device provided by the embodiment of the present invention, the multimodal information acquisition module is specifically used to obtain information of at least two modes among the three different modes of intensity, polarization and frequency of the target object.

[0038] Preferably, in the above-mentioned multimodal data processing device provided by the embodiment of the present invention, the multimodal information acquisition module includes: a spectroscopic system, an optical micropolarizer system and a Fourier 4f system;

[0039] The optical splitting system is used to split the reflected light from the target object into a first light beam and a second light beam; the first light beam is transmitted to the optical micropolarizer system; the second light beam is transmitted to the Fourier 4f system;

[0040] The optical micropolarizer system is used to obtain intensity information and polarization information;

[0041] The Fourier 4f system is used to obtain frequency information.

[0042] Preferably, in the above-mentioned multimodal data processing device provided by an embodiment of the present invention, the optical micropolarizer system includes a first convex lens, a micropolarizer and a first detector; wherein,

[0043] The first convex lens is used to converge the first light beam onto the micro-polarizer;

[0044] The micro-polarizer is used to simultaneously collect intensity information and polarization information;

[0045] The first detector is used to convert the intensity information and polarization information collected by the micro-polarizer into two-dimensional matrix data.

[0046] Preferably, in the multimodal data processing device provided by an embodiment of the present invention, each pixel unit in the micro-polarizer includes two anti-reflection sub-units for collecting intensity information and two linear polarization sub-units for collecting polarization information;

[0047] The two linear polarization sub-units are distributed diagonally; the two anti-reflection sub-units are distributed diagonally.

[0048] Preferably, in the above-mentioned multimodal data processing device provided by an embodiment of the present invention, the Fourier 4f system includes a second convex lens, a third convex lens, a diffraction screen and a second detector; wherein,

[0049] The second convex lens is located between the target object and the light splitting system, and is used to converge the reflected light from the target object to obtain parallel light and transmit it to the light splitting system;

[0050] The diffraction screen is located between the light splitting system and the third convex lens, and is used to diffract the second light beam to obtain diffracted light;

[0051] The third convex lens is used to converge the diffracted light onto the second detector;

[0052] The second detector is used to collect spectrum signals.

[0053] An embodiment of the present invention further provides a multimodal data processing device, comprising a processor and a memory, wherein the processor implements the above-mentioned multimodal data processing method provided in the embodiment of the present invention when executing a computer program stored in the memory.

[0054] An embodiment of the present invention further provides a computer-readable storage medium for storing a computer program, wherein when the computer program is executed by a processor, the multimodal data processing method provided in the embodiment of the present invention is implemented.

[0055] It can be seen from the above technical solution that the present invention provides a multimodal data processing method, including: obtaining different optical modal information of the target object to produce a multimodal data set; constructing a multimodal fusion network model; the multimodal fusion network model includes a modal feature extraction network for extracting each modal feature, a modal feature fusion network for merging each modal feature, and a decision network for performing a classification task or a regression task on the merged target features; using the multimodal data set to train the multimodal fusion network model; obtaining different optical modal information of the object to be measured, and inputting it into the trained multimodal fusion network model, and outputting the classification result or regression result.

[0056] The above-mentioned multimodal data processing method provided by the present invention mainly includes two parts: obtaining different optical modal information of the object and multimodal information fusion based on neural network. In this way, the rich features of each optical mode of the object and the intrinsic relationship between different optical modes can be obtained, and multimodal information fusion can be realized. The rich target features can be efficiently completed to complete the classification or regression tasks, thereby promoting the development of multimodal artificial intelligence information extraction and fusion, and enhancing the competitiveness in the combined application field of optical information and multimodal artificial intelligence.

[0057] In addition, the present invention also provides corresponding devices, equipment and computer-readable storage media for the multimodal data processing method, further making the above method more practical, and the devices, equipment and computer-readable storage media have corresponding advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0059] Figure 1 A flowchart of a multimodal data processing method provided by an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of the structure of a multimodal information acquisition module provided in an embodiment of the present invention;

[0061] Figure 3 A schematic structural diagram of each pixel unit in a micro-polarizer provided by an embodiment of the present invention;

[0062] Figure 4 It is a structural diagram of the existing Fourier 4f system;

[0063] Figure 5 A schematic structural diagram of a Fourier 4f system provided in an embodiment of the present invention;

[0064] Figure 6 A schematic diagram of the structure of a multimodal fusion network model provided by an embodiment of the present invention;

[0065] Figure 7 A schematic diagram of multimodal tensor fusion provided by an embodiment of the present invention;

[0066] Figure 8 A schematic diagram of the structure of a multimodal data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0068] The present invention provides a multimodal data processing method, such as Figure 1 As shown, the following steps are included:

[0069] S101, obtaining different optical modal information of the target object and creating a multimodal data set;

[0070] S102, constructing a multimodal fusion network model; the multimodal fusion network model includes a modal feature extraction network for extracting features of each modality, a modal feature fusion network for merging features of each modality, and a decision network for performing a classification task or a regression task on the merged target features;

[0071] Specifically, a modal feature extraction network based on the attention mechanism is constructed to complete the feature extraction of each modality, a modal feature fusion network based on Cartesian product is used to merge multimodal information, and finally a decision network is used to complete the classification and regression tasks;

[0072] S103, using the multimodal dataset to train a multimodal fusion network model;

[0073] S104: Obtain different optical modal information of the object to be measured, input it into the trained multimodal fusion network model, and output classification results or regression results.

[0074] The multimodal data processing method provided in the embodiment of the present invention mainly includes two parts: obtaining different optical modal information of the object and multimodal information fusion based on a neural network. In this way, the rich features of each optical mode of the object and the intrinsic relationship between different optical modes can be obtained, and multimodal information fusion can be realized. The rich target features can be used to efficiently complete classification or regression tasks, thereby improving the network discrimination accuracy and model robustness, thereby promoting the development of multimodal artificial intelligence information extraction and fusion, and enhancing competitiveness in the combined application field of optical information and multimodal artificial intelligence.

[0075] In a specific implementation, in the multimodal data processing method provided in an embodiment of the present invention, step S101 of acquiring different optical modal information of a target object may specifically include acquiring information on at least two of the target object's three modalities: intensity, polarization, and frequency. In practical applications, step S101 may acquire information on only at least two of the target object's modalities: intensity, polarization, and frequency, or may acquire information on at least two other modalities in addition to intensity, polarization, and frequency. This description is omitted here.

[0076] Multimodal datasets can be generated based on information from at least two of the three different modalities of a target object: intensity, polarization, and frequency. The rich target and environmental characteristics in these modalities are of great significance in fields such as target recognition, security, and biomedicine.

[0077] Specifically, the intensity modality is a spectral radiation intensity measurement, which mainly obtains the distribution of different materials and objects in the scene, and obtains an optical image in the traditional sense.

[0078] Polarization modality measures light field vector information, which is largely uncorrelated with spectral radiation intensity images. It can capture target surface features, shapes, shadows, and roughness in complex environments like haze. It has broad applications in atmospheric environmental monitoring, biomedical diagnostics, and autonomous driving. Incorporating polarization modality not only increases the probability of target identification but also enhances detection accuracy.

[0079] Frequency modes capture the frequency distribution and variations of an image. Low-frequency components in the spectrum represent slowly changing parts of the spatial distribution function and coarse outline structures, while high-frequency components represent rapidly changing parts and details in the image. By capturing frequency modes, more detailed features of the target object can be extracted.

[0080] In a specific implementation, in the above-mentioned multimodal data processing method provided in an embodiment of the present invention, the above-mentioned steps of obtaining information of three different modes of intensity, polarization and frequency of the target object can specifically include: first, dividing the reflected light from the target object into a first light beam and a second light beam through a spectroscopic system; the first light beam is transmitted to an optical micropolarizer system; the second light beam is transmitted to a Fourier 4f system; then, intensity information and polarization information are obtained through the optical micropolarizer system; and at the same time, frequency information is obtained through the Fourier 4f system.

[0081] It should be noted that the above steps can be performed by a multimodal information acquisition module, which includes a spectroscopic system, an optical micropolarizer system, and a Fourier 4f system. It can not only simultaneously extract three types of optical multimodal information and construct an optical multimodal dataset, but also solve the alignment problem between different modalities.

[0082] In practical applications, such as Figure 2 As shown, the spectroscopic system can use a spectroscopic prism 1 for spectrometry. The spectroscopic prism 1 splits the light flux reflected from the target object into two parts. One part is input into the micropolarizer system to obtain intensity information a and polarization information b, and the other part is input into the Fourier 4f system to obtain frequency information c. The specific type of spectroscopic system can be determined based on actual conditions and is not limited here.

[0083] In specific implementation, in the above multimodal data processing method provided in the embodiment of the present invention, if Figure 2 As shown, the optical micro-polarizer system may include a first convex lens 2, a micro-polarizer 3 and a first detector 4; wherein,

[0084] a first convex lens 2, for converging the first light beam onto the micro-polarizer 3;

[0085] The micro-polarizer 3 is used to simultaneously collect intensity information a and polarization information b; Figure 3As shown, each pixel unit in the micro-polarizer 3 includes four sub-units arranged in 2×2, specifically including two anti-reflection sub-units 31 for collecting intensity information a and two linear polarization sub-units 32 for collecting polarization information b; the two linear polarization sub-units 31 are diagonally distributed; the two anti-reflection sub-units 32 are diagonally distributed.

[0086] The first detector 4 is used to convert the intensity information a and polarization information b collected by the micro-polarizer 3 into two-dimensional matrix data.

[0087] Specifically, if Figure 2 As shown, the first convex lens 2 converges the light emitted from the beam splitter prism 1 onto the micropolarizer 3. After passing through the micropolarizer 3, the light beam is collected by the first detector 4, generating intensity information a and polarization signal b. Each pixel unit in the micropolarizer 3 includes four subunits, corresponding to the four pixels of the first detector 4. These subunits include two anti-reflection subunits 31 and two linear polarization subunits 32, which are arranged diagonally. The diagonal distribution of the anti-reflection subunits 31 and linear polarization subunits 32 allows the entire first detector 4 to uniformly collect polarized light and natural light. Although the presence of the micropolarizer 3 reduces the pixel resolution by half, the target features contained in the acquired different modal information can significantly improve the accuracy of subsequent models. The linear polarizer subunit 32 generates linearly polarized light using the polarization principle of a subwavelength metal wire grid. The anti-reflection subunit 31 increases light transmittance by vapor-depositing an anti-reflection film of a specific wavelength band on a substrate. The addition of the anti-reflection film compensates to some extent for the reduced pixel resolution of the polarization image and the intensity image. The micropolarizer 3 can be manufactured using processes such as nanoimprinting or electron beam lithography. Because each unit of the micropolarizer 3 needs to be aligned with a pixel of the first detector 4, the micropolarizer 3 and the first detector 4 can be integrated on the same substrate. Finally, the two-dimensional matrix data converted by the first detector 4 is transmitted to a computer, which then splits the two-dimensional matrix data into two-dimensional matrix data corresponding to the intensity mode and the polarization mode.

[0088] In specific implementation, in the above multimodal data processing method provided in the embodiment of the present invention, if Figure 2 As shown, the Fourier 4f system may include a second convex lens 5, a third convex lens 6, a diffraction screen 7 and a second detector 8; wherein,

[0089] The second convex lens 5 is located between the target object and the spectroscopic system 1 and is used to converge the reflected light from the target object to obtain parallel light and transmit it to the spectroscopic system 1. This ensures the homology of the images of different modalities. The second convex lens 5 first converges the light flux of the target object into parallel light, and then the beam splitter prism 1 splits the input light flux into two, generating transmitted light and reflected light, which are respectively transmitted to the optical micropolarizer system and the Fourier 4f system.

[0090] a diffraction screen 7, located between the light splitting system and the third convex lens 6, for diffracting the second light beam to obtain diffracted light;

[0091] a third convex lens 6, for converging the diffracted light onto a second detector 8;

[0092] The second detector 8 is used to collect spectrum signals.

[0093] In practical applications, such as Figure 4 As shown, the Fourier 4f system is a "4f system" composed of two convex lenses with the same focal length of f, which can realize two cascaded Fourier transforms. The distribution of the plane wave carrying the target object information on the rear focal plane of the lens is proportional to the Fourier transform of the sample distribution, and is inversely Fourier transformed on the rear focal plane of the second lens to restore it to a clear image of the original sample. The Fourier 4f system has multiple diffraction systems and application scenarios. Therefore, in a specific embodiment of the present invention, a derivative system of the Fourier 4f system, the Fourier spectrum analyzer, that is, the Fraunhofer diffraction system is used to collect the frequency information of the target object. As shown in FIG. Figure 5 As shown, the Fourier 4f system (i.e., Fraunhofer diffraction system) provided by the present invention includes a second convex lens 5, a third convex lens 6, a diffraction screen 7, and a second detector 8. The diffraction screen 7 can be a slit or a window. The second convex lens 5 converges the reflected light from the target object, and the resulting parallel light is incident on the diffraction screen 7. The third convex lens 6 then converges the diffracted light to obtain a spectrum image. Placing the second detector 8 on the spectrum plane can collect spectrum modal signals. This system physically implements the Fourier transform, and can examine the response of the optical system to the image spectrum in the frequency domain, thereby processing the information contained in the image.

[0094] It should be noted that both the second convex lens 5 and the third convex lens 6 are confocal convex lenses. According to Fourier optics, specific optical lenses are configured to apply a forward or inverse Fourier transform to the wavefield. The Fourier transform can extract the global characteristics of the imaged object. Because the light field distribution on the confocal plane of the dual lenses is equal to the Fourier transform of the intensity distribution of the target object, various operations can be performed on this plane. By placing various modulations or filters, a wide range of functions can be realized, such as Abbe-Board spatial filtering.

[0095] In specific implementation, in the above multimodal data processing method provided in the embodiment of the present invention, if Figure 6As shown, the modal feature extraction network includes multiple modal feature extraction subnetworks; each modal feature extraction subnetwork corresponds one-to-one to each modality. The input of each modal feature extraction subnetwork is a multimodal dataset in the form of a two-dimensional matrix, namely the three modal signals of intensity, polarization, and frequency collected by the first and second detectors, and the output is a modal embedding vector. The structure of each modal feature extraction subnetwork is consistent, including an input layer, a flatten layer, a fully connected layer, and an attention layer. However, the input and weight parameters of each subnetwork are not shared; the attention layer includes a linear mapping, a ReLU activation, and a normalization layer.

[0096] It should be noted that the flatten layer is used to "flatten" the input, that is, to convert multidimensional input into one dimension. ReLU (Rectified Linear Unit) is a commonly used activation function in artificial neural networks.

[0097] Take the intensity mode as an example, Figure 6 As shown, the network input layer is a two-dimensional matrix of intensity mode, assuming I 64*64 (The data size of the input layer is related to the number of pixels of the detector), and the flatten layer is used to convert it into a one-dimensional vector I 4096 Input to the fully connected layer. Then the output of the fully connected layer I 128 , input to the attention layer. The attention layer includes linear mapping, ReLU activation and normalization layer, where the ReLU layer contains 128 units, which ensures that the output of the attention layer has the same dimension as the input, and the corresponding normalization layer outputs 128 weights W I Finally, the output weight vector W I with I 128 Multiply the corresponding points to get the output of the modal feature extraction network:

[0098] z I ∈R 128

[0099] Similarly, the outputs of polarization and frequency modes are z P , z f ∈R 128 .

[0100] In a specific implementation, in the multimodal data processing method provided in an embodiment of the present invention, the input of the modal feature fusion network is a modal embedding vector, and the output is a fused modality obtained by calculating a triple Cartesian product. That is, the modal feature fusion network extracts the intrinsic relationship between different modalities and converts multiple modal inputs into a tensor (i.e., a three-dimensional matrix) output. When calculating the triple Cartesian product, dual and trimodal modalities are calculated from a single modality.

[0101] To improve the versatility and flexibility of the network, if the number of pixels of the detector or the number of neurons in the feature extraction network are inconsistent, resulting in different output vector sizes, the length can be supplemented by adding a constant C to each modal vector. For example, C can be 0 or 1.

[0102] The coordinates of each neuron (z I 、z P and z f ) can be viewed as a point in the triple Cartesian space defined by the unimodal output vectors of intensity, polarization, and frequency. This definition is mathematically equivalent to the intensity embedding vector z l , polarization embedding vector z p and frequency embedding vector z f The differentiable outer product between :

[0103]

[0104] in, Represents the outer product between vectors, z l 、z p and z f is the single modal output vector from the modal feature extraction network. Specifically, the three z I 、z P and z f ∈R 128 Indicates single mode, three and Indicates the dual mode of acquisition, one Finally, as Figure 7 As shown in the figure, seven three-dimensional cubes with different semantic sub-regions can be spliced ​​to obtain z m ∈R 129 *129*129 .

[0105] It should be noted that although modality fusion computes a Cartesian product and has no learnable parameters, its chance of overfitting is very low because the output neurons of tensor fusion are easy to interpret and semantically meaningful. Therefore, subsequent layers of the network can easily decode meaningful information.

[0106] In the specific implementation, in the above multimodal data processing method provided by the embodiment of the present invention, the input of the decision network is the fusion modality, and the output is the result after completing the classification task or regression task; the decision network includes a flatten layer, two ReLU layers and an output layer. The decision network sets different network output layers and loss functions according to different tasks. After the modal feature fusion network, the feature data of each target object can be represented as a multimodal tensor z m .

[0107] Specifically, zm The input is fed into the flatten layer to obtain a one-dimensional vector, which is then fed into the ReLU layer. The ReLU layer includes linear mapping operations and ReLU nonlinear activation function operations. Finally, the output layer of the network is a softmax layer or a sigmoid layer, which completes the classification or regression tasks respectively. It should be noted that softmax is a normalized exponential function; sigmoid is used as the activation function of the neural network to map variables between 0 and 1. When the output layer is a softmax layer, the loss function of the decision network can be a classification cross entropy loss function for image classification; when the output layer is a sigmoid layer, the loss function of the decision network can be a mean error loss function to complete the regression task.

[0108] It is important to note that the present invention utilizes a simple and compact optical system to not only simultaneously extract three types of optical multimodal information and construct an optical multimodal dataset, but also solve the alignment problem between different modalities. A multimodal data fusion network based on an attention mechanism and Cartesian product is then used to extract the distinct features of each modality and learn the intrinsic relationships between the modalities, significantly improving network discrimination accuracy and model robustness. Furthermore, the inference network used supports different output layers, enabling flexible implementation of various tasks such as classification or regression, thus providing multiple possibilities for subsequent applications.

[0109] Based on the same inventive concept, an embodiment of the present invention further provides a multimodal data processing device. Since the principle of solving the problem of this device is similar to that of the aforementioned multimodal data processing method, the implementation of this device can refer to the implementation of the multimodal data processing method, and the repeated parts will not be repeated.

[0110] In specific implementation, the multimodal data processing device provided by the embodiment of the present invention is as follows: Figure 8 As shown, specifically including:

[0111] The multimodal information acquisition module 11 is used to obtain different optical modal information of the target object; it is also used to obtain different optical modal information of the object to be measured;

[0112] A data set preparation module 12 is used to prepare a multimodal data set based on different optical modal information of the target object;

[0113] A model construction module 13 is used to construct a multimodal fusion network model; the multimodal fusion network model includes a modal feature extraction network for extracting features of each modality, a modal feature fusion network for merging features of each modality, and a decision network for performing classification or regression tasks on the merged target features;

[0114] A model training module 14 is used to train a multimodal fusion network model using a multimodal dataset;

[0115] The model inference module 15 is used to input different optical modal information of the object to be measured into the trained multimodal fusion network model and output classification results or regression results.

[0116] In the above-mentioned multimodal data processing device provided in the embodiment of the present invention, the interaction of the above-mentioned five modules can be used to obtain the rich features of each optical mode of the object and the intrinsic relationship between different optical modes, and realize multimodal information fusion, so as to efficiently complete the classification or regression task with the enriched target features, thereby promoting the development of multimodal artificial intelligence information extraction and fusion, and enhancing the competitiveness in the combined application field of optical information and multimodal artificial intelligence.

[0117] In a specific implementation, in the above-mentioned multimodal data processing device provided by an embodiment of the present invention, the multimodal information acquisition module 11 can be specifically used to obtain information of at least two modes among the three different modes of intensity, polarization and frequency of the target object.

[0118] In a specific implementation, in the above-mentioned multimodal data processing device provided by an embodiment of the present invention, in order to simplify and compact the structure, the multimodal information acquisition module may include: a spectroscopic system (such as a spectroscopic prism), an optical micropolarizer system and a Fourier 4f system;

[0119] A beam splitting system is used to split the reflected light from the target object into a first beam and a second beam; the first beam is transmitted to the optical micropolarizer system; and the second beam is transmitted to the Fourier 4f system;

[0120] an optical micropolarizer system for acquiring intensity and polarization information;

[0121] Fourier 4f system, used to obtain frequency information.

[0122] In a specific implementation, in the multimodal data processing device provided in the embodiment of the present invention, if Figure 2 As shown, the optical micro-polarizer system may include a first convex lens 2, a micro-polarizer 3 and a first detector 4; wherein,

[0123] a first convex lens 2, for converging the first light beam onto the micro-polarizer 3;

[0124] The micro-polarizer 3 is used to simultaneously collect intensity information a and polarization information b; Figure 3 As shown, each pixel unit in the micro-polarizer 3 includes four sub-units arranged in 2×2, specifically including two anti-reflection sub-units 31 for collecting intensity information a and two linear polarization sub-units 32 for collecting polarization information b; the two linear polarization sub-units 31 are diagonally distributed; the two anti-reflection sub-units 32 are diagonally distributed.

[0125] The first detector 4 is used to convert the intensity information a and polarization information b collected by the micro-polarizer 3 into two-dimensional matrix data.

[0126] In a specific implementation, in the multimodal data processing device provided in the embodiment of the present invention, if Figure 2 As shown, the Fourier 4f system may include a second convex lens 5, a third convex lens 6, a diffraction screen 7 and a second detector 8; wherein,

[0127] The second convex lens 5 is located between the target object and the spectroscopic system 1 and is used to converge the reflected light from the target object to obtain parallel light and transmit it to the spectroscopic system 1. This ensures the homology of the images of different modalities. The second convex lens 5 first converges the light flux of the target object into parallel light, and then the beam splitter prism 1 splits the input light flux into two, generating transmitted light and reflected light, which are respectively transmitted to the optical micropolarizer system and the Fourier 4f system.

[0128] a diffraction screen 7, located between the light splitting system and the third convex lens 6, for diffracting the second light beam to obtain diffracted light;

[0129] a third convex lens 6, for converging the diffracted light onto a second detector 8;

[0130] The second detector 8 is used to collect spectrum signals.

[0131] For more specific working processes of the above-mentioned components, please refer to the corresponding contents disclosed in the above-mentioned embodiments, which will not be repeated here.

[0132] Correspondingly, an embodiment of the present invention further discloses a multimodal data processing device, comprising a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the multimodal data processing method disclosed in the aforementioned embodiment is implemented.

[0133] For more specific details of the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0134] Furthermore, the present invention also discloses a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the multimodal data processing method disclosed above is implemented.

[0135] For more specific details of the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0136] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar portions of the various embodiments will be sufficient. The devices, equipment, and storage media disclosed in the embodiments are described briefly because they correspond to the methods disclosed in the embodiments. For relevant details, refer to the method descriptions.

[0137] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0139] In summary, an embodiment of the present invention provides a multimodal data processing method, comprising: obtaining different optical modal information of a target object to create a multimodal data set; constructing a multimodal fusion network model; the multimodal fusion network model includes a modal feature extraction network for extracting features of each modal, a modal feature fusion network for merging features of each modal, and a decision network for performing classification or regression tasks on the merged target features; training the multimodal fusion network model using the multimodal data set; obtaining different optical modal information of the object to be measured, inputting it into the trained multimodal fusion network model, and outputting classification or regression results. The above-mentioned multimodal data processing method mainly includes two parts: obtaining different optical modal information of the object and multimodal information fusion based on a neural network. In this way, rich features of each optical modality of the object and the intrinsic relationship between different optical modalities can be obtained, and multimodal information fusion can be achieved, so that the rich target features can be efficiently completed. Classification or regression tasks can be achieved, thereby promoting the development of multimodal artificial intelligence information extraction and fusion, and enhancing competitiveness in the application field of combined optical information and multimodal artificial intelligence. In addition, the present invention also provides corresponding devices, equipment and computer-readable storage media for the multimodal data processing method, further making the above method more practical, and the devices, equipment and computer-readable storage media have corresponding advantages.

[0140] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0141] The multimodal data processing method, device, equipment and storage medium provided by the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multimodal data processing method, characterized in that: include: The reflected light from the target object is divided into a first light beam and a second light beam by a spectroscopic system; the first light beam is transmitted to an optical micropolarizer system; the second light beam is transmitted to a Fourier 4f system; intensity information and polarization information are obtained by the optical micropolarizer system; at the same time, frequency information is obtained by the Fourier 4f system; the optical micropolarizer system includes a first convex lens, a micropolarizer and a first detector; wherein the first convex lens is used to converge the first light beam onto the micropolarizer; the micropolarizer is used to simultaneously collect intensity information and polarization information; the first detector is used to convert the intensity information and polarization information collected by the micropolarizer into two-dimensional matrix data; each pixel unit in the micropolarizer includes two anti-reflection subunits for collecting intensity information and two linear polarization subunits for collecting polarization information; the two linear polarization subunits are diagonally distributed; the two anti-reflection subunits are diagonally distributed; Producing a multimodal data set based on the intensity information, polarization information, and frequency information of the target object; the multimodal data set is in the form of a two-dimensional matrix; Construct a multimodal fusion network model; the multimodal fusion network model includes a modal feature extraction network for extracting features of each modality, a modal feature fusion network for merging features of each modality, and a decision network for performing classification tasks or regression tasks on the merged target features; the modal feature extraction network includes multiple modal feature extraction subnetworks; each modal feature extraction subnetwork corresponds to each modality one-to-one; the input of each modal feature extraction subnetwork is a multimodal data set, and the output is a modal embedding vector; each modal feature extraction subnetwork uses a flatten layer to convert the input two-dimensional matrix into a one-dimensional vector and input it to the fully connected layer, and the output result of the fully connected layer is input to the attention layer ... output of each modal feature extraction subnetwork is a multimodal data set, and the output of the modal feature extraction subnetwork is a multimodal data set, and the output of the modal feature extraction subnetwork is a multimodal data set, and the output of the modal feature extraction subnetwork is input to the attention layer; the output of each modal feature extraction subnetwork is a multimodal data set, and the output of the modal feature extraction subnetwork is a multimodal data set, and the output of the modal feature extraction subnetwork is input to the attention layer; the output of each modal feature extraction subnetwork is a multimodal data set, and the output of the modal feature extraction subnetwork is input to the attention layer; the output of each modal feature extraction subnetwork is a multimodal data set, and the output of the modal feature extraction subnetwork is input to the attention layer; the output of each modal feature extraction subnetwork is a multimodal data set, and the output of the modal feature extraction subnetwork is input to the attention layer; the output of each modal feature extraction subnetwork The input and weight parameters are not shared; the attention layer includes linear mapping, ReLU activation and normalization layers; the output of the attention layer has the same dimension as the input; the weight vector output by the normalization layer is multiplied by the corresponding point of the output result of the fully connected layer to obtain the output of the modal feature extraction network; the modal feature fusion network extracts the intrinsic relationship between different modalities and converts multiple modal inputs into three-dimensional matrix outputs; the decision network inputs the three-dimensional matrix into the flatten layer to obtain a one-dimensional vector, and then inputs it into the ReLU layer; the ReLU layer includes linear mapping operations and ReLU nonlinear activation function operations; the output layer of the decision network is a softmax layer or a sigmoid layer, which respectively completes the classification or regression task; Training the multimodal fusion network model using the multimodal dataset; The different optical modal information of the object to be measured is obtained and input into the trained multimodal fusion network model, and the classification result or regression result is output.

2. The multimodal data processing method according to claim 1, wherein: The Fourier 4f system includes a second convex lens, a third convex lens, a diffraction screen and a second detector; wherein, The second convex lens is located between the target object and the light splitting system, and is used to converge the reflected light from the target object to obtain parallel light and transmit it to the light splitting system; The diffraction screen is located between the light splitting system and the third convex lens, and is used to diffract the second light beam to obtain diffracted light; The third convex lens is used to converge the diffracted light onto the second detector; The second detector is used to collect spectrum signals.

3. The multimodal data processing method according to claim 1, wherein: The input of the modality feature fusion network is the modality embedding vector, and the output is the fused modality obtained by calculating the triple Cartesian product; The input of the decision network is the fusion modality, and the output is the result after completing the classification task or regression task.

4. A multimodal data processing device, characterized in that: include: A multimodal information acquisition module, comprising: a spectroscopic system, an optical micropolarizer system, and a Fourier 4f system, wherein the spectroscopic system is used to split the reflected light from the target object into a first light beam and a second light beam; the first light beam is transmitted to the optical micropolarizer system; the second light beam is transmitted to the Fourier 4f system; the optical micropolarizer system is used to obtain intensity information and polarization information; the Fourier 4f system is used to obtain frequency information; the optical micropolarizer system comprises a first convex lens, a micropolarizer, and a first detector; wherein the first convex lens is used to converge the first light beam onto the micropolarizer; the micropolarizer is used to simultaneously collect intensity information and polarization information; the first detector is used to convert the intensity information and polarization information collected by the micropolarizer into two-dimensional matrix data; each pixel unit in the micropolarizer comprises two anti-reflection subunits for collecting intensity information and two linear polarization subunits for collecting polarization information; the two linear polarization subunits are diagonally distributed; the two anti-reflection subunits are diagonally distributed; and the module is also used to obtain different optical modal information of the object to be measured; A data set preparation module is used to prepare a multimodal data set based on the intensity information, polarization information and frequency information of the target object; the multimodal data set is in the form of a two-dimensional matrix; A model construction module is used to construct a multimodal fusion network model; the multimodal fusion network model includes a modal feature extraction network for extracting features of each modality, a modal feature fusion network for merging features of each modality, and a decision network for performing classification tasks or regression tasks on the merged target features; the modal feature extraction network includes multiple modal feature extraction subnetworks; each modal feature extraction subnetwork corresponds to each modality one-to-one; the input of each modal feature extraction subnetwork is a multimodal data set, and the output is a modal embedding vector; each modal feature extraction subnetwork uses a flatten layer to convert the input two-dimensional matrix into a one-dimensional vector and input it to the fully connected layer, and the output result of the fully connected layer is input to the attention layer; each modal feature extraction subnetwork uses a flatten layer to convert the input two-dimensional matrix into a one-dimensional vector and input it to the fully connected layer, and the output result of the fully connected layer is input to the attention layer; each modal feature extraction subnetwork The input and weight parameters of the sub-network are not shared; the attention layer includes linear mapping, ReLU activation and normalization layers; the output of the attention layer has the same dimension as the input; the weight vector output by the normalization layer is multiplied by the corresponding point of the output result of the fully connected layer to obtain the output of the modal feature extraction network; the modal feature fusion network extracts the intrinsic relationship between different modalities and converts multiple modal inputs into three-dimensional matrix outputs; the decision network inputs the three-dimensional matrix into the flatten layer to obtain a one-dimensional vector, and then inputs it into the ReLU layer; the ReLU layer includes linear mapping operations and ReLU nonlinear activation function operations; the output layer of the decision network is a softmax layer or a sigmoid layer, which respectively completes the classification or regression task; A model training module, configured to train the multimodal fusion network model using the multimodal dataset; The model inference module is used to input different optical modal information of the object to be measured into the trained multimodal fusion network model and output classification results or regression results.

5. The multimodal data processing device according to claim 4, characterized in that: The Fourier 4f system includes a second convex lens, a third convex lens, a diffraction screen and a second detector; wherein, The second convex lens is located between the target object and the light splitting system, and is used to converge the reflected light from the target object to obtain parallel light and transmit it to the light splitting system; The diffraction screen is located between the light splitting system and the third convex lens, and is used to diffract the second light beam to obtain diffracted light; The third convex lens is used to converge the diffracted light onto the second detector; The second detector is used to collect spectrum signals.

6. A multimodal data processing device, characterized in that: The device comprises a processor and a memory, wherein the processor implements the multimodal data processing method according to any one of claims 1 to 3 when executing the computer program stored in the memory.

7. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, it implements the multimodal data processing method according to any one of claims 1 to 3.

Images