Method and related device for processing multimodal fine-grained data using quantum neural network

Abstract

The present application provides a method and related device for processing multimodal fine-grained data by a quantum neural network, which relates to the field of quantum machine learning. Among them, the electronic device encodes multiple classical fine-grained features of the multimodal state into their respective corresponding quantum bit sets, so that the multiple quantum bit sets are in the first quantum state, and each feature extraction circuit acts on the corresponding quantum bit set to evolve the multiple quantum bit sets into the second quantum state; then, each quantum bit in the two adjacent quantum bit sets is entangled with the remaining quantum bits through each feature fusion circuit, so that the multiple quantum bit sets evolve into the third quantum state; finally, the measurement circuit acts on the multiple quantum bit sets to obtain the processing results of the above features. In this way, the feature information of different classical fine-grained features is deeply integrated, effectively revealing and mining the intrinsic connections between them, thereby significantly improving the accuracy of the processing results of multiple classical fine-grained features.

Description

Technical Field

[0001] The present application relates to the field of quantum machine learning, and more specifically, to a method and related device for processing multimodal fine-grained data using a quantum neural network. Background Art

[0002] The rapid development of information technology has led to an explosive growth in data types, structures, and generation speeds, which in turn has increased the difficulty of handling large-scale, complex tasks, bringing unprecedented challenges to traditional machine learning algorithms. As a core technology in machine learning, neural networks have demonstrated significant performance in many fields. However, when dealing with large and complex tasks, the development and training of neural networks requires huge computing resources. With the gradual decline of the Moore's Law effect and the gradual emergence of the physical limits of classical computing, the ability of traditional computing devices to handle these demanding tasks is increasingly restricted.

[0003] In this context, quantum computing, as a new physical computing paradigm, provides a unique solution to counter traditional computing technology and thermodynamic limitations. This computing method utilizes the basic principles of quantum mechanics, such as quantum state superposition and quantum entanglement, and can theoretically achieve exponential storage capacity and parallel processing capabilities that exceed traditional computing, opening up new ways to solve complex problems in traditional machine learning algorithms. With the emergence of quantum processors with stable computing performance, quantum machine learning and quantum neural networks have become a hot topic in academia and industry. It combines the efficiency of machine learning technology with the computing power of quantum systems, showing great potential in promoting the development of artificial intelligence.

[0004] At present, the surge in multimodal data has triggered a new round of data analysis revolution. These data integrate multiple forms of information such as images, text, audio and video, and different modalities influence each other, which greatly increases the complexity and amount of information in the data. However, efficiently processing these complex data, analyzing the mutual influence between different modalities, and fusing the correlations between different modalities to perform cross-modal understanding poses a huge challenge to traditional machine learning algorithms. In response to this challenge, we hope to use the advantages of quantum neural networks to cope with it, but how to use quantum neural networks to fully integrate the characteristics of multimodal data has become an urgent problem to be solved. Summary of the invention

[0005] In order to overcome at least one of the deficiencies in the prior art, the present application provides a method and a related device for processing multimodal fine-grained data using a quantum neural network, specifically comprising:

[0006] In a first aspect, the present application provides a method for processing multimodal fine-grained data using a quantum neural network, the method comprising:

[0007] Acquire multiple classical fine-grained features of multiple modes, and prepare corresponding quantum bit sets for each of the classical fine-grained features to obtain multiple quantum bit sets;

[0008] Encoding each classical fine-grained feature into a corresponding quantum bit set so that the multiple quantum bit sets are in a first quantum state, wherein the quantum processing circuits of the multiple quantum bit sets include a feature extraction circuit of each quantum bit set, a feature fusion circuit between two adjacent quantum bit sets, and a measurement circuit of the multiple quantum bit sets;

[0009] By each feature extraction circuit acting on the corresponding quantum bit set, the plurality of quantum bit sets are evolved from the first quantum state to a second quantum state;

[0010] Establishing an entanglement relationship between each quantum bit in two adjacent quantum bit sets and the remaining quantum bits through each characteristic fusion circuit, so that the multiple quantum bit sets evolve from the second quantum state to a third quantum state;

[0011] The measurement circuit acts on the multiple quantum bit sets to obtain processing results of the multiple classical fine-grained features.

[0012] In combination with the optional implementation manner of the first aspect, establishing an entangled relationship between each qubit in two adjacent qubit sets and the remaining qubits through each of the feature fusion circuits includes:

[0013] For two adjacent quantum bit sets, firstly act on each quantum bit therein through a single quantum bit gate;

[0014] Then, according to the arrangement order of each quantum bit in the two adjacent quantum bit sets, each quantum bit is taken as a quantum bit to be entangled in turn, and an entanglement relationship is established between each quantum bit to be entangled and the remaining quantum bits in the two adjacent quantum bit sets.

[0015] In combination with the optional implementation manner of the first aspect, establishing an entanglement relationship between each of the to-be-entangled quantum bits and the remaining quantum bits in two adjacent quantum bit sets includes:

[0016] For each of the quantum bits to be entangled, an entanglement relationship is established between the quantum bit to be entangled and the remaining quantum bits in two adjacent quantum bit sets through a CNOT gate.

[0017] In combination with the optional implementation manner of the first aspect, the remaining quantum bits in the two adjacent quantum bit sets are divided into a first quantum bit subset and a second quantum bit subset by the quantum bits to be entangled;

[0018] The step of establishing an entanglement relationship between the quantum bit to be entangled and the remaining quantum bits in two adjacent quantum bit sets through a CNOT gate includes:

[0019] For the first qubit subset, according to the order from far to near between each qubit in the first qubit subset and the qubit to be entangled, an entanglement relationship between the qubit to be entangled and each qubit in the first qubit subset is established in sequence through CNOT gates;

[0020] For the second quantum bit subset, an entanglement relationship between the quantum bit to be entangled and each quantum bit in the second quantum bit subset is established in sequence through CNOT gates according to the order from far to near between each quantum bit in the second quantum bit subset and the quantum bit to be entangled.

[0021] In combination with an optional implementation manner of the first aspect, the plurality of classic fine-grained features include scene features of the image to be processed, object features of the image to be processed, and text features of the text associated with the image to be processed.

[0022] In combination with the optional implementation manner of the first aspect, the acquiring of multiple multimodal classical fine-grained features includes:

[0023] Processing the scene data of the image to be processed by a pre-trained first neural network model to obtain scene features of the image to be processed;

[0024] Processing the object data of the image to be processed by a pre-trained second neural network model to obtain object features of the image to be processed;

[0025] The associated text is processed through a word embedding model to obtain text features of the associated text.

[0026] In combination with the optional implementation manner of the first aspect, the quantum processing circuit further includes a preparation coding circuit, and the preparation coding circuit includes a coding sub-circuit for each of the quantum bit sets;

[0027] The encoding of each classical fine-grained feature into a corresponding set of quantum bits includes:

[0028] Each classical fine-grained feature is set as a parameter of a quantum gate in a corresponding coding subcircuit, and acts on a corresponding set of quantum bits through the coding subcircuit.

[0029] In a second aspect, the present application also provides a device for processing multimodal fine-grained data using a quantum neural network, the device comprising:

[0030] A quantum initialization module, used to obtain a plurality of classical fine-grained features of multi-modes, and prepare a corresponding quantum bit set for each of the classical fine-grained features to obtain a plurality of quantum bit sets;

[0031] A quantum coding module, used for encoding each classical fine-grained feature into a corresponding quantum bit set, so that the multiple quantum bit sets are in a first quantum state, wherein the quantum processing circuits of the multiple quantum bit sets include a feature extraction circuit of each quantum bit set, a feature fusion circuit between two adjacent quantum bit sets, and a measurement circuit of the multiple quantum bit sets;

[0032] A quantum processing module, configured to act on a corresponding quantum bit set through each feature extraction circuit to evolve the plurality of quantum bit sets from the first quantum state to a second quantum state;

[0033] The quantum processing module is further used to establish an entanglement relationship between each quantum bit in two adjacent quantum bit sets and the remaining quantum bits through each characteristic fusion circuit, so that the multiple quantum bit sets evolve from the second quantum state to a third quantum state;

[0034] A quantum measurement module is used to act on the multiple quantum bit sets through the measurement circuit to obtain processing results of the multiple classical fine-grained features.

[0035] In conjunction with the optional implementation manner of the second aspect, the quantum processing module is further specifically used for:

[0036] For two adjacent quantum bit sets, firstly act on each quantum bit therein through a single quantum bit gate;

[0037] Then, according to the arrangement order of each quantum bit in the two adjacent quantum bit sets, each quantum bit is taken as a quantum bit to be entangled in turn, and an entanglement relationship is established between each quantum bit to be entangled and the remaining quantum bits in the two adjacent quantum bit sets.

[0038] In conjunction with the optional implementation manner of the second aspect, the quantum processing module is further specifically used for:

[0039] For each of the quantum bits to be entangled, an entanglement relationship is established between the quantum bit to be entangled and the remaining quantum bits in two adjacent quantum bit sets through a CNOT gate.

[0040] In combination with the optional implementation manner of the second aspect, the remaining quantum bits in the two adjacent quantum bit sets are divided into a first quantum bit subset and a second quantum bit subset by the quantum bits to be entangled; and the quantum processing module is further specifically used for:

[0041] For the first qubit subset, according to the order from far to near between each qubit in the first qubit subset and the qubit to be entangled, an entanglement relationship between the qubit to be entangled and each qubit in the first qubit subset is established in sequence through CNOT gates;

[0042] For the second quantum bit subset, an entanglement relationship between the quantum bit to be entangled and each quantum bit in the second quantum bit subset is established in sequence through CNOT gates according to the order from far to near between each quantum bit in the second quantum bit subset and the quantum bit to be entangled.

[0043] In combination with an optional implementation manner of the second aspect, the plurality of classic fine-grained features include scene features of the image to be processed, object features of the image to be processed, and text features of the text associated with the image to be processed.

[0044] In conjunction with the optional implementation manner of the second aspect, the quantum initial module is further specifically used for:

[0045] Processing the scene data of the image to be processed by a pre-trained first neural network model to obtain scene features of the image to be processed;

[0046] Processing the object data of the image to be processed by a pre-trained second neural network model to obtain object features of the image to be processed;

[0047] The associated text is processed through a word embedding model to obtain text features of the associated text.

[0048] In conjunction with the optional implementation manner of the second aspect, the quantum processing circuit further includes a preparation coding circuit, the preparation coding circuit includes a coding sub-circuit for each of the quantum bit sets; the quantum coding module is further specifically used for:

[0049] Each classical fine-grained feature is set as a parameter of a quantum gate in a corresponding coding subcircuit, and acts on a corresponding set of quantum bits through the coding subcircuit.

[0050] In a third aspect, the present application also provides a storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of processing multimodal fine-grained data with a quantum neural network.

[0051] In a fourth aspect, the present application also provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method of processing multimodal fine-grained data using a quantum neural network is implemented.

[0052] Compared with the prior art, this application has the following beneficial effects:

[0053] This embodiment provides a method and related device for processing multimodal fine-grained data using a quantum neural network, in which the electronic device encodes multiple classical fine-grained features of the multimodal state into their respective corresponding quantum bit sets, so that the multiple quantum bit sets are in the first quantum state, and each feature extraction circuit acts on the corresponding quantum bit set to evolve the multiple quantum bit sets into the second quantum state; then, each quantum bit in the two adjacent quantum bit sets is entangled with the remaining quantum bits through each feature fusion circuit, so that the multiple quantum bit sets evolve into the third quantum state; finally, the measurement circuit acts on the multiple quantum bit sets to obtain the processing results of multiple classical fine-grained features. In this way, the feature information of different classical fine-grained features is deeply integrated, the intrinsic connection between them is effectively revealed and mined, and the accuracy of the processing results of multiple classical fine-grained features is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1A One of the symbolic schematic diagrams of the CNOT gate provided in the embodiment of the present application;

[0056] Figure 1B The second symbolic schematic diagram of the CNOT gate provided in the embodiment of the present application;

[0057] Figure 2 A flow chart of the method provided in the embodiment of the present application;

[0058] Figure 3 A schematic diagram of the overall structure of a quantum processing circuit provided in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of the structure of the coding circuit provided in the embodiment of the present application;

[0060] Figure 5 A schematic diagram of the structure of a feature extraction circuit provided in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of the structure of a feature fusion circuit provided in an embodiment of the present application;

[0062] Figure 7 A schematic diagram of the structure of a virtual device provided in an embodiment of the present application;

[0063] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0064] Icons: 11-quantum initial module; 12-quantum encoding module; 13-quantum processing module; 14-quantum measurement module; 21-memory; 22-processor; 23-communication unit; 24-system bus. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0066] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0067] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0068] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the invented product is usually placed when in use. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0069] In addition, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0070] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0071] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0072] Based on the above statement, as introduced in the background technology, although the combination of the efficiency of machine learning technology and the computing power of quantum systems has shown great potential in promoting the development of artificial intelligence, when processing the characteristics of multimodal data, how to use quantum neural networks to fully integrate the characteristics of multimodal data has become an urgent problem to be solved.

[0073] Since the present embodiment needs to realize the function of the neural network through quantum circuits, in order to make the embodiments introduced later easier to understand, the relevant terms involved in quantum computing are first explained.

[0074] 1) Quantum computing, quantum computing is a technology that uses the principles of quantum mechanics to process information. In classical computing, the smallest unit of information is a bit, which can only be in a state of 0 or 1. However, in quantum computing, the smallest unit of information is a qubit. The difference between a qubit and a classical bit is that a qubit can exist in a superposition state of 0 and 1 at the same time. This means that a qubit can represent both 0 and 1 at the same time, not just one of them. When we have multiple qubits, this superposition state capability increases exponentially because each qubit can be in a superposition of multiple states at the same time.

[0075] 2) Quantum bit, the basic unit of quantum computing. Compared with traditional computers that use 0 and 1 as the basic units of binary, quantum computing can process 0 and 1 at the same time, so that the system can be in a linear superposition state of 0 and 1: ,in, represents the complex probability amplitude of the system between 0 and 1, and their modulus squared represent the probabilities of being 0 and 1 respectively.

[0076] 3) Quantum processing circuit, an abstract representation of a quantum universal computer, describes the specific hardware implementation of a quantum algorithm or program under the quantum gate model. In quantum computing, quantum gates are basic building blocks used to perform quantum operations, similar to logic gates in classical computing. A quantum processing circuit consists of a series of quantum gates, which, when arranged in a specific order, have specific processing functions for processing information on quantum bits. A parameterized quantum processing circuit refers to a quantum processing circuit that contains some adjustable parameters for controlling quantum gates. These parameters can adjust the operation of quantum gates, thereby changing the behavior of the quantum processing circuit. By adjusting these parameters, the execution of quantum algorithms can be optimized, computing efficiency can be improved, or specific quantum operations can be achieved. Therefore, parameterized quantum processing circuits have a wide range of applications in quantum optimization problems, machine learning and other fields.

[0077] 4) Quantum gates: Quantum gates are the basic operations in quantum computing and are used to transform the state of quantum bits. Mathematically, quantum gates are usually represented by unitary matrices. Unitary matrices are complex square matrices whose characteristic is that their inverse matrices are equal to their conjugate transposed matrices. A quantum gate with 1 qubit can be used The operation of a quantum gate can be achieved by matrix multiplication of the unitary matrix representing the quantum gate and the column vector representing the state of the quantum bit. The state of a quantum bit is usually represented by a state vector, and each element of the state vector corresponds to the probability amplitude of the quantum bit in a specific ground state. When a quantum gate acts on a quantum bit, the state vector will be transformed according to the unitary matrix of the quantum gate. According to the number of quantum bits acting, it can be divided into single-qubit gates, such as Hadamard gates and rotation gates; two-qubit gates, such as CNOT gates and SWAP gates; three-qubit gates, such as Toffoli gates and Fredkin gates.

[0078] 5) Revolving door, a type of quantum gate, is a set of three The unitary matrix (also known as unitary Hermitian complex matrix) of Door, revolving Doors and revolving The mathematical expressions of these revolving doors are explained below:

[0079] Rotation Door (also known as The gate) can be expressed as:

[0080]

[0081] Rotation Door (also known as The gate) can be expressed as:

[0082]

[0083] Rotation Door (also known as The gate) can be expressed as:

[0084]

[0085] 6) CNOT gate, which is used to operate on two qubits, one of which is used as a control bit and the other as a target bit. The basic function of the CNOT gate is that when the control bit is When the control bit is , then the state of the target bit remains unchanged. The CNOT gate can be represented as the following unitary matrix:

[0086]

[0087] It should be noted that in quantum processing circuits, CNOT gates have Figure 1A , Figure 1B There are two ways of representation. In the figure, between any two quantum bits, the smaller circular pattern (not marked with the symbol "+") represents the control bit, and the larger circular pattern (marked with the symbol "+") represents the target bit; the meaning of the CNOT gate is that when the control bit is When the control bit is When the quantum bit is in the state, a quantum NOT gate operation is performed on the target bit, thereby realizing quantum entanglement between quantum bits.

[0088] 7) Quantum measurement: Quantum measurement is a concept in quantum mechanics that describes the process by which a property of a quantum system is deterministically acquired under certain conditions. Before measurement, the quantum system is in a superposition of multiple possible states. The measurement operation causes the wave function to collapse to a specific eigenstate, which corresponds to the measurement result. Since the measurement process causes the quantum system to enter a deterministic reality state from a possible superposition state, the quantum system exhibits the characteristics of classical physics according to the measurement result. Therefore, quantum measurement is often used to convert quantum states into classical data.

[0089] Based on the introduction of the above-mentioned related terms, the method for processing multimodal fine-grained data using a quantum neural network provided in the embodiment of the present application will be clearly and completely described below in combination with the provided drawings. However, it should be noted that the defects existing in the above-mentioned solutions in the prior art are the results obtained by the inventor after practice and careful study. Therefore, the discovery process of the above-mentioned problems and the solutions proposed by the embodiments of the present application for the above-mentioned problems below should all be the contributions made by the inventor to the present application in the process of invention and creation, and should not be understood as the technical content known to those skilled in the art.

[0090] In view of the above problems, this embodiment provides a method for processing multimodal fine-grained data by a quantum neural network. In this method, an electronic device obtains multiple classical fine-grained features of a multimodal state; and prepares a corresponding quantum bit set for each classical fine-grained feature to obtain multiple quantum bit sets; each classical fine-grained feature is encoded into a corresponding quantum bit set, so that the multiple quantum bit sets are in a first quantum state, wherein the quantum processing circuits of the multiple quantum bit sets include a feature extraction circuit for each quantum bit set, a feature fusion circuit between two adjacent quantum bit sets, and a measurement circuit for multiple quantum bit sets; each feature extraction circuit acts on the corresponding quantum bit set to evolve the multiple quantum bit sets from the first quantum state to the second quantum state; each quantum bit in the two adjacent quantum bit sets is entangled with the remaining quantum bits through each feature fusion circuit, so that the multiple quantum bit sets evolve from the second quantum state to the third quantum state; the measurement circuit acts on the multiple quantum bit sets to obtain the processing results of multiple classical fine-grained features.

[0091] In this way, after converting a variety of classical fine-grained features into quantum states, in the feature fusion circuit, each quantum bit in two adjacent quantum bit sets is entangled with the remaining quantum bits, thereby deeply integrating the feature information of different classical fine-grained features, effectively revealing and mining the intrinsic connections between them, and thus significantly improving the accuracy of the processing results of a variety of classical fine-grained features.

[0092] It should be noted that the electronic device implementing the method of processing multimodal fine-grained data with a quantum neural network can be a quantum computer based on the principles of quantum mechanics or a classical computer capable of simulating the quantum computing process. The quantum computer can be, but is not limited to, a superconducting quantum computer, an ion trap quantum computer, an optical quantum computer, a nuclear magnetic resonance quantum computer, etc. These quantum computers realize the manipulation of quantum bits based on their own unique physical systems and manipulation technologies, and each technical path has its advantages and limitations.

[0093] To make the solution provided by this embodiment clearer, Figure 2Each step of the method is described in detail. However, it should be understood that the operations of the flowchart may not be implemented in order, and steps without logical contextual relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart, or remove one or more operations from the flowchart under the guidance of the content of this application. Figure 2 As shown, the method includes:

[0094] S1, obtain multiple classical fine-grained features of multi-mode, and prepare corresponding quantum bit sets for each classical fine-grained feature to obtain multiple quantum bit sets.

[0095] At present, in the related technology of processing multimodal data through quantum processing circuits, the multimodal data will be simply pre-processed and then input into the quantum processing circuit for processing. In practice, it is found that this method requires a large number of quantum bits to represent high-dimensional data, resulting in extremely high requirements for quantum hardware; in addition, the noise and redundant information of the original data may cause the error rate of quantum computing to increase. Therefore, the complexity of the data may mask the advantages of quantum computing.

[0096] Compared with the data input into the quantum processing circuit which has only undergone simple preprocessing, this embodiment inputs a variety of classical fine-grained features extracted from the multimodal data into the quantum processing circuit for processing. Specifically, the multimodal data in this embodiment includes the image data to be processed and its associated text, and the image data to be processed includes the object data and scene data of the image to be processed. It can be understood that for the image to be processed, this embodiment separates the scene and object in the image by image segmentation, and extracts features respectively, so that a variety of classical fine-grained features include the scene features of the image to be processed, the object features of the image to be processed, and the text features of the corresponding associated text of the image to be processed.

[0097] These classical fine-grained features can be extracted from the corresponding data by classical feature extraction methods. In this embodiment, the scene data of the image to be processed can be processed by a pre-trained first neural network model to obtain the scene features of the image to be processed; the object data of the image to be processed can be processed by a pre-trained second neural network model to obtain the object features of the image to be processed; and the associated text can be processed by a word embedding model to obtain the text features of the associated text.

[0098] Exemplarily, for the above-mentioned fine-grained image data, high-quality image data with uniform format is first obtained through image preprocessing methods such as scaling, normalization, denoising, centering and standardization, and then the preprocessed image data with different fine-grained attributes are vectorized by deep learning-based methods (autoencoders, neural networks, etc.) for different fine-grained attributes of the image, such as scenes and objects in the image, to obtain vector representations of the original data, wherein the appropriate vector dimension can be set manually. For associated text, the associated text can be cleaned by text preprocessing methods such as punctuation removal, case normalization, word segmentation, and stop word removal, and then the cleaned associated text is converted into a fixed-dimensional vector representation by using discrete representation (such as one-hot encoding) or distributed representation methods (a method for efficiently encoding text data into dense, low-dimensional vector space).

[0099] In addition, since this embodiment needs to process multiple classical fine-grained features of multi-modes through quantum processing circuits, it is necessary to prepare corresponding quantum bits for converting multiple classical fine-grained features into quantum states. In this embodiment, the prepared quantum bits are divided into multiple quantum bit sets in a logical sense according to the number of multiple classical fine-grained features, and each quantum bit set is used to encode the corresponding classical fine-grained features.

[0100] For the quantum processing circuit provided in this embodiment, Figure 3 The overall structure of the circuit is shown, including the preparation encoding circuit, the feature extraction circuit of each quantum bit set, the feature fusion circuit between two adjacent quantum bit sets, and the measurement circuit of multiple quantum bit sets. Figure 3 For an introduction to the quantum processing circuit shown, see Figure 2 The method for processing multimodal fine-grained data using a quantum neural network provided in this embodiment further includes:

[0101] S2, encoding each classical fine-grained feature into a corresponding quantum bit set so that the multiple quantum bit sets are in a first quantum state.

[0102] For the above multi-modal classical fine-grained features, Figure 3 The preparation encoding circuit in the quantum processing circuit shown encodes a variety of classical fine-grained features into quantum states; Figure 4 The circuit structure of the coding circuit is further shown. Since the present embodiment involves multiple classical fine-grained features of multi-mode, the coding circuit includes a coding subcircuit for each quantum bit set at the logical level, each classical fine-grained feature is set as a parameter of a quantum gate in the corresponding coding subcircuit, and acts on the corresponding quantum bit set through the coding subcircuit, so that the multiple quantum bit sets are in the first quantum state.

[0103] For example, the quantum circuit is initialized as state, among which, The number of qubits in the quantum circuit is represented by the vector representation of each classical fine-grained feature, and then the value in the vector representation is set as the parameter of the quantum gate in the encoding subcircuit; and the single-qubit gate is cyclically applied to each qubit in turn until all the values ​​in the vector representation are loaded, so that multiple qubit sets are The state evolves into the first quantum state.

[0104] It should be noted that the quantum state of a single quantum bit can be expressed as ,in, and are the two ground states of a single quantum bit, with coefficients and Satisfy the normalization condition Multiple qubits can form a multi-qubit system, whose quantum state space is the tensor product of the quantum state of each qubit, which can be expressed as ,in yes The ground state of qubits corresponds to the integer The binary representation of .

[0105] Quantum processing circuits consist of wires and quantum gates to transmit and manipulate quantum information. Quantum gates operate on the state of quantum bits, determining the evolution of the entire quantum system. The evolution of a quantum system can be described by quantum gates (unitary transformations), which satisfy Unitary operator It indicates that, for The conjugate transpose of express dimensional identity matrix. Quantum gate Acts on The following transformation will be obtained:

[0106] ;

[0107] Continue to see Figure 3 , Figure 3 In the quantum processing circuit shown, each feature extraction circuit is used to extract features from single-modal data, and then multiple feature fusion circuits are combined to fuse the features of multi-modal data.

[0108] S3, by acting on the corresponding quantum bit set through each feature extraction circuit, multiple quantum bit sets are evolved from the first quantum state to the second quantum state.

[0109] In this embodiment, each feature extraction circuit can be personalized for the classic fine-grained features of different modalities. Figure 5 As shown, regardless of whether the configuration is the same, the key is to ensure that each feature extraction circuit is specifically designed to extract the classical fine-grained features of its corresponding mode. In this embodiment, a general configuration for extracting features through quantum circuits is adopted. This configuration is similar to the architecture of convolutional neural networks. Based on the characteristics of local connections in convolutional neural networks, two-qubit gates are connected. The two-qubit gates in each feature extraction circuit are used to extract the features of the corresponding mode by applying them to adjacent single qubits in sequence and using these quantum gates to operate on qubits to realize the evolution of quantum states. The parameters are the same as in .

[0110] In addition, the two-qubit gate in each feature extraction circuit It is a universal two-qubit gate that can realize arbitrary quantum state conversion to improve the expression ability of the circuit. Figure 5 , decomposes the universal two-qubit gate into a series of basic quantum gates, including 3 controlled NOT gates and 15 single-qubit rotation gates. This decomposition strategy minimizes the number of quantum gates, thereby enhancing the compactness of the circuit and the feasibility of practical implementation. Among them, the single-qubit rotation gate contains adjustable parameters Door, Door and Door.

[0111] Based on the above implementation, the evolution of multiple quantum bit sets from the second quantum state to the third quantum state is introduced. Figure 2 The method for processing multimodal fine-grained data using a quantum neural network provided in this embodiment further includes:

[0112] S4, through each feature fusion circuit, each quantum bit in two adjacent quantum bit sets is entangled with the remaining quantum bits, so that multiple quantum bit sets evolve from the second quantum state to the third quantum state.

[0113] In practice, it is found that if each quantum bit is entangled with the other quantum bits in a specific order, a better fusion effect can be achieved. Therefore, for the above step S4, this embodiment provides the following implementation method:

[0114] For two adjacent qubit sets, first act on each qubit therein through a single qubit gate; then, according to the arrangement order of each qubit in the two adjacent qubit sets, take each qubit as a qubit to be entangled in turn, and establish an entangled relationship between each qubit to be entangled and the remaining qubits in the two adjacent qubit sets. In a specific implementation, for each qubit to be entangled, establish an entangled relationship between the qubit to be entangled and the remaining qubits in the two adjacent qubit sets through a CNOT gate.

[0115] For example, Figure 6 In the characteristic fusion circuit shown, six qubits in two qubit sets are shown, represented as According to the above embodiment, first The single-qubit gates represented by Each qubit is used as the entangled qubit in turn, and an entanglement relationship is established between it and the remaining 5 qubits through the CNOT gate. As the qubit to be entangled, Establish entanglement relationship through CNOT gate respectively; then, select As the qubit to be entangled, The entanglement relationship is established through the CNOT gate respectively, and so on, so that each quantum bit is entangled with the remaining quantum bits.

[0116] In addition, the study also found that the entanglement order between each qubit to be entangled and the remaining qubits will also affect the feature fusion effect. In this embodiment, the remaining qubits in the two adjacent qubit sets are divided into a first qubit subset and a second qubit subset by the qubit to be entangled; for the first qubit subset, according to the order from far to near between each qubit in the first qubit subset and the qubit to be entangled, the entanglement relationship between the qubit to be entangled and each qubit in the first qubit subset is established through the CNOT gate in sequence; for the second qubit subset, according to the order from far to near between each qubit in the second qubit subset and the qubit to be entangled, the entanglement relationship between the qubit to be entangled and each qubit in the second qubit subset is established through the CNOT gate in sequence.

[0117] For example, see Figure 6 The feature fusion line shown in the figure, when selecting As a quantum bit to be entangled, First, the one with the furthest position in the arrangement Establish an entanglement relationship; then, with the second farthest Establish entanglement relationship; and so on, establish The entanglement relationship between the qubits and the rest. As a quantum bit to be entangled, Will Figure 6 The remaining qubits are divided into the first qubit subset With the second subset of qubits ; For the first subset of qubits , distance The farthest, then build first and The entanglement relationship between them, then, establish and Similarly, for the second quantum bit subset , First, the one with the furthest position in the arrangement Establish an entanglement relationship; then, with the second farthest Establish entanglement relationship, and so on, establish The entangled relationship between it and the rest of the quantum bits.

[0118] In the above implementation, it is introduced that in the feature fusion circuit, the universal single-qubit rotation gate is applied to all qubits respectively, and the CNOT gate is used on each qubit to realize the connection with other qubits, forming a quantum processing circuit with strong entanglement characteristics, thereby realizing the fusion of the above-mentioned acquired single-modal fine-grained features. Figure 2 The method for processing multimodal fine-grained data using a quantum neural network provided in this embodiment also includes:

[0119] S5, by measuring the circuit acting on multiple quantum bit sets, obtains the processing results of multiple classical fine-grained features.

[0120] The processing result is related to the training target of the line, and the processing result can be a sentiment classification result, an image classification result or a target detection result of multiple classic fine-grained features of multimodality.

[0121] It should be noted that in quantum computing, information can be extracted from quantum states through measurement circuits to obtain processing results in classical form. Specifically, for projection measurement, assume is a quantum observable and can be represented by a Hermitian operator. According to the spectral decomposition theorem, It can be expressed as ,in, Express Possible results when performing projection measurements, Representatives and The eigenvalue of The corresponding feature space projector. Assume that the quantum system is in the state , then the measurement observable The expected value of is calculated as .

[0122] In this embodiment, a measurement circuit is constructed by performing projection measurement on a specific quantum bit, and the entire quantum processing circuit is repeatedly executed and the quantum observable measurement is used. To measure the last qubit, the measurement result can be expressed as Expected value:

[0123] ;

[0124] in, The input quantum states represent multimodal fine-grained data, represents the entire unitary operation of the quantum neural network model, Represents the parameter vector of the model, each element corresponds to a single-qubit rotation gate Parameters in . Using Z measurement, this configuration ensures the expected value exist In the range of 100,000, especially for the binary classification task of multimodal fine-grained data, it can be directly used as the objective function of the model. For other tasks, the mapping between the output of the quantum processing circuit and the expected category can be used to construct the objective function of the model.

[0125] It should be noted that, similar to the classical neural network, after the above quantum processing circuit is built, it needs to be trained to complete the desired data processing task. In this embodiment, the cross entropy loss function is used as a measurement indicator to evaluate the error between the output result of the quantum processing circuit and the target output corresponding to the input data. For the cross entropy loss function of a single sample, this embodiment defines it as:

[0126] ;

[0127] in, is the actual target output, is the expected value of the output of the quantum neural network According to the error analysis results, the gradient calculation method suitable for parameterized quantum processing circuits and the gradient descent optimization algorithm are used to update the adjustable parameters in the circuit to minimize the loss function. The specific implementation process is as follows:

[0128] 1) Calculate the gradient of the loss function

[0129] In this embodiment, the gradient calculation method applicable to parameterized quantum processing circuits is used to calculate the gradient, namely, the parameter offset rule and the chain rule. Specifically, in order to simplify the expression, (The same applies to the case of multiple samples), For parameters The derivative of is:

[0130]

[0131] in, The calculation of is based on the parameter shift rule:

[0132]

[0133] in, It is a A vector with 1 elements and the rest 0.

[0134] 2) Update parameters

[0135] According to the calculated , the parameters are updated using the stochastic gradient optimization algorithm, which updates the parameters of the quantum neural network according to the gradient descent direction of the loss function .in, is the learning rate, which controls the step size of the parameter update. By iterating the parameters in the quantum processing circuit in the above way, the output of the quantum neural network is close to the target output of the training data.

[0136] Based on the same inventive concept as the method for processing multimodal fine-grained data with a quantum neural network provided in the above embodiment, this embodiment also provides a device for processing multimodal fine-grained data with a quantum neural network. The device includes at least one software function module that can be stored in the memory 21 or solidified in the electronic device in the form of software. The processor 22 in the electronic device is used to execute the executable module stored in the memory 21. For example, the software function module and computer program included in the device. Please refer to Figure 7 , functionally speaking, the device may include:

[0137] A quantum initialization module 11 is used to obtain multiple classical fine-grained features of multi-modes, and prepare a corresponding quantum bit set for each classical fine-grained feature to obtain multiple quantum bit sets;

[0138] A quantum coding module 12, used to encode each classical fine-grained feature into a corresponding quantum bit set, so that the multiple quantum bit sets are in a first quantum state, wherein the quantum processing circuits of the multiple quantum bit sets include a feature extraction circuit of each quantum bit set, a feature fusion circuit between two adjacent quantum bit sets, and a measurement circuit of the multiple quantum bit sets;

[0139] A quantum processing module 13, used to act on the corresponding quantum bit set through each feature extraction circuit to evolve the multiple quantum bit sets from the first quantum state to the second quantum state;

[0140] The quantum processing module 13 is further used to establish an entanglement relationship between each quantum bit in two adjacent quantum bit sets and the remaining quantum bits through each characteristic fusion circuit, so that the multiple quantum bit sets evolve from the second quantum state to the third quantum state;

[0141] The quantum measurement module 14 is used to act on multiple quantum bit sets through measurement circuits to obtain processing results of multiple classical fine-grained features.

[0142] In this embodiment, the quantum initial module 11 is used to implement Figure 2 In step S1, the quantum coding module 12 is used to implement Figure 2 In step S2, the quantum processing module 13 is used to implement Figure 2 In steps S3 and S4, the quantum measurement module 14 is used to implement Figure 2 Therefore, for the detailed description of the above modules, please refer to the specific implementation of the corresponding steps. Of course, in view of the same inventive concept as the method for processing multimodal fine-grained data with a quantum neural network, the above modules can also be used to implement other steps or sub-steps of the method, which will not be described in detail in this embodiment.

[0143] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0144] It should also be understood that if the above implementation is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0145] Therefore, this embodiment also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, for example, the computer program describes the structure of the quantum processing circuit. When the computer program is executed by the processor, the method for processing multimodal fine-grained data by a quantum neural network provided in this embodiment is implemented. Among them, the storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, etc., which can store program codes.

[0146] This embodiment also provides an electronic device for implementing a method of quantum neural network processing multi-modal fine-grained data. Figure 8 As shown, the electronic device may include a processor 22 and a memory 21. In addition, the memory 21 stores a computer program, and the processor implements the method for processing multimodal fine-grained data using a quantum neural network provided in this embodiment by reading and executing the computer program corresponding to the above implementation in the memory 21.

[0147] Continue to see Figure 8 The electronic device further includes a communication unit 23. The memory 21, the processor 22 and the communication unit 23 are electrically connected to each other directly or indirectly through a system bus 24 to achieve data transmission or interaction.

[0148] The memory 21 may be an information recording device based on any electronic, magnetic, optical or other physical principle, used to record execution instructions, data, etc. In some embodiments, the memory 21 may be, but is not limited to, a volatile memory, a non-volatile memory, a storage drive, etc.

[0149] In some embodiments, the volatile memory may be a random access memory (RAM); in some embodiments, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a flash memory, etc.; in some embodiments, the storage drive may be a disk drive, a solid-state drive, any type of storage disk (such as a CD, a DVD, etc.), or a similar storage medium, or a combination thereof, etc.

[0150] The communication unit 23 is used to send and receive data through a network. In some embodiments, the network may include a wired network, a wireless network, a fiber optic network, a telecommunication network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof. In some embodiments, the network may include one or more network access points. For example, the network may include a wired or wireless network access point, such as a base station and / or a network switching node, and one or more components of the service request processing system may be connected to the network through the access point to exchange data and / or information.

[0151] The processor 22 may include a quantum processor, i.e., a processor that uses quantum bits for information processing. Unlike traditional classical computer processors, quantum processors use quantum mechanical principles, such as superposition and entanglement, to perform calculations. In some embodiments, the processor may also include a classical computer processor, which, as an integrated circuit chip, has the ability to process signals, and the processor may include one or more processing cores (e.g., a single-core processor or a multi-core processor). By way of example only, the above-mentioned processor may include a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), or a microprocessor, or any combination thereof.

[0152] Understandably, Figure 8 The structure shown is for illustration only. The electronic device may also have Figure 8 More or fewer components than shown, or with Figure 8 Different configurations are shown. Figure 8 The components shown may be implemented in hardware, software or a combination thereof.

[0153] It should be understood that the apparatus and method disclosed in the above-mentioned embodiments can also be implemented in other ways. The apparatus embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0154] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for processing multimodal fine-grained data using a quantum neural network, characterized in that: The method comprises: Acquire multiple classical fine-grained features of multimodality, and prepare corresponding quantum bit sets for each of the classical fine-grained features to obtain multiple quantum bit sets, wherein the multiple classical fine-grained features include scene features of the image to be processed, object features of the image to be processed, and text features of the text associated with the image to be processed; Encoding each classical fine-grained feature into a corresponding quantum bit set so that the multiple quantum bit sets are in a first quantum state, wherein the quantum processing circuits of the multiple quantum bit sets include a feature extraction circuit of each quantum bit set, a feature fusion circuit between two adjacent quantum bit sets, and a measurement circuit of the multiple quantum bit sets; By each feature extraction circuit acting on the corresponding quantum bit set, the plurality of quantum bit sets are evolved from the first quantum state to a second quantum state; Establishing an entanglement relationship between each quantum bit in two adjacent quantum bit sets and the remaining quantum bits through each characteristic fusion circuit, so that the multiple quantum bit sets evolve from the second quantum state to a third quantum state; The measurement circuit acts on the multiple quantum bit sets to obtain processing results of the multiple classical fine-grained features.

2. The method for processing multimodal fine-grained data using a quantum neural network according to claim 1, characterized in that: The step of establishing an entangled relationship between each quantum bit in two adjacent quantum bit sets and the remaining quantum bits through each characteristic fusion circuit comprises: For two adjacent quantum bit sets, firstly act on each quantum bit therein through a single quantum bit gate; Then, according to the arrangement order of each quantum bit in the two adjacent quantum bit sets, each quantum bit is taken as a quantum bit to be entangled in turn, and an entanglement relationship is established between each quantum bit to be entangled and the remaining quantum bits in the two adjacent quantum bit sets.

3. The method for processing multimodal fine-grained data using a quantum neural network according to claim 2, characterized in that: Establishing an entanglement relationship between each of the to-be-entangled quantum bits and the remaining quantum bits in two adjacent quantum bit sets, comprising: For each of the quantum bits to be entangled, an entanglement relationship is established between the quantum bit to be entangled and the remaining quantum bits in two adjacent quantum bit sets through a CNOT gate.

4. The method for processing multimodal fine-grained data using a quantum neural network according to claim 3, characterized in that: The remaining qubits in the two adjacent qubit sets are divided into a first qubit subset and a second qubit subset by the qubits to be entangled; The step of establishing an entanglement relationship between the quantum bit to be entangled and the remaining quantum bits in two adjacent quantum bit sets through a CNOT gate includes: For the first qubit subset, according to the order from far to near between each qubit in the first qubit subset and the qubit to be entangled, an entanglement relationship between the qubit to be entangled and each qubit in the first qubit subset is established in sequence through CNOT gates; For the second quantum bit subset, an entanglement relationship between the quantum bit to be entangled and each quantum bit in the second quantum bit subset is established in sequence through CNOT gates according to the order from far to near between each quantum bit in the second quantum bit subset and the quantum bit to be entangled.

5. The method for processing multimodal fine-grained data using a quantum neural network according to claim 1, characterized in that: The multi-modal classical fine-grained features include: Processing the scene data of the image to be processed by a pre-trained first neural network model to obtain scene features of the image to be processed; Processing the object data of the image to be processed by a pre-trained second neural network model to obtain object features of the image to be processed; The associated text is processed through a word embedding model to obtain text features of the associated text.

6. The method for processing multimodal fine-grained data using a quantum neural network according to claim 1, characterized in that: The quantum processing circuit also includes a preparation coding circuit, and the preparation coding circuit includes a coding subcircuit for each of the quantum bit sets; The encoding of each classical fine-grained feature into a corresponding set of quantum bits includes: Each classical fine-grained feature is set as a parameter of a quantum gate in a corresponding coding subcircuit, and acts on a corresponding set of quantum bits through the coding subcircuit.

7. A device for processing multimodal fine-grained data using a quantum neural network, characterized in that: The device comprises: A quantum initialization module, used to obtain a plurality of classical fine-grained features of multimodality, and prepare a corresponding quantum bit set for each of the classical fine-grained features to obtain a plurality of quantum bit sets, wherein the plurality of classical fine-grained features include scene features of the image to be processed, object features of the image to be processed, and text features of the text associated with the image to be processed; A quantum coding module, used for encoding each classical fine-grained feature into a corresponding quantum bit set, so that the multiple quantum bit sets are in a first quantum state, wherein the quantum processing circuits of the multiple quantum bit sets include a feature extraction circuit of each quantum bit set, a feature fusion circuit between two adjacent quantum bit sets, and a measurement circuit of the multiple quantum bit sets; A quantum processing module, configured to act on a corresponding quantum bit set through each feature extraction circuit to evolve the plurality of quantum bit sets from the first quantum state to a second quantum state; The quantum processing module is further used to establish an entanglement relationship between each quantum bit in two adjacent quantum bit sets and the remaining quantum bits through each characteristic fusion circuit, so that the multiple quantum bit sets evolve from the second quantum state to a third quantum state; A quantum measurement module is used to act on the multiple quantum bit sets through the measurement circuit to obtain processing results of the multiple classical fine-grained features.

8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method for processing multimodal fine-grained data using a quantum neural network as described in any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method for processing multimodal fine-grained data using a quantum neural network as described in any one of claims 1 to 6 is implemented.

Images