Visual information identification method based on pulse Transform network

Abstract

The invention provides a visual information identification method based on a pulse Transform network. The method comprises the following steps: acquiring a pulse sequence of visual information to be identified; based on a pulse Transform network, constructing a visual information identification model; wherein one or more primary functions in a Fourier primary function or a wavelet primary function are adopted to construct a priori knowledge embedding layer; a priori knowledge embedding layer is adopted to replace an attention module in the pulse Transform network; and inputting the pulse sequence into the visual information identification model to obtain an identification result of the visual information to be identified. Through adoption of the method, the problem that the accuracy is relatively low when visual information identification is carried out by using a pulse Transform network due to a discrete pulse coding mode of a pulse neural network for a real value in related technologies is solved.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a visual information recognition method based on a pulse Transformer network. Background Art

[0002] The Spiking Transformer Network combines the advantages of SNN in processing spike data with the ability of Transformer to capture long-distance dependencies, and is suitable for processing spike data with spatiotemporal characteristics. Through the self-attention mechanism, the Spiking Transformer Network can capture long-distance dependencies in spike data and improve the modeling ability of the model.

[0003] The pulse Transformer network can realize the recognition function of visual information, but due to the discrete pulse encoding method of the pulse neural network for real values, the accuracy of visual information recognition using the pulse Transformer network is low. Summary of the invention

[0004] The present invention provides a visual information recognition method based on a pulse Transformer network, which is used to solve the problem in the prior art that the accuracy of visual information recognition using a pulse Transformer network is low due to the discrete pulse encoding method of the pulse neural network for real values.

[0005] The present invention provides a visual information recognition method based on a pulse Transformer network, comprising: Acquire a pulse sequence of visual information to be recognized; Based on the pulse Transformer network, a visual information recognition model is constructed; wherein one or more basis functions of Fourier basis functions or wavelet basis functions are used to construct a priori knowledge embedding layer; and the priori knowledge embedding layer is used to replace the attention module in the pulse Transformer network; The pulse sequence is input into the visual information recognition model to obtain the recognition result of the visual information to be recognized.

[0006] According to the visual information recognition method based on the pulse Transformer network provided by the present invention, the step of inputting the pulse sequence into the visual information recognition model to obtain the recognition result of the visual information to be recognized includes: Inputting the pulse sequence into the convolutional coding layer in the visual information recognition model for preliminary coding; Inputting the initially encoded pulse sequence into the deep encoding layer in the visual information recognition model for deep encoding; The pulse sequence after depth encoding is input into the classification layer in the visual information recognition model to obtain the recognition result of the visual information to be recognized.

[0007] According to the visual information recognition method based on the pulse Transformer network provided by the present invention, the prior knowledge embedding layer is located in the deep encoding layer.

[0008] According to the visual information recognition method based on the pulse Transformer network provided by the present invention, the pulse sequence of obtaining the visual information to be recognized includes: When the visual information to be identified is a static RGB image, the static RGB image is input into an RGB encoding module to obtain a pulse sequence of the visual information to be identified.

[0009] According to the visual information recognition method based on the pulse Transformer network provided by the present invention, the pulse sequence of obtaining the visual information to be recognized includes: When the visual information to be identified is DVS data, the DVS data is directly used as the pulse sequence of the visual information to be identified.

[0010] According to the visual information recognition method based on the pulse Transformer network provided by the present invention, the prior knowledge embedding layer is constructed by using one or more basis functions of Fourier basis functions or wavelet basis functions, including: Using sine functions and cosine functions of different frequencies in Fourier basis functions to construct a priori knowledge embedding matrix to obtain the priori knowledge embedding layer; or

[0011] Cosine functions of different frequencies in Fourier basis functions are used to construct a priori knowledge embedding matrix to obtain the priori knowledge embedding layer.

[0012] According to the visual information recognition method based on the pulse Transformer network provided by the present invention, the prior knowledge embedding layer is constructed by using one or more basis functions of Fourier basis functions or wavelet basis functions, including: One or more discrete wavelet basis functions are used to construct a priori knowledge embedding matrix to obtain the priori knowledge embedding layer.

[0013] The present invention also provides a visual information recognition device based on a pulse Transformer network, comprising: A visual information processing unit, used for acquiring a pulse sequence of visual information to be recognized; A model building unit, for building a visual information recognition model based on a pulse Transformer network; wherein one or more basis functions of Fourier basis functions or wavelet basis functions are used to build a priori knowledge embedding layer; and the priori knowledge embedding layer is used to replace the attention module in the pulse Transformer network; The visual information recognition unit is used to input the pulse sequence into the visual information recognition model to obtain the recognition result of the visual information to be recognized.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a visual information recognition method based on a pulse Transformer network as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a visual information recognition method based on a pulse Transformer network as described in any one of the above.

[0016] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described visual information recognition methods based on a pulse Transformer network.

[0017] The visual information recognition method based on the pulse Transformer network provided by the present invention constructs a visual information recognition model through a Transformer network based on a pulse framework, and introduces Fourier basis functions or wavelet basis functions to construct a priori knowledge embedding layer to replace the attention module of the pulse Transformer network. The accuracy of visual information recognition is improved, and the energy consumption of the visual information recognition process is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the visual information recognition method based on the pulse Transformer network provided by the present invention.

[0020] Figure 2It is a schematic diagram of the principle of the visual information recognition method based on the pulse Transformer network provided by the present invention.

[0021] Figure 3 This is the second flow chart of the visual information recognition method based on the pulse Transformer network provided by the present invention.

[0022] Figure 4 It is a schematic diagram of the principle of the visual information recognition model provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of a visual information recognition device based on a pulse Transformer network provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the descriptions used in this way can be interchanged where appropriate, so that the embodiments can be implemented in a sequence other than that illustrated or described in the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps that appear in the present application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed in multiple circuit units, and some or all of the units may be selected according to actual needs to achieve the purpose of the present application.

[0027] In the present invention, the pulse Transformer network can realize the recognition function of visual information, but due to the discrete pulse encoding method of the pulse neural network for real values, the accuracy of visual information recognition using the pulse Transformer network model is low. In order to solve the problems existing in the related art, the present invention constructs a visual information recognition model based on the improved pulse Transformer network, so that the accuracy of the visual information recognition model in recognizing visual information is improved.

[0028] In the present invention, the recognition results of visual information can be applied to various fields, such as for dynamically changing scenes, such as moving object detection, tracking and behavior analysis, the visual information recognition model captures and analyzes the changes in the scene by recognizing real-time visual information, and provides accurate dynamic information. In the embodiments of the present invention, the specific application of the recognition results of visual information is not described in detail.

[0029] The following describes relevant concepts in the embodiments of the present invention.

[0030] Fourier transform is a mathematical method for converting a signal from the time domain to the frequency domain. It is based on the concept of the Fourier series, which states that any periodic function can be represented as the sum of sine and cosine functions. Discrete Fourier transform (DFT) is a discrete version of the continuous Fourier transform and is used to process discrete signals. Fast Fourier transform (FFT) is an efficient algorithm for calculating DFT, and its time complexity is much lower than directly calculating DFT.

[0031] The wavelet transform is a mathematical tool for signal analysis that provides a time-frequency representation of a signal. Unlike the Fourier transform, the wavelet transform allows the analysis of local characteristics of a signal, which makes it very useful when dealing with non-stationary signals (signals whose frequency varies over time). The discrete wavelet transform (DWT) is a discrete version of the continuous wavelet transform.

[0032] Combine the following Figure 1-Figure 5 The specific contents of the present invention are described.

[0033] Figure 1 One of the flow charts of the visual information recognition method based on the pulse Transformer network provided by the present invention includes steps S101-S103.

[0034] Step S101, obtaining a pulse sequence of visual information to be recognized.

[0035] Step S102, constructing a visual information recognition model based on the pulse Transformer network; wherein, one or more basis functions of Fourier basis functions or wavelet basis functions are used to construct a priori knowledge embedding layer; and the priori knowledge embedding layer is used to replace the attention module in the pulse Transformer network.

[0036] Specifically, the prior knowledge embedding layer can be a Fourier transform embedding layer or a wavelet transform embedding layer, wherein the dimension of the prior knowledge embedding matrix is ​​consistent with the QK matrix in the self-attention module, and the value of each basis function remains fixed during the training process.

[0037] Step S103, inputting the pulse sequence into the visual information recognition model to obtain the recognition result of the visual information to be recognized.

[0038] In a possible implementation, in step S101, a pulse sequence of visual information to be identified is obtained, including: when the visual information to be identified is a static RGB image, the static RGB image is input into an RGB encoding module to obtain a pulse sequence of the visual information to be identified.

[0039] Specifically, if Figure 2As shown in FIG. 1 , a static RGB image is input into an RGB data encoding module to obtain an encoded pulse sequence as the input of a visual information recognition model. In an embodiment of the present invention, for a static RGB image, the time step of the pulse sequence is The time step can be determined based on the size of the static RGB image. is proportional to the size of the static RGB image. It is usually set between 1 and 4.

[0040] In a possible implementation, in step S101, obtaining a pulse sequence of visual information to be identified includes: when the visual information to be identified is DVS data, directly using the DVS data as the pulse sequence of the visual information to be identified.

[0041] In an embodiment of the present invention, DVS data (Dynamic Vision Sensor Dataset) is generated by an event camera, which generates events based on brightness changes at the pixel level. Each event contains the location of the pixel, a timestamp, and the polarity of the brightness change (brightening or darkening). Because DVS data records discrete events that change over time, it is essentially a pulse sequence, so DVS data can be directly used as a pulse sequence of visual information to be identified.

[0042] Figure 3 The second flowchart of the visual information recognition method based on the pulse Transformer network provided by the present invention.

[0043] In a possible implementation, in step S103, the pulse sequence is input into a visual information recognition model to obtain a recognition result of the visual information to be recognized, including steps S301-S303.

[0044] Step S301, input the pulse sequence into the convolutional coding layer in the visual information recognition model for preliminary coding.

[0045] Step S302, inputting the initially encoded pulse sequence into a deep encoding layer in the visual information recognition model for deep encoding.

[0046] Step S303, inputting the pulse sequence after the depth encoding into the classification layer in the visual information recognition model to obtain the recognition result of the visual information to be recognized.

[0047] In one possible implementation, the prior knowledge embedding layer is located in a deep encoding layer.

[0048] Figure 4 A schematic diagram of the principle of a visual information recognition model provided by the present invention.

[0049] Combine the following Figure 4 , steps S301-S303 are explained in detail.

[0050] Specifically, in step S301, the convolutional coding layer is composed of a two-dimensional convolutional layer, a normalization layer, a LIF spike neuron layer, and a maximum pooling layer. First, the two-dimensional convolutional layer encodes the input spike sequence; then, the normalization layer, the LIF spike neuron layer, and the maximum pooling layer pulse process the initially encoded spike sequence.

[0051] Specifically, in step S302, the deep coding layer is composed of a priori knowledge embedding layer, a normalization layer, a LIF pulse neuron layer, a residual layer, and a sparse connection layer.

[0052] Specifically, in step S303, the classification layer performs classification according to the pulse sequence information obtained after depth encoding to obtain a recognition result of the visual information.

[0053] In one possible implementation, for the modules of the spiking Transformer network, a LIF neuron layer is added at the output of each module to simulate Neuronal membrane potential accumulation and pulse firing, and the resulting pulse train is used as input to the next module.

[0054] In an embodiment of the present invention, the membrane potential of any LIF neuron in the visual information recognition model is updated based on the following formula: (X (V )

[0055] in, is the membrane potential time constant, for The input current at the moment, for The initial neuronal membrane potential at time is the resting potential, is the updated membrane potential, as The initial membrane potential at time is the membrane potential after the last update, as The initial membrane potential at time Function used to confirm the flag The value of .

[0056] when Greater than the release threshold When =1, confirm the output pulse release.

[0057] Dangdang Less than or equal to the release threshold When =0, output pulse is not issued.

[0058] In an embodiment of the present invention, the prior knowledge embedding layer replaces the self-attention module in the conventional pulse Transformer network.

[0059] In a possible implementation, one or more basis functions of Fourier basis functions or wavelet basis functions are used to construct a priori knowledge embedding layer, including: Sine functions and cosine functions of different frequencies in Fourier basis functions are used to construct a priori knowledge embedding matrix to obtain a priori knowledge embedding layer; or cosine functions of different frequencies in Fourier basis functions are used to construct a priori knowledge embedding matrix to obtain a priori knowledge embedding layer.

[0060] In the embodiment of the present invention, the Fourier basis function is global and orthogonal, and the advantage of the prior knowledge embedding layer constructed based on the Fourier basis function mainly lies in the extraction of global information.

[0061] In a possible implementation, one or more basis functions of Fourier basis functions or wavelet basis functions are used to construct a priori knowledge embedding layer, including: One or more discrete wavelet basis functions are used to construct a priori knowledge embedding matrix to obtain a priori knowledge embedding layer.

[0062] In the embodiment of the present invention, for the wavelet basis function, one or more discrete wavelet basis functions of Biorthogonal 1.1 Wavelet, HaarWavelet, and Daubechies 1 Wavelet are used as a combination to construct a priori knowledge embedding matrix to obtain a priori knowledge embedding layer. The above three discrete wavelet basis functions are all orthogonal and compactly supported, and the advantage of the wavelet basis function is the extraction of local information, which is more suitable for the processing of pulse signals.

[0063] In a possible implementation, one or more basis functions of Fourier basis functions or wavelet basis functions are used to construct a priori knowledge embedding layer, including: When multiple Fourier basis functions are used to construct a priori knowledge embedding layer, an initial weighting parameter is configured for each Fourier basis function, and the initial weighting parameter is continuously adjusted as the visual information recognition model is trained.

[0064] When multiple wavelet basis functions are used to construct a priori knowledge embedding layer, an initial weighting parameter is configured for each Fourier basis function, and the initial weighting parameter is continuously adjusted as the visual information recognition model is trained.

[0065] By adopting the above method, the present invention can achieve the following beneficial effects: constructing a visual information recognition model through a Transformer network based on a pulse framework, and introducing Fourier basis functions or wavelet basis functions to construct a priori knowledge embedding layer to replace the attention module of the pulse Transformer network. The accuracy of visual information recognition is improved, and the energy consumption of the visual information recognition process is reduced.

[0066] The following is a description of the visual information recognition device based on the pulse Transformer network provided by the present invention. The visual information recognition device based on the pulse Transformer network described below and the visual information recognition method based on the pulse Transformer network described above can be referenced to each other.

[0067] Figure 5 The schematic diagram of the structure of the visual information recognition device based on the pulse Transformer network provided in the embodiment of the present application includes: The visual information processing unit 510 is used to obtain a pulse sequence of visual information to be recognized.

[0068] The model construction unit 520 is used to construct a visual information recognition model based on a pulse Transformer network; wherein one or more basis functions of Fourier basis functions or wavelet basis functions are used to construct a priori knowledge embedding layer; and the priori knowledge embedding layer is used to replace the attention module in the pulse Transformer network.

[0069] The visual information recognition unit 530 is used to input the pulse sequence into the visual information recognition model to obtain the recognition result of the visual information to be recognized.

[0070] In a possible implementation, the visual information recognition unit 530 is specifically used to input the pulse sequence into the convolutional coding layer in the visual information recognition model for preliminary encoding; input the pulse sequence after preliminary encoding into the deep coding layer in the visual information recognition model for deep encoding; input the pulse sequence after deep encoding into the classification layer in the visual information recognition model to obtain the recognition result of the visual information to be recognized.

[0071] In one possible implementation, the prior knowledge embedding layer is located in a deep encoding layer.

[0072] In a possible implementation, the visual information processing unit 510 is specifically configured to, when the visual information to be identified is a static RGB image, input the static RGB image into the RGB encoding module to obtain a pulse sequence of the visual information to be identified.

[0073] In a possible implementation manner, the visual information processing unit 510 is specifically configured to, when the visual information to be identified is DVS data, directly use the DVS data as a pulse sequence of the visual information to be identified.

[0074] In one possible implementation, the model construction unit 520 is specifically used to use sine functions and cosine functions of different frequencies in the Fourier basis function to construct a prior knowledge embedding matrix to obtain a prior knowledge embedding layer; or to use cosine functions of different frequencies in the Fourier basis function to construct a prior knowledge embedding matrix to obtain a prior knowledge embedding layer.

[0075] In a possible implementation, the model building unit 520 is specifically configured to use one or more discrete wavelet basis functions to build a priori knowledge embedding matrix to obtain a priori knowledge embedding layer.

[0076] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communications interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a visual information recognition method based on a pulse Transformer network, comprising the following steps: Acquire a pulse sequence of visual information to be recognized; Based on the pulse Transformer network, a visual information recognition model is constructed; wherein one or more basis functions of Fourier basis functions or wavelet basis functions are used to construct a priori knowledge embedding layer; and the priori knowledge embedding layer is used to replace the attention module in the pulse Transformer network; The pulse sequence is input into the visual information recognition model to obtain the recognition result of the visual information to be recognized.

[0077] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0078] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute a visual information recognition method based on a pulse Transformer network provided by the above methods, including the following steps: Acquire a pulse sequence of visual information to be recognized; Based on the pulse Transformer network, a visual information recognition model is constructed; wherein one or more basis functions of Fourier basis functions or wavelet basis functions are used to construct a priori knowledge embedding layer; and the priori knowledge embedding layer is used to replace the attention module in the pulse Transformer network; The pulse sequence is input into the visual information recognition model to obtain the recognition result of the visual information to be recognized.

[0079] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform a visual information recognition method based on a pulse Transformer network provided by the above methods, comprising the following steps: Acquire a pulse sequence of visual information to be recognized; Based on the pulse Transformer network, a visual information recognition model is constructed; wherein one or more basis functions of Fourier basis functions or wavelet basis functions are used to construct a priori knowledge embedding layer; and the priori knowledge embedding layer is used to replace the attention module in the pulse Transformer network; The pulse sequence is input into the visual information recognition model to obtain the recognition result of the visual information to be recognized.

[0080] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0081] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual information recognition method based on a pulse Transformer network, characterized in that: include: Acquire a pulse sequence of visual information to be recognized; Based on the pulse Transformer network, a visual information recognition model is constructed; wherein one or more basis functions of Fourier basis functions or wavelet basis functions are used to construct a priori knowledge embedding layer; and the priori knowledge embedding layer is used to replace the attention module in the pulse Transformer network; The pulse sequence is input into the visual information recognition model to obtain the recognition result of the visual information to be recognized.

2. The visual information recognition method based on the pulse Transformer network according to claim 1 is characterized in that: The step of inputting the pulse sequence into the visual information recognition model to obtain a recognition result of the visual information to be recognized includes: Inputting the pulse sequence into the convolutional coding layer in the visual information recognition model for preliminary coding; Inputting the initially encoded pulse sequence into the deep encoding layer in the visual information recognition model for deep encoding; The pulse sequence after depth encoding is input into the classification layer in the visual information recognition model to obtain the recognition result of the visual information to be recognized.

3. The visual information recognition method based on the pulse Transformer network according to claim 2 is characterized in that: The prior knowledge embedding layer is located in the deep encoding layer.

4. The visual information recognition method based on the pulse Transformer network according to claim 1 is characterized in that: The pulse sequence of obtaining the visual information to be recognized includes: When the visual information to be identified is a static RGB image, the static RGB image is input into an RGB encoding module to obtain a pulse sequence of the visual information to be identified.

5. The visual information recognition method based on the pulse Transformer network according to claim 1 is characterized in that: The pulse sequence of obtaining the visual information to be recognized includes: When the visual information to be identified is DVS data, the DVS data is directly used as the pulse sequence of the visual information to be identified.

6. The method for visual information recognition based on a pulse Transformer network according to claim 1, characterized in that: The method of using one or more basis functions of Fourier basis functions or wavelet basis functions to construct a priori knowledge embedding layer includes: Using sine functions and cosine functions of different frequencies in Fourier basis functions to construct a priori knowledge embedding matrix to obtain the priori knowledge embedding layer; or Cosine functions of different frequencies in Fourier basis functions are used to construct a priori knowledge embedding matrix to obtain the priori knowledge embedding layer.

7. The visual information recognition method based on the pulse Transformer network according to claim 1 is characterized in that: The method of using one or more basis functions of Fourier basis functions or wavelet basis functions to construct a priori knowledge embedding layer includes: One or more discrete wavelet basis functions are used to construct a priori knowledge embedding matrix to obtain the priori knowledge embedding layer.

8. A visual information recognition device based on a pulse Transformer network, characterized in that: include: A visual information processing unit, used for acquiring a pulse sequence of visual information to be recognized; A model building unit, for building a visual information recognition model based on a pulse Transformer network; wherein one or more basis functions of Fourier basis functions or wavelet basis functions are used to build a priori knowledge embedding layer; and the priori knowledge embedding layer is used to replace the attention module in the pulse Transformer network; The visual information recognition unit is used to input the pulse sequence into the visual information recognition model to obtain the recognition result of the visual information to be recognized.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the visual information recognition method based on the pulse Transformer network as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for visual information recognition based on a pulse Transformer network as described in any one of claims 1 to 7 is implemented.

Images