Lightweight in-memory computing target recognition network method and system

Abstract

The invention discloses a lightweight in-memory computing target recognition network method and system, and the method comprises the steps: obtaining an infrared image data set of a moving target, and labeling an infrared image in the data set; constructing a Mangban efficient lightweight deep neural network, wherein the Mangban efficient lightweight deep neural network comprises a task identification module; taking the category and the position of the marked moving target as gold standards for calculation output of a task identification module, taking an infrared image data set as training data, inputting the training data into a Mangban efficient lightweight deep neural network for deep learning, and obtaining a Mangban efficient lightweight deep neural network model; converting a weight value of the trained Mangbar efficient lightweight deep neural network model into a conductance value of a memristor in a to-be-deployed memory computing chip; and designing a computing hardware circuit in the memristor memory, and deploying the trained Mangban efficient lightweight deep neural network model on the computing hardware circuit in the memristor memory. According to the invention, the demand on the number of memristors in the memory computing chip is reduced, and the computing speed is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent perception and image recognition technology, and specifically to a lightweight in-memory computing target recognition network method and system. Background Art

[0002] With the development of deep learning technology, the network structure is becoming more and more complex, and the number of parameters and floating-point operations of neural networks are increasing. For example, the number of parameters of the commonly used ResNet18 is 11.7M, the number of parameters of ResNet50 is 25M, and the number of parameters of ResNet101 is 44M. The network multiplication and addition calculations are even larger. The latest lightweight networks such as MobileNet V3 have an MAdds of 747.9M in the task of human key point detection, and MicroNet's MAdds is 77.7M.

[0003] The sensing, storage and computing all-in-one chip integrates sensing and artificial intelligence computing. The artificial intelligence algorithm is mainly deployed in the storage and computing array of the sensing, storage and computing all-in-one chip. The current storage and computing array is mainly composed of memristors, and one or more memristors are used to represent a neuron of the neural network algorithm. The uniformity and stability of the memristor array preparation process and its compatibility with CMOS manufacturing technology are technical factors that limit the scale of memristor arrays. Due to the limitations of the current processing technology, most of the memristor arrays are <10M in size. For example, the memristor array published by Wu Huaqiang's team at Tsinghua University in Nature in 2020 contains 2048 neurons, and the memristor array published by Miao Feng's team at Nanjing University in Nature Nanotechnology in 2021 is 32×32 in size. The memristor array published by Samsung Electronics in Nature in 2022 is 64×64 in size, and the memristor array published by the Korea Institute of Chemical Technology in Nature Communications is 320×320 in size.

[0004] The number of memristors contained in the current memory computing array is relatively small, which cannot meet the number of neurons required for the current mainstream neural network deployment. Therefore, it is necessary to develop an efficient and lightweight in-memory computing network.

[0005] In the prior art, the invention patent with the patent publication number CN116883807A discloses an integrated sensing, storage and computing integrated target detection and recognition system and method. The system includes: a lightweight target detection network structure module, which is used to design a lightweight target detection network. The lightweight target detection network structure module obtains non-redundant features and redundant features through convolution operations, and outputs the location category information of the target to be detected; a weight training module, which is used to obtain the resistance instability parameters of the memristor, to model the Gaussian noise model, and train the network neuron weights of the lightweight target detection network; a weight deployment module, which maps and deploys the network neuron weights to the memristor array, thereby obtaining a suitable lightweight target detection network, so as to obtain the applicable location category information of the target to be detected, so as to identify and obtain the integrated sensing, storage and computing integrated target. The technical problems of low recognition accuracy caused by network weight disturbance and high power consumption, large size and low integration degree of the device caused by high number of memristors are solved. The patent uses lightweight convolution to reduce the number of neurons.

[0006] Traditional image segmentation is usually achieved using deep learning networks represented by convolution and Transformer architectures. Convolution has excellent local feature extraction capabilities, but it is insufficient in establishing the correlation of remote information. Although the self-attention mechanism can solve the problem of extracting remote information using continuous patch sequences, it also brings a large computational load. This is not suitable for actual application scenarios, especially mobile detection equipment. Due to the limitation of computing resources, the computationally loaded model is not suitable for real application scenarios.

[0007] In recent years, state-space models (SSMs), represented by the Mamba network, have become a strong competitor to traditional convolutional neural networks and Transformer architectures. State-space models (SSMs) show linear complexity in input size and memory usage, which makes them key to the foundation of lightweight models. Although the Mamba network has a small computational footprint, the spatial scanning leads to a lack of spatial correlation between feature sub-blocks and low perception of spatial structural information. Summary of the invention

[0008] The technical problem to be solved by the present invention is to solve the problem that conventional neural networks cannot be deployed in in-memory computing chips due to the large number of parameters and complex calculations, while the Mamba network has a small amount of calculation but the spatial scanning leads to a lack of spatial correlation between feature sub-blocks.

[0009] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0010] A lightweight in-memory computing target recognition network method, comprising:

[0011] Obtain infrared image datasets of moving targets and label the infrared images in the dataset;

[0012] Construct a Mamba efficient and lightweight deep neural network, including an initial feature extraction module, a feature slicing block operation module, a lightweight Mamba calculation module and a task identification module; the feature slicing block operation module performs multiple feature map slicing operations on the feature map output by the initial feature extraction module, and at least one lightweight Mamba calculation module is connected after each feature map slicing operation, and the output result of the last lightweight Mamba calculation module is used as the input of the task identification module;

[0013] The marked moving target category and position are used as the gold standard for the calculation output of the task recognition module, and the infrared image dataset is used as the training data to input the Mamba efficient and lightweight deep neural network for deep learning to obtain the Mamba efficient and lightweight deep neural network model;

[0014] Convert the weight values ​​of the trained Mamba efficient and lightweight deep neural network model into the conductance values ​​of the memristors in the in-memory computing chip to be deployed;

[0015] Design a memristor in-memory computing hardware circuit, and deploy the trained Mamba efficient and lightweight deep neural network model on the memristor in-memory computing hardware circuit.

[0016] In one embodiment of the present invention, the initial feature extraction module is constructed in a lightweight manner, using only conventional convolution, and the convolution kernel size k×k of the conventional convolution should be no greater than and close to the number of rows of the memristor array.

[0017] In one embodiment of the present invention, the feature segmentation block operation module is composed of a pooling operation with a size of s×s and a step size of s; for a feature map with an input size of C×H×W, the size after the segmentation operation is Among them, C represents the number of channels, H and W represent the height and width respectively, and the feature dimension of each small block feature map after the splitting operation is C.

[0018] In one embodiment of the present invention, the lightweight Mamba calculation module includes a random partial convolution feature extraction unit; the calculation process of the random partial convolution feature extraction unit is:

[0019] Assume the input feature map is X∈R c×h×w , where c is the number of channels, which is also the feature dimension of the feature map, h and w represent the height and width respectively, and R is a real number; the output feature map after convolution of the random partial convolution feature extraction unit is Y∈R c′×h×w , where c ′ is the number of channels after convolution, then the conventional convolution kernel that completes this convolution operation can be expressed as f∈R c ×k×k×c′ ;

[0020] The random partial convolution feature extraction unit only performs the input feature map X∈R c×h×w A small part of the length n features in the dimension c of the convolution operation is used to obtain the feature map X ′ ∈R n×h×w , and the feature map X∈R c×h×w Remove the remaining features of n from the dimension c and use X ori Indicates that X ori ∈R (c-n)×h×w No processing is done, and the final output feature is X ′ and X ori The superimposed characteristic map is Y∈R c′×h×w , whose length is n, the partial convolution kernel f partial ∈R n×k×k×c′ .

[0021] In one embodiment of the present invention, the portion to be convolved with a length of n selected on the dimension C is randomly intercepted along the dimension C, and each time a random partial convolution feature extraction unit is added, the position of n on the dimension C is randomly set. When the output of the feature slice block operation module is used as the input of the random partial convolution feature extraction unit, the sizes c, h, and w of the feature map output by the feature slice block operation module are as follows:

[0022] In one embodiment of the present invention, the lightweight Mamba calculation module includes a spatial feature rearrangement unit and a state space model SSM unit; the implementation process of the spatial feature rearrangement unit and the state space model SSM unit includes:

[0023] The output feature map after calculating the random part convolution feature extraction unit is Y∈R c′×h×w , merge the height h and width w dimensions to obtain the feature map Y after the dimension is merged ′ ∈R c′×(h×w) , each point in the (h×w) dimension corresponds to each small feature map after the segmentation operation, and the dimension c ′ Corresponding to the features of each small feature map, M groups are set on the dimension (h×w), and the length of the M groups on the dimension (h×w) is c using channel rearrangement ′ of The feature vectors are rearranged between groups to complete the spatial feature rearrangement operation. The small-block feature maps after spatial rearrangement are sent to the Mamba state space model SSM module to calculate the spatial correlation between the small-block feature maps.

[0024] In one embodiment of the present invention, the convolution kernel size k×k used by the task identification module should be no greater than and close to the number of rows of the memristor array.

[0025] In one embodiment of the present invention, the memristor in-memory computing hardware circuit includes a random partial convolution in-memory computing hardware circuit implementation module;

[0026] The random partial convolution in-memory calculation hardware circuit implementation module includes an addressing shift input circuit, a convolution calculation circuit and an output storage circuit;

[0027] The addressing shift input circuit is used to obtain the characteristic spectrum X∈R c×h×w Take out some features X that need to be convolutional ′ ∈R n×h×w , the characteristic spectrum X∈R during the calculation process c×h×w Stored in register, addressing shift input circuit serially reads X ′ Then, the data is output in parallel to the convolution calculation circuit. The number of bits of serial input data each time is k×k. The addressing shift input circuit includes a clock signal and k×k flip-flops, where the value of k×k is consistent with the convolution kernel size k×k used by the task identification module; and the clock signal is connected to the CLK terminal of the flip-flop.

[0028] The convolution calculation circuit is composed of a memristor array, and the i-th convolution kernel of size k×k is mapped to the conductance value of the memristor at the corresponding position in the i-th column of the memristor array;

[0029] The output storage circuit is used to convert the parallel output results of the convolution calculation circuit into serial output and store the results in the register. The output storage circuit is composed of triggers and clock signals, and all triggers share the clock signal. Among them, the number of triggers in the output storage circuit is proportional to the number of channels after convolution c ′ Consistent.

[0030] In one embodiment of the present invention, the memristor in-memory computing hardware circuit includes a spatial feature rearrangement in-memory computing hardware circuit implementation module;

[0031] The spatial feature rearrangement in-memory computing hardware circuit implementation module is used to generate the feature map Y after the dimension is merged ′ ∈R c′×(h×w) The random permutation number of the upper dimension (h×w);

[0032] The circuit of the spatial feature rearrangement in-memory calculation hardware circuit implementation module is composed of an embedded MCU. After generating the random arrangement sequence of (h×w) vectors, it will read the feature spectrum Y from the register in the order of the new sequence number. ′ ∈R c′×(h×w) .

[0033] The present invention also provides a lightweight in-memory computing target recognition network system, which applies the lightweight in-memory computing target recognition network method described above, including:

[0034] The data module is used to obtain infrared image datasets of moving targets and label the infrared images in the datasets;

[0035] A construction module is used to construct a Mamba efficient and lightweight deep neural network, including an initial feature extraction module, a feature slicing block operation module, a lightweight Mamba calculation module and a task identification module; the feature slicing block operation module performs multiple feature map slicing operations on the feature map output by the initial feature extraction module, and at least one lightweight Mamba calculation module is connected after each feature map slicing operation, and the output result of the last lightweight Mamba calculation module is used as the input of the task identification module;

[0036] A model module is used to use the marked moving target category and position as the gold standard for the task recognition module to calculate the output, and use the infrared image data set as training data to input the Mamba efficient and lightweight deep neural network for deep learning, so as to obtain the Mamba efficient and lightweight deep neural network model;

[0037] A conversion module, used to convert the weight value of the trained Mamba efficient and lightweight deep neural network model into the conductance value of the memristor in the in-memory computing chip to be deployed;

[0038] The deployment module is used to design the memristor in-memory computing hardware circuit and deploy the trained Mamba efficient and lightweight deep neural network model on the memristor in-memory computing hardware circuit.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention can greatly reduce the number of parameters and the amount of calculation of the in-memory computing neural network while maintaining high recognition accuracy, reduce the demand for the number of memristors in the in-memory computing chip, and improve the computing speed.

[0041] The spatial feature rearrangement unit is used to solve the problem of spatial correlation information loss between feature blocks caused by the traditional Mamba algorithm when the feature map is cut and arranged sequentially, thereby improving the ability to perceive spatial structure information.

[0042] The present invention designs a more lightweight neural network architecture to adapt to the problem that the number of memristors in the integrated sensing, storage and computing chip is relatively small and cannot meet the number of neurons required for mainstream neural network deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The present invention is a flowchart of a lightweight in-memory computing target recognition network method according to an embodiment of the present invention.

[0044] Figure 2 This is a diagram of the Mamba efficient and lightweight deep neural network architecture of an embodiment of the present invention.

[0045] Figure 3 Schematic diagram of the block cutting operation of the feature block cutting operation module in an embodiment of the present invention.

[0046] Figure 4 Schematic diagram of the calculation process of the random partial convolution feature extraction unit according to an embodiment of the present invention.

[0047] Figure 5 Schematic diagram of the rearrangement operation of the spatial feature rearrangement unit according to an embodiment of the present invention.

[0048] Figure 6 Schematic diagram of an addressing shift input circuit according to an embodiment of the present invention.

[0049] Figure 7 Schematic diagram of a convolution calculation circuit according to an embodiment of the present invention.

[0050] Figure 8 The present invention is a schematic circuit diagram of an output storage circuit and a spatial feature rearrangement in-memory computing hardware circuit implementation module.

[0051] Fig. 9 A block diagram of a lightweight in-memory computing target recognition network system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to facilitate those skilled in the art to understand the technical solution of the present invention, the technical solution of the present invention is further described in conjunction with the accompanying drawings of the specification.

[0053] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0054] See also Figure 1 As shown, the present invention provides a lightweight in-memory computing target recognition network method, comprising:

[0055] S10, obtaining an infrared image dataset of a moving target, and labeling the infrared images in the dataset.

[0056] In one embodiment of the present invention, an infrared target recognition environment scene is constructed, and an infrared camera is used to capture moving targets in the scene. The collection environment includes indoor and outdoor scenes, scenes with different light intensities, shooting angles, far and near scales, day and night, and the collected data is cleaned and other pre-processing operations are performed.

[0057] In this embodiment, the category of the moving target in each image in the preprocessed infrared image data set, the maximum circumscribed rectangular box and the pixel-by-pixel category are labeled to obtain the category of the moving target in each image, the position box of the target detection and the pixel-level label of the target segmentation.

[0058] S20, constructing a Mamba efficient and lightweight deep neural network, including an initial feature extraction module, a feature slicing block operation module, a lightweight Mamba computing module and a task identification module. The feature slicing block operation module performs multiple feature map slicing operations on the feature map output by the initial feature extraction module, and at least one lightweight Mamba computing module is connected after each feature map slicing operation, and the output result of the last lightweight Mamba computing module is used as the input of the task identification module.

[0059] See also Figures 1 to 3 As shown, in one embodiment of the present invention, the initial feature extraction module is used to extract the initial features of the input image. The initial feature extraction module is constructed in a lightweight manner, using only conventional convolution, and the conventional convolution kernel size k×k should be no greater than and close to the number of rows of the memristor array. Conventional convolution is used to generate a small number of non-redundant features, thereby maximizing the utilization rate of the memristor array, and the remaining redundant features are obtained by simple linear changes of the non-redundant features.

[0060] In this embodiment, the feature slice block operation module is used to slice the input feature map into small blocks as the input of the lightweight Mamba calculation module. The feature slice block operation module is composed of pooling operations with a size of s×s and a step size of s. Usually, s is 2. For a feature map with an input size of C×H×W, the size after the slice operation is Among them, C represents the number of channels, H and W represent the height and width respectively, and the feature dimension of each small block feature map after the splitting operation is C.

[0061] In this embodiment, the lightweight Mamba calculation module includes a random partial convolution feature extraction unit, a spatial feature rearrangement unit and a state space model SSM unit.

[0062] See also Figures 1 to 4 As shown, the calculation process of the random partial convolution feature extraction unit is:

[0063] Assume the input feature map is X∈R c×h×w , where c is the number of channels, which is also the feature dimension of the feature map, h and w represent the height and width respectively, and R is a real number. The output feature map after convolution of the random partial convolution feature extraction unit is Y∈R c′×h×w , where c ′ is the number of channels after convolution, then the conventional convolution kernel that completes this convolution operation can be expressed as f∈R c×k×k×c′ .

[0064] The random partial convolution feature extraction unit only performs the input feature map X∈R c×h×w A small part of the length n features in the dimension c of the convolution operation is used to obtain the feature map X ′ ∈R n×h×w , and the feature map X∈R c×h×w Remove the remaining features of n from the dimension c and use X ori Indicates that X ori ∈R (c-n)×h×w No processing is done, and the final output feature is X ′ and X ori The superimposed characteristic map is Y∈R c′×h×w , whose length is n, the partial convolution kernel f partial ∈R n×k×k×c′ .

[0065] The random operation of the random partial convolution feature extraction unit is reflected in that the length n of the part to be convolved selected on the dimension C is randomly intercepted along the dimension C for the first time.

[0066] The length n of the convolution part selected on the dimension C is randomly intercepted along the dimension C for the first time, and each time a random partial convolution feature extraction unit is added, the position of n on the dimension C is randomly set. When the output of the feature slice block operation module is used as the input of the random partial convolution feature extraction unit, the size c, h, and w of the feature map output by the feature slice block operation module are as follows:

[0067] See also Figures 1 to 5 As shown, in this embodiment, the spatial feature rearrangement unit solves the problem that the spatial correlation information between the feature blocks is lost due to the sequential arrangement after the feature map is cut into blocks in the traditional Mamba. The specific implementation process is as follows:

[0068] The output feature map after calculating the random part convolution feature extraction unit is Y∈R c′×h×w , merge the height h and width w dimensions to obtain the feature map Y after the dimension is merged ′ ∈R c′×(h×w) , each point in the (h×w) dimension corresponds to each small feature map after the segmentation operation, and the dimension c ′ Corresponding to the features of each small feature map, M groups are set on the dimension (h×w), and channel shuffle is used to reorder the length of c in the M groups on the dimension (h×w). ′ of The feature vectors are rearranged between groups to complete the spatial feature rearrangement operation. The small feature maps after spatial rearrangement are sent to the Mamba state space model SSM module to calculate the spatial correlation between the small feature maps. Since each channel rearrangement randomly re-shoots the arrangement order of the small feature maps after segmentation, the SSM operation can extract the spatial correlation between the feature sub-blocks and improve the perception of spatial structure information.

[0069] S30, using the marked moving target category and position as the gold standard for the task recognition module to calculate the output, and using the infrared image data set as training data to input the Mamba efficient and lightweight deep neural network for deep learning, to obtain the Mamba efficient and lightweight deep neural network model.

[0070] In one embodiment of the present invention, the convolution kernel size k×k used by the task identification module should be no greater than and close to the number of rows of the memristor array, so as to maximize the utilization rate of the memristor array.

[0071] S40, converting the weight value of the trained Mamba efficient and lightweight deep neural network model into the conductance value of the memristor in the in-memory computing chip to be deployed.

[0072] S50, designs the memristor in-memory computing hardware circuit, and deploys the trained Mamba efficient and lightweight deep neural network model on the memristor in-memory computing hardware circuit.

[0073] See also Figures 1 to 8 As shown, in one embodiment of the present invention, the memristor in-memory computing hardware circuit includes a random partial convolution in-memory computing hardware circuit implementation module and a spatial feature rearrangement in-memory computing hardware circuit implementation module.

[0074] In this embodiment, the random partial convolution in-memory calculation hardware circuit implementation module includes an addressing shift input circuit, a convolution calculation circuit and an output storage circuit.

[0075] The addressing shift input circuit is used to obtain the characteristic spectrum X∈R c×h×w Take out some features X that need to be convolutional ′ ∈R n×h×w , the characteristic spectrum X∈R during the calculation process c×h×w Stored in register, addressing shift input circuit serially reads X ′ Then it is output in parallel to the convolution calculation circuit. The number of bits of serial input data each time is k×k. The addressing shift input circuit includes a clock signal and k×k triggers, where the value of k×k is consistent with the convolution kernel size k×k used by the task identification module; and the clock signal is connected to the CLK terminal of the trigger.

[0076] The convolution calculation circuit is composed of a memristor array, and the i-th convolution kernel of size k×k is mapped to the conductance value of the memristor at the corresponding position in the i-th column of the memristor array.

[0077] The output storage circuit is used to convert the parallel output results of the convolution calculation circuit into serial output and store the results in the register. The output storage circuit is composed of triggers and clock signals, and all triggers share the clock signal. Among them, the number of triggers in the output storage circuit is proportional to the number of channels after convolution c ′ Consistent.

[0078] The spatial feature rearrangement in-memory computing hardware circuit implementation module is used to generate the feature map Y after the dimension is merged ′ ∈R c′×(h×w) The circuit of the spatial feature rearrangement memory calculation hardware circuit implementation module is composed of an embedded MCU. After generating the random permutation sequence of (h×w) vectors, it will read the feature map Y from the register in the order of the new sequence number. ′ ∈R c′×(h×w) .

[0079] See also Figures 1 to 9 As shown, the present invention also provides a lightweight in-memory computing target recognition network system, which applies the above-mentioned lightweight in-memory computing target recognition network method, including:

[0080] The data module 10 is used to obtain an infrared image data set of a moving target and label the infrared images in the data set.

[0081] The construction module 20 is used to construct a Mamba efficient and lightweight deep neural network, including an initial feature extraction module, a feature slicing block operation module, a lightweight Mamba calculation module and a task identification module; the feature slicing block operation module performs multiple feature map slicing operations on the feature map output by the initial feature extraction module, and at least one lightweight Mamba calculation module is connected after each feature map slicing operation, and the output result of the last lightweight Mamba calculation module is used as the input of the task identification module.

[0082] The model module 30 is used to use the marked moving target category and position as the gold standard for the task recognition module to calculate the output, and use the infrared image data set as training data to input the Mamba efficient and lightweight deep neural network for deep learning, so as to obtain the Mamba efficient and lightweight deep neural network model.

[0083] The conversion module 40 is used to convert the weight value of the trained Mamba efficient and lightweight deep neural network model into the conductance value of the memristor in the in-memory computing chip to be deployed.

[0084] The deployment module 50 is used to design the memristor in-memory computing hardware circuit and deploy the trained Mamba efficient and lightweight deep neural network model on the memristor in-memory computing hardware circuit.

[0085] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting from any point of view, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0086] The embodiments described above only represent implementation methods of the invention. The protection scope of the present invention is not limited to the embodiments described above. For those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A lightweight in-memory computing target recognition network method, characterized in that: include: Obtain infrared image datasets of moving targets and label the infrared images in the dataset; Construct a Mamba efficient and lightweight deep neural network, including an initial feature extraction module, a feature slicing block operation module, a lightweight Mamba calculation module and a task identification module; the feature slicing block operation module performs multiple feature map slicing operations on the feature map output by the initial feature extraction module, and at least one lightweight Mamba calculation module is connected after each feature map slicing operation, and the output result of the last lightweight Mamba calculation module is used as the input of the task identification module; The marked moving target category and position are used as the gold standard for the calculation output of the task recognition module, and the infrared image dataset is used as the training data to input the Mamba efficient and lightweight deep neural network for deep learning to obtain the Mamba efficient and lightweight deep neural network model; Convert the weight values ​​of the trained Mamba efficient and lightweight deep neural network model into the conductance values ​​of the memristors in the in-memory computing chip to be deployed; Design a memristor in-memory computing hardware circuit, and deploy the trained Mamba efficient and lightweight deep neural network model on the memristor in-memory computing hardware circuit.

2. The lightweight in-memory computing target recognition network method according to claim 1, characterized in that: The initial feature extraction module is constructed in a lightweight way, using only conventional convolution, and the convolution kernel size k×k of the conventional convolution should be no greater than and close to the number of rows of the memristor array.

3. The lightweight in-memory computing target recognition network method according to claim 1, characterized in that: The feature slice block operation module is composed of a pooling operation with a size of s×s and a step size of s; for a feature map with an input size of C×H×W, the size after the slice operation is Among them, C represents the number of channels, H and W represent the height and width respectively, and the feature dimension of each small block feature map after the splitting operation is C.

4. The lightweight in-memory computing target recognition network method according to claim 1, characterized in that: The lightweight Mamba calculation module includes a random partial convolution feature extraction unit; the calculation process of the random partial convolution feature extraction unit is: Assume the input feature map is X∈R c×h×w , where c is the number of channels, which is also the feature dimension of the feature map, h and w represent the height and width respectively, and R is a real number; the output feature map after convolution of the random partial convolution feature extraction unit is Y∈R c ′×h×w , where c ′ is the number of channels after convolution, then the conventional convolution kernel that completes this convolution operation can be expressed as f∈R c ×k×k×c′ ; The random partial convolution feature extraction unit only performs the input feature map X∈R c×h×w A small part of the length n features in the dimension c of the convolution operation is used to obtain the feature map X ′ ∈R n×h×w , and the feature map X∈R c×h×w Remove the remaining features of n from the dimension c and use X ori Indicates that X ori ∈R (c-n)×h×w No processing is done, and the final output feature is X ′ and X ori The superimposed characteristic map is Y∈R c′×h×w , whose length is n, the partial convolution kernel f partial ∈R n×k×k×c′ .

5. The lightweight in-memory computing target recognition network method according to claim 4 is characterized in that: The length of the convolution part selected on the dimension C is first set to be randomly intercepted along the dimension C, and each time a random partial convolution feature extraction unit is added, the position of n on the dimension C is randomly set. When the output of the feature slice block operation module is used as the input of the random partial convolution feature extraction unit, the size c, h, and w of the feature map output by the feature slice block operation module are as follows: c = C, 6. The lightweight in-memory computing target recognition network method according to claim 4, characterized in that: The lightweight Mamba computing module includes a spatial feature rearrangement unit and a state space model SSM unit; The implementation process of the spatial feature rearrangement unit and the state space model SSM unit includes: The output feature map after calculating the random part convolution feature extraction unit is Y∈R c′×h×w , merge the height h and width w dimensions to obtain the feature map Y after the dimension is merged ′ ∈R c′×(h×w) , each point in the (h×w) dimension corresponds to each small feature map after the segmentation operation, and the dimension c ′ Corresponding to the features of each small feature map, M groups are set on the dimension (h×w), and the length of the M groups on the dimension (h×w) is c using channel rearrangement ′ of The feature vectors are rearranged between groups to complete the spatial feature rearrangement operation. The small-block feature maps after spatial rearrangement are sent to the Mamba state space model SSM module to calculate the spatial correlation between the small-block feature maps.

7. The lightweight in-memory computing target recognition network method according to claim 1, characterized in that: The convolution kernel size k×k used by the task identification module should be no greater than and close to the number of rows of the memristor array.

8. The lightweight in-memory computing target recognition network method according to claim 4, characterized in that: The memristor in-memory computing hardware circuit includes a random partial convolution in-memory computing hardware circuit implementation module; The random partial convolution in-memory calculation hardware circuit implementation module includes an addressing shift input circuit, a convolution calculation circuit and an output storage circuit; The addressing shift input circuit is used to obtain the characteristic spectrum X∈R c×h×w Take out some features X that need to be convolutional ′ ∈R n×h×w , the characteristic spectrum X∈R during the calculation process c×h×w Stored in register, addressing shift input circuit serially reads X ′ Then, the data is output in parallel to the convolution calculation circuit. The number of bits of serial input data each time is k×k. The addressing shift input circuit includes a clock signal and k×k flip-flops, where the value of k×k is consistent with the convolution kernel size k×k used by the task identification module; and the clock signal is connected to the CLK terminal of the flip-flop. The convolution calculation circuit is composed of a memristor array, and the i-th convolution kernel of size k×k is mapped to the conductance value of the memristor at the corresponding position in the i-th column of the memristor array; The output storage circuit is used to convert the parallel output results of the convolution calculation circuit into serial output and store the results in the register. The output storage circuit is composed of triggers and clock signals, and all triggers share the clock signal. Among them, the number of triggers in the output storage circuit is proportional to the number of channels after convolution c ′ Consistent.

9. The lightweight in-memory computing target recognition network method according to claim 6, characterized in that: The memristor in-memory computing hardware circuit includes a spatial feature rearrangement in-memory computing hardware circuit implementation module; The spatial feature rearrangement in-memory computing hardware circuit implementation module is used to generate the feature map Y after the dimension is merged ′ ∈R c ′×(h×w) The random permutation number of the upper dimension (h×w); The circuit of the spatial feature rearrangement in-memory calculation hardware circuit implementation module is composed of an embedded MCU. After generating the random arrangement sequence of (h×w) vectors, it will read the feature spectrum Y from the register in the order of the new sequence number. ′ ∈r c′×(h×w) .

10. A lightweight in-memory computing target recognition network system, characterized in that: The lightweight in-memory computing target recognition network method according to any one of claims 1 to 9 is applied, comprising: The data module is used to obtain infrared image datasets of moving targets and label the infrared images in the datasets; A construction module is used to construct a Mamba efficient and lightweight deep neural network, including an initial feature extraction module, a feature slicing block operation module, a lightweight Mamba calculation module and a task identification module; the feature slicing block operation module performs multiple feature map slicing operations on the feature map output by the initial feature extraction module, and at least one lightweight Mamba calculation module is connected after each feature map slicing operation, and the output result of the last lightweight Mamba calculation module is used as the input of the task identification module; A model module is used to use the marked moving target category and position as the gold standard for the task recognition module to calculate the output, and use the infrared image data set as training data to input the Mamba efficient and lightweight deep neural network for deep learning, so as to obtain the Mamba efficient and lightweight deep neural network model; A conversion module, used to convert the weight value of the trained Mamba efficient and lightweight deep neural network model into the conductance value of the memristor in the in-memory computing chip to be deployed; The deployment module is used to design the memristor in-memory computing hardware circuit and deploy the trained Mamba efficient and lightweight deep neural network model on the memristor in-memory computing hardware circuit.

Images