An image classification method and system based on a memory-computing integrated chip

Abstract

The application discloses an image classification method and system based on a memory-computing integrated chip, and belongs to the technical field of memory-computing hardware circuits. The method is executed by a master control unit: a clock and a memory are configured, and a serial peripheral interface with a coprocessor is initialized; a memory-computing integrated module is initialized, a memory-computing array calculation mode is set, and a data link is configured; an initialization instruction is sent to the coprocessor; a real-time operating system is started and a driver is loaded; model data is loaded to an external memory, weight data is loaded to a weight cache area and is rearranged to determine an offset; image data to be classified is received and is preprocessed; in response to a preprocessing completion signal, quantized data groups are packed, are sent to the coprocessor, and after a calculation starting instruction is sent, sleep is entered; in response to an inference completion signal, inference results are read after being woken up, and image classification results are obtained after post-processing. The application solves the problems that the existing memory-computing integrated chip has high power consumption, cannot process floating point operations, and has a high development threshold in image classification applications.

Description

Technical Field

[0001] This invention relates to an image classification method and system based on a memory computing chip, belonging to the field of memory computing hardware circuit technology. Background Technology

[0002] The rapid development of artificial intelligence technology has driven the widespread application of deep neural networks in image classification. Deep neural networks typically include various network structures such as convolutional layers, pooling layers, and fully connected layers. They are computationally intensive and have many parameters, placing high demands on computing resources and storage bandwidth. Traditional von Neumann computing architectures separate storage and computation, requiring frequent data transfer between storage and computation units when processing deep neural network tasks, leading to increasingly prominent problems of low computational efficiency and high energy consumption. The emergence of in-memory computing chips provides a new technological path to solve these problems. By integrating storage and computation into a single unit, each in-memory computing unit can both store data and perform multiplication and accumulation operations, thereby significantly reducing the frequent data transfer between storage and computation units, effectively improving computational efficiency and reducing power consumption.

[0003] However, existing deep neural network accelerators based on in-memory computing chips still have the following drawbacks in image classification applications: First, the in-memory arrays in in-memory computing chips typically only support integer multiplication and accumulation operations, and cannot directly perform floating-point operations. However, quantization, dequantization, linear rectified activation function calculations, and pooling operations in deep neural networks all involve floating-point operations, making these operations difficult to implement efficiently in the in-memory array and often requiring additional processing or external assistance. Second, in existing in-memory computing chip acceleration solutions, the main control unit needs to participate in data scheduling, instruction sending, and result collection throughout the deep neural network inference process. It cannot enter a sleep state during the in-memory array's computation, resulting in the main control unit being in a working state for extended periods, leading to high overall system power consumption. Third, development based on in-memory computing chips typically requires developers to have in-depth knowledge of the in-memory array's hardware specifications, instruction set, and memory layout, resulting in high learning costs and long development cycles, limiting the widespread adoption of in-memory computing chips in image classification applications. Therefore, existing in-memory computing chips suffer from high power consumption, inability to handle floating-point operations, and high development barriers in image classification applications. Summary of the Invention

[0004] The purpose of this invention is to provide an image classification method and system based on a memory computing chip. The method and system are based on a three-level heterogeneous architecture consisting of a main control unit, a coprocessor, and a memory computing chip. The main control unit is responsible for task scheduling and sleep / wake-up, the coprocessor is responsible for auxiliary calculation and data scheduling, and the memory computing chip is responsible for multiplication and accumulation operations. This invention aims to solve the problems of high power consumption, inability to handle floating-point operations, and high development threshold of existing memory computing chips in image classification applications.

[0005] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution:

[0006] In a first aspect, the present invention provides an image classification method based on a memory computing chip, executed by a main control unit, comprising:

[0007] Configure the real-time operating system clock and memory, and initialize the serial peripheral interface communication protocol parameters with the coprocessor;

[0008] Perform initialization on the in-memory computing module, set the computing mode of the in-memory computing array, and configure the data link;

[0009] Send an initialization command to the coprocessor to configure the clock source, initialize the local data cache, and set the first-in-first-out cache strategy.

[0010] Start the real-time operating system and load peripheral drivers;

[0011] Load the model data of the deep neural network into an external dynamic random access memory;

[0012] The weight data of the deep neural network is loaded into the weight cache of the in-memory computing module and rearranged according to the storage layout and capacity of the in-memory computing array to determine the offset for each weight update.

[0013] Receive image data to be classified and perform preprocessing;

[0014] Send a quantization instruction to the coprocessor, causing the coprocessor to generate quantized data based on the preprocessed image data and return a preprocessing completion signal;

[0015] In response to the preprocessing completion signal sent by the coprocessor, the quantized data is grouped and packaged according to the structure of the deep neural network and the number of columns of the storage array to generate packaged data;

[0016] After sending the packaged data to the coprocessor and sending a start computation instruction including a first value fetching instruction for convolution operations and a second value fetching instruction for fully connected operations, it enters a sleep state.

[0017] In response to an inference completion signal sent by the coprocessor, the system is awakened from a sleep state and reads the inference results from a specified memory address, the inference results including convolution operation results and fully connected operation results;

[0018] Post-processing operations are performed on the inference results to obtain the image classification results.

[0019] Furthermore, the weight data of the deep neural network is loaded into the weight cache of the in-memory computing module and rearranged according to the storage layout and capacity of the in-memory array to determine the offset for each weight update, including:

[0020] The weight data of the deep neural network is sent to the weight cache of the in-memory computing module;

[0021] The weight data is arranged in the weight cache according to the storage layout determined by the number of rows and columns of the in-memory array, and the capacity determined by the total amount of weight data that the in-memory array can store.

[0022] The offset for each weight update is determined based on the number of columns in the storage array. The offset is the number of weight entries that the weight cache read pointer moves backward each time.

[0023] Furthermore, the quantized data is grouped and packaged according to the structure of the deep neural network and the number of columns in the storage array to generate packaged data, including:

[0024] Based on the structure of a deep neural network, the connection order of each layer is determined as the processing order of quantized data.

[0025] Based on the number of columns in the storage array and the processing order of the quantized data, the quantized data is divided into multiple data blocks;

[0026] Each data block is encapsulated into a data packet containing a header and a payload, generating packaged data.

[0027] Secondly, this invention provides an image classification method based on a memory computing chip, executed by a coprocessor, comprising:

[0028] In response to the initialization command sent by the main control unit, configure the clock source, initialize the local data buffer, and set the first-in-first-out buffer strategy;

[0029] In response to the quantization command sent by the main control unit, quantized data is generated based on the preprocessed image data, and a preprocessing completion signal is returned;

[0030] In response to the start calculation command sent by the main control unit, the system sends a first value retrieval command for convolution operation and a second value retrieval command for fully connected operation to the in-memory computing module, and performs collaborative calculation with the in-memory computing module. The collaborative calculation includes performing auxiliary calculation tasks and monitoring the status signals returned by the in-memory computing module.

[0031] After the collaborative computation is completed, the inference result is stored in a specified memory address, and an inference completion signal is sent to the main control unit. The inference result includes the convolution operation result and the fully connected operation result.

[0032] Furthermore, the auxiliary computing tasks include: quantization operation, dequantization operation, data transformation operation, linear rectification operation, pooling operation, and convolutional layer filling operation;

[0033] The quantization operation includes performing linear uniform quantization on floating-point data to convert floating-point values ​​into integer values;

[0034] The dequantization operation includes performing linear uniform dequantization on integer values ​​to convert integer values ​​into floating-point values;

[0035] The data transformation operations include the reshape operation, which changes the data dimension but not the total number of data, and the flatten operation, which flattens multidimensional data into a one-dimensional vector.

[0036] The linear rectification operation includes sequentially reading each value in the local data cache, replacing the value with 0 if the value is less than 0, otherwise keeping the value unchanged.

[0037] The pooling operations include max pooling, which takes the maximum value within the pooling window, and average pooling, which takes the arithmetic mean within the pooling window.

[0038] The convolutional layer padding operation includes adding zero values ​​within a preset boundary range of the input data.

[0039] Furthermore, if no status signal is received from the in-memory computing module within a preset time, an abnormal status signal is sent to the main control unit.

[0040] Thirdly, this invention provides an image classification method based on an in-memory computing chip, executed by an in-memory computing module, comprising:

[0041] In response to the initialization command sent by the main control unit, the computing mode of the in-memory array is set and the data link is configured;

[0042] Obtain the offset for each weight update determined by the main control unit;

[0043] In response to the first fetch instruction sent by the coprocessor, the following convolution operation steps are performed:

[0044] Repeat the following steps until there is no uncomputed weight data in the weight buffer:

[0045] In the coprocessor's local data cache, data is read sequentially starting from the first data address. After each data is read, the address is increased by the step size, and the multiplication and accumulation operations are performed simultaneously by each storage unit in the storage array until the read address reaches the end address, thus obtaining the first intermediate result corresponding to the weight data stored in the current storage array.

[0046] Determine if there is any uncomputed weight data in the weight cache;

[0047] If there are weight data that have not been included in the calculation, then the next set of weight data is read from the weight cache, and the weight data stored in the storage array is updated with the next set of weight data. The starting address of the next set of weight data is shifted backward by the offset relative to the starting address of the weight data stored in the current storage array.

[0048] The first intermediate result obtained from the accumulation is output as the result of the convolution operation.

[0049] Repeatedly read data sequentially from the coprocessor's local data cache and perform multiply-accumulate operations using the memory array until the read address reaches the end address:

[0050] In response to the second fetch instruction sent by the coprocessor, the following fully connected operation steps are performed:

[0051] Repeat the following steps until there is no uncomputed weight data in the weight buffer:

[0052] In the coprocessor's local data cache, data is read sequentially starting from the first data address. After each data is read, the address is increased by the step size, and a multiplication and accumulation operation is performed until the read address reaches the end address, thus obtaining the second intermediate result corresponding to the weight data stored in the current storage array.

[0053] Determine if there is any uncomputed weight data in the weight cache;

[0054] If there are weight data that have not been included in the calculation, then read the next set of weight data from the weight cache and update the weight data stored in the storage array with the next set of weight data.

[0055] The accumulated second intermediate result is output as the result of the fully connected operation.

[0056] Furthermore, the step of sequentially reading data from the coprocessor's local data cache, starting from the first data address and increasing the address by a step size after each data item is read, until the end address is reached, includes:

[0057] Based on the data start address, stride length, and end address carried in the first or second value fetch instruction, starting from the data start address, the address is incremented by the stride length after each data read, and this process is repeated until the current read address reaches the end address.

[0058] Fourthly, the present invention provides an image classification system based on a memory computing chip, comprising:

[0059] The main control unit is used to execute the image classification method based on a memory computing chip as described in the first aspect;

[0060] The coprocessor communicates bidirectionally with the main control unit via a serial peripheral interface to execute the image classification method based on a memory computing chip as described in the second aspect.

[0061] The in-memory computing module communicates bidirectionally with the main control unit via a high-speed data bus and bidirectionally with the coprocessor via a data link, and is used to execute the image classification method based on the in-memory computing chip described in the third aspect.

[0062] A clock generator, connected to the main control unit, is used to provide clock signals for the real-time operating system;

[0063] An external dynamic random access memory, connected to the main control unit, is used to store model data and intermediate data during the inference process of the deep neural network.

[0064] The power management module, connected to the main control unit, is used to supply power to the main control unit, coprocessor, and in-memory computing chip, and to control the sleep state and wake-up operation of the main control unit.

[0065] Furthermore, the main control unit includes a main controller, a data path management module, an image sensor interface, an instruction decoding module, an interrupt handling module, and a control logic module;

[0066] The data path management module, the image sensor interface, the instruction decoding module, the interrupt handling module, and the control logic module are all connected to the main controller.

[0067] The data path management module manages the data transmission paths between modules; the image sensor interface acquires image data to be classified; the instruction decoding module receives and decodes external instructions; the interrupt handling module responds to interrupt events; the control logic module generates control signals based on the decoded external instructions; and the main controller executes control logic and data processing.

[0068] The coprocessor includes a cocontroller, a specific arithmetic unit, a local data cache module, and a coprocessor control module;

[0069] The specific computing unit, the local data cache module, and the coprocessing control module are respectively connected to the coprocessor controller;

[0070] The co-controller is used to execute control logic and data processing;

[0071] The specific computational unit includes a convolutional computational unit and a fully connected computational unit, which are used to perform data scheduling, data transformation and data padding.

[0072] The specific computational unit is used to perform quantization operations, dequantization operations, data transformation operations, linear rectification operations, pooling operations, and convolutional layer filling operations;

[0073] The local data caching module is used to temporarily store data to be processed and intermediate results;

[0074] The coprocessor control module is used to parse the control commands of the main control unit and feed back status information;

[0075] The in-memory computing module includes an in-memory computing array, a data input / output interface, a control interface, and a weight cache;

[0076] The data input / output interface, the control interface, and the weight cache are all bidirectionally connected to the memory array.

[0077] The storage array consists of multiple storage units, each of which is used to store weight data and perform multiplication and accumulation operations.

[0078] The data input / output interface is used to receive data to be processed and output the calculation results;

[0079] The control interface is used to receive control signals from the main control unit;

[0080] The weight cache is used to store the weight data of the deep neural network and adopts a first-in, first-out (FIFO) strategy.

[0081] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0082] 1. This invention utilizes the main control unit to send a start computation command to the coprocessor and then enter a sleep state. It is only awakened from the sleep state in response to the inference completion signal sent by the coprocessor. This allows the main control unit to remain in a sleep state and not participate in computation during the convolution and fully connected operations performed by the in-memory computing module. This avoids the main control unit being in a working state for a long time during inference, reduces the power consumption of the main control unit, and thus reduces the overall power consumption of the image classification method. It solves the problems of high power consumption, inability to handle floating-point operations, and high development threshold of existing in-memory computing chips in image classification applications.

[0083] 2. This invention utilizes a coprocessor that responds to a start computation command sent by the main control unit, sending a first value-retrieval instruction for convolution operations and a second value-retrieval instruction for fully connected operations to the in-memory computing module. The coprocessor also performs collaborative computation with the in-memory computing module, including executing auxiliary computation tasks and monitoring the status signals returned by the in-memory computing module. After the collaborative computation is completed, the inference result is stored in a designated memory address, and an inference completion signal is sent to the main control unit. When performing auxiliary computation tasks, the coprocessor compensates for the in-memory computing module's inability to perform floating-point operations. Simultaneously, by monitoring the status signals returned by the in-memory computing module, it ensures the correct flow of data between the coprocessor and the in-memory computing module, thereby improving the operational reliability of the deep neural network inference process.

[0084] 3. In this invention, the in-memory computing module responds to a first value fetch instruction sent by the coprocessor, repeatedly executing data reading and multiplication-accumulation operations. During each weight update, it reads the next set of weight data from the weight cache based on the offset, until there is no un-calculated weight data in the weight cache. Through these operations, the in-memory computing module can complete the convolution operation of multiple sets of weight data with the same input data under a single value fetch instruction, eliminating the need for the coprocessor to repeatedly send value fetch instructions. This reduces the number of instruction interactions, improves the efficiency of weight data updates, and enhances the computational efficiency of deep neural network inference.

[0085] 4. In this invention, the main control unit, coprocessor, and in-memory computing module have clearly defined roles: the main control unit is responsible for task scheduling and result processing, the coprocessor is responsible for auxiliary calculation and data scheduling, and the in-memory computing module is responsible for multiplication and accumulation operations. These three components are connected via serial peripheral interfaces, high-speed data buses, and data links, forming a three-level heterogeneous architecture. This architecture leverages the high energy efficiency of the in-memory computing module while the coprocessor compensates for its inability to perform floating-point operations. Furthermore, the main control unit can enter a sleep state after sending a start-up command, reducing overall system power consumption. Attached Figure Description

[0086] Figure 1 This is a flowchart illustrating an image classification method based on a memory computing chip, executed by a main control unit, provided in an embodiment of the present invention.

[0087] Figure 2 This is a schematic flowchart of an image classification method based on a memory computing chip, executed by a coprocessor, provided in an embodiment of the present invention.

[0088] Figure 3 This is a flowchart illustrating an image classification method based on a memory computing chip, executed by a memory computing module, according to an embodiment of the present invention.

[0089] Figure 4This is a schematic diagram of the structure of an image classification system based on a memory computing chip provided in an embodiment of the present invention. Detailed Implementation

[0090] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0091] Example 1

[0092] like Figure 1 As shown, this embodiment introduces an image classification method based on a memory computing chip, executed by the main control unit, including:

[0093] Step 1: Configure the real-time operating system clock and memory, and initialize the serial peripheral interface communication protocol parameters between the coprocessor and the coprocessor.

[0094] This embodiment configures the real-time operating system clock and memory, initializes the serial peripheral interface communication protocol parameters between the main control unit and the coprocessor, and establishes a stable communication link between the main control unit and the coprocessor, providing a communication basis for the main control unit to send instructions to the coprocessor and receive status information.

[0095] Step 2: Initialize the in-memory computing module, set the computing mode of the in-memory computing array, and configure the data link.

[0096] This embodiment initializes the in-memory computing module, sets the computing mode of the in-memory computing array, and configures the data link, so that the in-memory computing array works in a preset mode, providing a data transmission path between the main control unit and the in-memory computing module.

[0097] Step 3: Send an initialization command to the coprocessor to configure the clock source, initialize the local data cache, and set the first-in-first-out cache strategy.

[0098] In this embodiment, an initialization command is sent to the coprocessor to configure the clock source, initialize the local data cache, and set the first-in-first-out (FIFO) cache strategy. This ensures that the coprocessor's clock is synchronized with the main control unit, and the local data cache manages data in FIFO order to avoid data read / write conflicts.

[0099] Step 4: Start the real-time operating system and load peripheral drivers.

[0100] In this embodiment, the real-time operating system is started and peripheral drivers are loaded, enabling the main control unit to manage hardware resources such as the coprocessor and in-memory computing module, providing a software operating environment for the subsequent execution of the image classification method.

[0101] Step 5: Load the deep neural network model data into an external dynamic random access memory.

[0102] In this embodiment, the model data of the deep neural network is loaded into an external dynamic random access memory, so that the main control unit and the coprocessor can read the structural information of the deep neural network from the external dynamic random access memory and provide model parameters for the subsequent inference process.

[0103] Step 6: Load the weight data of the deep neural network into the weight cache of the in-memory computing module, and rearrange it according to the storage layout and capacity of the in-memory computing array to determine the offset for each weight update.

[0104] Step 6.1: Send the weight data of the deep neural network to the weight cache area of ​​the in-memory computing module.

[0105] Step 6.2: Arrange the weight data in the weight cache area according to the storage layout determined by the number of rows and columns of the in-memory array, and the capacity determined by the total amount of weight data that the in-memory array can store.

[0106] Step 6.3: Determine the offset for each weight update based on the number of columns in the storage array.

[0107] In this embodiment, the offset is the number of weight entries that the weight buffer read pointer moves backward each time.

[0108] In this embodiment, the weight data of the deep neural network is loaded into the weight cache of the in-memory computing module and rearranged according to the storage layout and capacity of the in-memory computing array to determine the offset for each weight update. This ensures that the weight data is arranged according to the number of rows and columns of the in-memory computing array. When the weight is updated, the in-memory computing array reads the next set of weight data from the weight cache according to the offset, without having to recalculate the address each time, thus improving the loading efficiency of the weight data.

[0109] Step 7: Receive the image data to be classified and perform preprocessing.

[0110] This embodiment receives image data to be classified and preprocesses it to convert the raw image data into a format suitable for input to a deep neural network, providing standardized image data for subsequent quantization operations.

[0111] Step 8: Send a quantization command to the coprocessor, so that the coprocessor generates quantized data based on the preprocessed image data and returns a preprocessing completion signal.

[0112] In this embodiment, a quantization instruction is sent to the coprocessor, which generates quantized data based on the preprocessed image data and returns a preprocessing completion signal. This allows the main control unit to know that the quantized data is ready after receiving the preprocessing completion signal, thus preventing the main control unit from executing subsequent steps prematurely when the data is not ready.

[0113] Step 9: In response to the preprocessing completion signal sent by the coprocessor, the quantized data is grouped and packaged according to the structure of the deep neural network and the number of columns of the storage array to generate packaged data.

[0114] Step 9.1: Based on the structure of the deep neural network, determine the connection order of each layer as the processing order of the quantized data.

[0115] Step 9.2: Based on the number of columns in the storage array and the processing order of the quantized data, divide the quantized data into multiple data blocks.

[0116] Step 9.3: Encapsulate each data block into a data packet containing a header and payload to generate packaged data.

[0117] In this embodiment, in response to the preprocessing completion signal sent by the coprocessor, the quantized data is grouped and packaged according to the structure of the deep neural network and the number of columns of the in-memory array, generating packaged data. This allows the quantized data to be divided into multiple data blocks according to the number of columns of the in-memory array and encapsulated into data packets, which match the number of columns of the in-memory array. After receiving the data, the in-memory array does not need to split it again, thus improving the data input efficiency.

[0118] Step 10: After sending the packaged data to the coprocessor and sending a start computation instruction including the first value fetching instruction for convolution operation and the second value fetching instruction for fully connected operation, enter the sleep state.

[0119] In this embodiment, the packaged data is sent to the coprocessor, and a start calculation instruction including a first value-taking instruction for convolution operation and a second value-taking instruction for fully connected operation is sent. Then, the system enters a sleep state, so that the main control unit does not participate in the calculation while the in-memory computing module is performing convolution and fully connected operations, thereby reducing the power consumption of the main control unit and thus reducing the overall power consumption of the image classification method.

[0120] Step 11: In response to the inference completion signal sent by the coprocessor, the system is awakened from the sleep state and reads the inference results from the specified memory address. The inference results include the convolution operation results and the fully connected operation results.

[0121] In this embodiment, the main control unit is awakened from sleep mode in response to the inference completion signal sent by the coprocessor, and the inference result is read from the specified memory address. The inference result includes the convolution operation result and the fully connected operation result, so that the main control unit is only awakened after the inference is completed, avoiding the main control unit from being in working state for a long time and further reducing power consumption.

[0122] Step 12: Perform post-processing operations on the inference results to obtain the image classification results.

[0123] This embodiment performs post-processing operations on the inference results to obtain image classification results, so that the output of the deep neural network is converted into identifiable classification labels, thus completing the image classification task.

[0124] Example 2

[0125] like Figure 2 As shown, based on the same inventive concept as Embodiment 1, this embodiment introduces an image classification method based on a memory computing chip, executed by a coprocessor, including:

[0126] Step 1: In response to the initialization command sent by the main control unit, configure the clock source, initialize the local data buffer, and set the first-in-first-out buffer strategy.

[0127] In this embodiment, in response to the initialization command sent by the main control unit, the clock source is configured, the local data cache is initialized, and a first-in-first-out (FIFO) cache strategy is set, so that the clock of the coprocessor is synchronized with the main control unit, and the local data cache manages data in FIFO order, thus preparing for instruction and data interaction between the main control unit and the coprocessor.

[0128] Step 2: In response to the quantization command sent by the main control unit, generate quantized data based on the preprocessed image data and return a preprocessing completion signal.

[0129] In this embodiment, in response to the quantization command sent by the main control unit, quantization data is generated based on the preprocessed image data, and a preprocessing completion signal is returned. This allows the main control unit to know that the quantization data is ready after receiving the preprocessing completion signal, without having to actively query the quantization operation status, thus reducing the burden on the main control unit.

[0130] Step 3: In response to the start calculation command sent by the main control unit, send the first value retrieval command for convolution operation and the second value retrieval command for fully connected operation to the in-memory computing module, and perform collaborative calculation with the in-memory computing module. The collaborative calculation includes executing auxiliary calculation tasks and monitoring the status signals returned by the in-memory computing module.

[0131] In this embodiment, the auxiliary computing tasks include: quantization operation, dequantization operation, data transformation operation, linear rectification operation, pooling operation, and convolutional layer padding operation; the quantization operation includes performing linear uniform quantization on floating-point data to convert floating-point values ​​into integer values; the dequantization operation includes performing linear uniform dequantization on integer values ​​to convert integer values ​​into floating-point values; the data transformation operation includes a reshape operation that changes the data dimension without changing the total number of data, and a flatten operation that flattens multidimensional data into a one-dimensional vector; the linear rectification operation includes sequentially reading each value in the local data cache, replacing the value with 0 if the value is less than 0, otherwise keeping the value unchanged; the pooling operation includes max pooling that takes the maximum value within the pooling window, and average pooling that takes the arithmetic mean within the pooling window; the convolutional layer padding operation includes padding the input data with zero values ​​within a preset boundary range.

[0132] In this embodiment, if no status signal is received from the in-memory computing module within a preset time, an abnormal status signal is sent to the main control unit.

[0133] In this embodiment, in response to the start computation command sent by the main control unit, the coprocessor sends a first value-taking instruction for convolution operations and a second value-taking instruction for fully connected operations to the in-memory computing module, and performs collaborative computation with the in-memory computing module. This collaborative computation includes executing auxiliary computation tasks and monitoring the status signals returned by the in-memory computing module. When executing auxiliary computation tasks, the coprocessor converts floating-point values ​​to integer values ​​through quantization operations, converts integer values ​​back to floating-point values ​​through dequantization operations, changes the data dimension or flattens multi-dimensional data into a one-dimensional vector through data transformation operations, replaces values ​​less than 0 with 0 through linear rectification operations, retrieves the maximum value or arithmetic mean within the pooling window through pooling operations, and fills zero values ​​at the input data boundaries through convolutional layer padding operations. These operations compensate for the in-memory computing module's inability to perform floating-point operations. Simultaneously, the coprocessor monitors the status signals returned by the in-memory computing module. If no status signal is received from the in-memory computing module within a preset time, an abnormal status signal is sent to the main control unit, preventing the system from continuously waiting when an anomaly occurs and improving the operational reliability of the image classification method.

[0134] Step 4: After the collaborative operation is completed, the inference result is stored in the specified memory address, and an inference completion signal is sent to the main control unit. The inference result includes the convolution operation result and the fully connected operation result.

[0135] In this embodiment, after the collaborative operation is completed, the inference result is stored in a designated memory address and an inference completion signal is sent to the main control unit. This eliminates the need for the main control unit to actively query whether the inference is complete, further reducing the burden on the main control unit. At the same time, since the inference result is stored in a designated memory address, the main control unit can directly read the result from that address without the need for the coprocessor to forward it again.

[0136] Example 3

[0137] like Figure 3 As shown, based on the same inventive concept as other embodiments, this embodiment introduces an image classification method based on a memory computing chip, executed by a memory computing module, including:

[0138] Step 1: In response to the initialization command sent by the main control unit, set the computing mode of the in-memory array and configure the data link.

[0139] In this embodiment, in response to the initialization command sent by the main control unit, the computing mode of the in-memory array is set and the data link is configured, so that the in-memory array works in a preset mode, providing a data transmission path between the main control unit and the in-memory computing module.

[0140] Step 2: Obtain the offset for each weight update determined by the main control unit.

[0141] This embodiment obtains the offset of each weight update determined by the main control unit, so that the in-memory array can read the next set of weight data from the weight cache according to the offset when the weight is updated, without having to recalculate the address each time, thus improving the loading efficiency of weight data.

[0142] Step 3: In response to the first value fetch instruction sent by the coprocessor, perform the following convolution operation steps:

[0143] Step 3.1: Repeat the following steps until there are no uncomputed weight data in the weight buffer:

[0144] Step 3.1.1: In the coprocessor's local data cache, read data sequentially starting from the first data address. After reading each data, increase the address by the step size, and simultaneously perform multiplication and accumulation operations using each storage unit in the storage array until the read address reaches the end address, thus obtaining the first intermediate result corresponding to the weight data stored in the current storage array.

[0145] In this embodiment, based on the data start address, stride length, and end address carried in the first or second value fetch instruction, starting from the data start address, the address is increased by the stride length after each data is read, and this process is repeated until the current read address reaches the end address.

[0146] Step 3.1.2: Determine whether there is any uncomputed weight data in the weight cache.

[0147] If there are weight data that have not been included in the calculation, the next set of weight data is read from the weight cache, and the weight data stored in the in-memory array is updated with the next set of weight data. The starting address of the next set of weight data is shifted backward by the offset relative to the starting address of the weight data stored in the current in-memory array.

[0148] Step 3.1.3: Output the accumulated first intermediate result as the result of the convolution operation.

[0149] Step 3.1.4: Repeatedly read data from the coprocessor's local data cache and perform multiply-accumulate operations using the memory array until the read address reaches the end address.

[0150] In this embodiment, in response to the first value fetching instruction sent by the coprocessor, the convolution operation step is executed. The following operations are repeated until there is no uncomputed weight data in the weight buffer: data is read sequentially from the starting address of the data in the coprocessor's local data buffer, with the address incremented by a stride after each data read, and multiplication and accumulation operations are performed simultaneously by each memory unit in the memory array until the read address reaches the end address, obtaining a first intermediate result corresponding to the weight data stored in the current memory array; it is determined whether there is uncomputed weight data in the weight buffer; if so, the next set of weight data is read from the weight buffer, and the weight data stored in the memory array is updated to the next set of weight data, wherein the starting address of the next set of weight data is shifted backward by the offset relative to the starting address of the weight data stored in the current memory array; the accumulated first intermediate result is output as the convolution operation result. Through the above operations, the in-memory computing module can complete the convolution operation of multiple sets of weight data with the same input data under a single first value retrieval instruction, without the need for the coprocessor to repeatedly send the first value retrieval instruction, reducing the number of instruction interactions. By utilizing the simultaneous execution of multiplication and accumulation operations by each in-memory computing unit in the in-memory computing array, the parallel computing efficiency of convolution operation is improved.

[0151] Step 4: In response to the second value fetch instruction sent by the coprocessor, perform the following fully connected operation steps:

[0152] Step 4.1: Repeat the following steps until there are no uncomputed weight data in the weight buffer:

[0153] Step 4.1.1: In the coprocessor's local data cache, read data sequentially starting from the first data address. After reading each data item, increase the address by the step size and perform a multiply-accumulate operation until the read address reaches the end address, thus obtaining the second intermediate result corresponding to the weight data stored in the current memory array.

[0154] Step 4.1.2: Determine whether there is any uncomputed weight data in the weight buffer.

[0155] If there are weight data that have not been included in the calculation, the next set of weight data is read from the weight cache, and the weight data stored in the storage array is updated with the next set of weight data.

[0156] Step 4.1.3: Output the accumulated second intermediate result as the result of the fully connected operation.

[0157] In this embodiment, in response to the second value fetch instruction sent by the coprocessor, the full-connection operation steps are executed. The following operations are repeated until there is no uncomputed weight data in the weight buffer: data is read sequentially from the coprocessor's local data buffer starting from the first data address. After each data read, the address is incremented by a step size, and a multiplication-accumulation operation is performed until the end address is reached, obtaining a second intermediate result corresponding to the weight data stored in the current in-memory array; it is then determined whether there is uncomputed weight data in the weight buffer; if so, the next set of weight data is read from the weight buffer, and the weight data stored in the in-memory array is updated to the next set of weight data; the accumulated second intermediate result is output as the full-connection operation result. Through the above operations, the in-memory computing module can complete the full-connection operation with a single second value fetch instruction, without the coprocessor repeatedly sending the second value fetch instruction. The in-memory computing units in the in-memory array perform multiplication-accumulation operations in parallel, improving the computational efficiency of the full-connection operation.

[0158] Example 4

[0159] like Figure 4 As shown, based on the same inventive concept as other embodiments, this embodiment introduces an image classification system based on a memory computing chip, including:

[0160] The main control unit is used to execute the image classification method based on in-memory computing chip described in Example 1.

[0161] In this embodiment, the main control unit includes a main controller, a data path management module, an image sensor interface, an instruction decoding module, an interrupt handling module, and a control logic module.

[0162] In this embodiment, the data path management module, the image sensor interface, the instruction decoding module, the interrupt handling module, and the control logic module are all connected to the main controller. The data path management module manages the data transmission paths between the modules. The image sensor interface acquires image data to be classified. The instruction decoding module receives and decodes external instructions. The interrupt handling module responds to interrupt events. The control logic module generates control signals based on the decoded external instructions. The main controller executes control logic and data processing.

[0163] The coprocessor communicates bidirectionally with the main control unit via a serial peripheral interface to execute the image classification method based on a memory computing chip as described in Example 2.

[0164] In this embodiment, the coprocessor includes a cocontroller, a specific arithmetic unit, a local data cache module, and a coprocessor control module.

[0165] In this embodiment, the specific computation unit, the local data cache module, and the coprocessing control module are respectively connected to the cocontroller; the cocontroller is used to execute control logic and data processing; the specific computation unit includes a convolution operation unit and a fully connected operation unit, used to perform data scheduling, data transformation, and data padding; the specific computation unit is used to perform quantization operations, dequantization operations, data transformation operations, linear rectification operations, pooling operations, and convolutional layer padding operations; the local data cache module is used to temporarily store data to be processed and intermediate results; the coprocessing control module is used to parse the control instructions of the main control unit and feed back status information.

[0166] The in-memory computing module communicates bidirectionally with the main control unit via a high-speed data bus and bidirectionally with the coprocessor via a data link, and is used to execute the image classification method based on the in-memory computing chip described in Example 3.

[0167] In this embodiment, the in-memory computing module includes an in-memory computing array, a data input / output interface, a control interface, and a weight cache.

[0168] In this embodiment, the data input / output interface, the control interface, and the weight cache communicate bidirectionally with the in-memory array; the in-memory array consists of multiple in-memory units, each of which stores weight data and performs multiplication-accumulation operations; the data input / output interface receives data to be processed and outputs the calculation results; the control interface receives control signals from the main control unit; and the weight cache stores the weight data of the deep neural network using a first-in-first-out (FIFO) strategy.

[0169] The clock generator, connected to the main control unit, is used to provide clock signals for the real-time operating system.

[0170] External dynamic random access memory, connected to the main control unit ( Figure 4 (Not shown in the image), used to store model data and intermediate data during the inference process of deep neural networks.

[0171] The power management module is connected to the main control unit. Figure 4 (Not shown in the image) is used to power the main control unit, coprocessor, and in-memory computing chip, and to control the sleep state and wake-up operation of the main control unit.

[0172] In summary, this invention utilizes the main control unit to send a start computation instruction to the coprocessor and then enter a sleep state. It is only awakened from the sleep state in response to the inference completion signal sent by the coprocessor. This allows the main control unit to remain in a sleep state and not participate in computation during the convolution and fully connected operations performed by the in-memory computing module. This avoids the main control unit being in a working state for a long time during inference, reduces the power consumption of the main control unit, and thus reduces the overall power consumption of the image classification method. It solves the problems of high power consumption, inability to handle floating-point operations, and high development threshold of existing in-memory computing chips in image classification applications.

[0173] This invention utilizes a coprocessor that responds to a start computation command sent by the main control unit. It sends a first value-retrieval instruction for convolution operations and a second value-retrieval instruction for fully connected operations to the in-memory computing module, and performs collaborative computation with the module. This collaborative computation includes executing auxiliary computation tasks and monitoring the status signals returned by the in-memory computing module. After the collaborative computation is completed, the inference result is stored in a designated memory address, and an inference completion signal is sent to the main control unit. When performing auxiliary computation tasks, the coprocessor compensates for the in-memory computing module's inability to perform floating-point operations. Simultaneously, by monitoring the status signals returned by the in-memory computing module, it ensures the correct flow of data between the coprocessor and the module, thereby improving the operational reliability of the deep neural network inference process.

[0174] In this invention, the in-memory computing module responds to a first value fetch instruction sent by the coprocessor, repeatedly executing data reading and multiplication-accumulation operations. During each weight update, it reads the next set of weight data from the weight cache based on the offset, until no uncomputed weight data remains in the weight cache. Through these operations, the in-memory computing module can complete the convolution operation of multiple sets of weight data with the same input data in a single value fetch instruction, eliminating the need for the coprocessor to repeatedly send value fetch instructions. This reduces the number of instruction interactions, improves the efficiency of weight data updates, and enhances the computational efficiency of deep neural network inference.

[0175] In this invention, the main control unit, coprocessor, and in-memory computing module have clearly defined roles: the main control unit is responsible for task scheduling and result processing, the coprocessor is responsible for auxiliary calculations and data scheduling, and the in-memory computing module is responsible for multiplication and accumulation operations. These three components are connected via a serial peripheral interface, a high-speed data bus, and a data link, forming a three-level heterogeneous architecture. This architecture leverages the high energy efficiency of the in-memory computing module while the coprocessor compensates for its inability to perform floating-point operations. Furthermore, the main control unit can enter a sleep state after sending a start-up command, reducing overall system power consumption.

[0176] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0178] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An image classification method based on a memory computing chip, characterized in that, Executed by the main control unit, including: Configure the real-time operating system clock and memory, and initialize the serial peripheral interface communication protocol parameters with the coprocessor; Perform initialization on the in-memory computing module, set the computing mode of the in-memory computing array, and configure the data link; Send an initialization command to the coprocessor to configure the clock source, initialize the local data cache, and set the first-in-first-out cache strategy. Start the real-time operating system and load peripheral drivers; Load the model data of the deep neural network into an external dynamic random access memory; The weight data of the deep neural network is loaded into the weight cache of the in-memory computing module and rearranged according to the storage layout and capacity of the in-memory computing array to determine the offset for each weight update. Receive image data to be classified and perform preprocessing; Send a quantization instruction to the coprocessor, causing the coprocessor to generate quantized data based on the preprocessed image data and return a preprocessing completion signal; In response to the preprocessing completion signal sent by the coprocessor, the quantized data is grouped and packaged according to the structure of the deep neural network and the number of columns of the storage array to generate packaged data; After sending the packaged data to the coprocessor and sending a start computation instruction including a first value fetching instruction for convolution operations and a second value fetching instruction for fully connected operations, it enters a sleep state. In response to an inference completion signal sent by the coprocessor, the system is awakened from a sleep state and reads the inference results from a specified memory address, the inference results including convolution operation results and fully connected operation results; Post-processing operations are performed on the inference results to obtain the image classification results.

2. The image classification method based on a memory computing chip according to claim 1, characterized in that, The weight data of the deep neural network is loaded into the weight cache of the in-memory computing module and rearranged according to the storage layout and capacity of the in-memory array to determine the offset for each weight update, including: The weight data of the deep neural network is sent to the weight cache of the in-memory computing module; The weight data is arranged in the weight cache according to the storage layout determined by the number of rows and columns of the in-memory array, and the capacity determined by the total amount of weight data that the in-memory array can store. The offset for each weight update is determined based on the number of columns in the storage array. The offset is the number of weight entries that the weight cache read pointer moves backward each time.

3. The image classification method based on a memory computing chip according to claim 1, characterized in that, The quantized data is grouped and packaged according to the structure of the deep neural network and the number of columns in the storage array to generate packaged data, including: Based on the structure of a deep neural network, the connection order of each layer is determined as the processing order of quantized data. Based on the number of columns in the storage array and the processing order of the quantized data, the quantized data is divided into multiple data blocks; Each data block is encapsulated into a data packet containing a header and a payload, generating packaged data.

4. An image classification method based on a memory computing chip, characterized in that, Executed by the coprocessor, including: In response to the initialization command sent by the main control unit, configure the clock source, initialize the local data buffer, and set the first-in-first-out buffer strategy; In response to the quantization command sent by the main control unit, quantized data is generated based on the preprocessed image data, and a preprocessing completion signal is returned; In response to the start calculation command sent by the main control unit, the system sends a first value retrieval command for convolution operation and a second value retrieval command for fully connected operation to the in-memory computing module, and performs collaborative calculation with the in-memory computing module. The collaborative calculation includes performing auxiliary calculation tasks and monitoring the status signals returned by the in-memory computing module. After the collaborative computation is completed, the inference result is stored in a specified memory address, and an inference completion signal is sent to the main control unit. The inference result includes the convolution operation result and the fully connected operation result.

5. The image classification method based on a memory computing chip according to claim 4, characterized in that, The auxiliary computing tasks include: quantization operation, dequantization operation, data transformation operation, linear rectification operation, pooling operation, and convolutional layer filling operation; The quantization operation includes performing linear uniform quantization on floating-point data to convert floating-point values ​​into integer values; The dequantization operation includes performing linear uniform dequantization on integer values ​​to convert integer values ​​into floating-point values; The data transformation operations include the reshape operation, which changes the data dimension but not the total number of data, and the flatten operation, which flattens multidimensional data into a one-dimensional vector. The linear rectification operation includes sequentially reading each value in the local data cache, replacing the value with 0 if the value is less than 0, otherwise keeping the value unchanged. The pooling operations include max pooling, which takes the maximum value within the pooling window, and average pooling, which takes the arithmetic mean within the pooling window. The convolutional layer padding operation includes adding zero values ​​within a preset boundary range of the input data.

6. The image classification method based on a memory computing chip according to claim 4, characterized in that, If no status signal is received from the in-memory computing module within the preset time, an abnormal status signal is sent to the main control unit.

7. An image classification method based on a memory computing chip, characterized in that, Executed by the in-memory computing module, including: In response to the initialization command sent by the main control unit, the computing mode of the in-memory array is set and the data link is configured; Obtain the offset for each weight update determined by the main control unit; In response to the first fetch instruction sent by the coprocessor, the following convolution operation steps are performed: Repeat the following steps until there is no uncomputed weight data in the weight buffer: In the coprocessor's local data cache, data is read sequentially starting from the first data address. After each data is read, the address is increased by the step size, and the multiplication and accumulation operations are performed simultaneously by each storage unit in the storage array until the read address reaches the end address, thus obtaining the first intermediate result corresponding to the weight data stored in the current storage array. Determine if there is any uncomputed weight data in the weight cache; If there are weight data that have not been included in the calculation, then the next set of weight data is read from the weight cache, and the weight data stored in the storage array is updated with the next set of weight data. The starting address of the next set of weight data is shifted backward by the offset relative to the starting address of the weight data stored in the current storage array. The first intermediate result obtained from the accumulation is output as the result of the convolution operation. Repeatedly read data sequentially from the coprocessor's local data cache and perform multiply-accumulate operations using the memory array until the read address reaches the end address: In response to the second fetch instruction sent by the coprocessor, the following fully connected operation steps are performed: Repeat the following steps until there is no uncomputed weight data in the weight buffer: In the coprocessor's local data cache, data is read sequentially starting from the first data address. After each data is read, the address is increased by the step size, and a multiplication and accumulation operation is performed until the read address reaches the end address, thus obtaining the second intermediate result corresponding to the weight data stored in the current storage array. Determine if there is any uncomputed weight data in the weight cache; If there are weight data that have not been included in the calculation, then read the next set of weight data from the weight cache and update the weight data stored in the storage array with the next set of weight data. The accumulated second intermediate result is output as the result of the fully connected operation.

8. The image classification method based on a memory computing chip according to claim 7, characterized in that, The process of reading data sequentially from the starting address in the coprocessor's local data cache, with the address incremented by a step size after each data entry, continues until the end address is reached, including: Based on the data start address, stride length, and end address carried in the first or second value fetch instruction, starting from the data start address, the address is incremented by the stride length after each data read, and this process is repeated until the current read address reaches the end address.

9. An image classification system based on a memory computing chip, characterized in that, include: The main control unit is used to execute the image classification method based on a memory computing chip as described in any one of claims 1-3; The coprocessor communicates bidirectionally with the main control unit via a serial peripheral interface to execute the image classification method based on a memory computing chip as described in any one of claims 4-6; The in-memory computing module communicates bidirectionally with the main control unit via a high-speed data bus and bidirectionally with the coprocessor via a data link, and is used to execute the image classification method based on the in-memory computing chip as described in claim 7 or 8. A clock generator, connected to the main control unit, is used to provide clock signals for the real-time operating system; An external dynamic random access memory, connected to the main control unit, is used to store model data and intermediate data during the inference process of the deep neural network. The power management module, connected to the main control unit, is used to supply power to the main control unit, coprocessor, and in-memory computing chip, and to control the sleep state and wake-up operation of the main control unit.

10. The image classification system based on a memory computing chip according to claim 9, characterized in that, The main control unit includes a main controller, a data path management module, an image sensor interface, an instruction decoding module, an interrupt handling module, and a control logic module; The data path management module, the image sensor interface, the instruction decoding module, the interrupt handling module, and the control logic module are all connected to the main controller. The data path management module manages the data transmission paths between modules; the image sensor interface acquires image data to be classified; the instruction decoding module receives and decodes external instructions; the interrupt handling module responds to interrupt events; the control logic module generates control signals based on the decoded external instructions; and the main controller executes control logic and data processing. The coprocessor includes a cocontroller, a specific arithmetic unit, a local data cache module, and a coprocessor control module; The specific computing unit, the local data cache module, and the coprocessing control module are respectively connected to the coprocessor controller; The co-controller is used to execute control logic and data processing; The specific computational unit includes a convolutional computational unit and a fully connected computational unit, which are used to perform data scheduling, data transformation and data padding. The specific computational unit is used to perform quantization operations, dequantization operations, data transformation operations, linear rectification operations, pooling operations, and convolutional layer filling operations; The local data caching module is used to temporarily store data to be processed and intermediate results; The coprocessor control module is used to parse the control commands of the main control unit and feed back status information; The in-memory computing module includes an in-memory computing array, a data input / output interface, a control interface, and a weight cache; The data input / output interface, the control interface, and the weight cache are all bidirectionally connected to the memory array. The storage array consists of multiple storage units, each of which is used to store weight data and perform multiplication and accumulation operations. The data input / output interface is used to receive data to be processed and output the calculation results; The control interface is used to receive control signals from the main control unit; The weight cache is used to store the weight data of the deep neural network and adopts a first-in, first-out (FIFO) strategy.

Images