Communication system, communication method, storage medium, and program product

Abstract

The present disclosure relates to a communication system, a communication method, a storage medium, and a program product. The communication system includes an event camera, a preprocessing module, and a neuromorphic chip. The preprocessing module is connected between the event camera and the neuromorphic chip. The preprocessing module is configured to: receive first data from the event camera; determine a storage location of second data on the neuromorphic chip according to the first data; process the first data according to the storage location to obtain a data packet including the second data; and output the data packet to the neuromorphic chip. The communication system proposed by the present disclosure can make full use of the sparse characteristics of the event camera, reduce the communication bandwidth between the event camera and the neuromorphic chip, and can effectively improve the frame rate, thereby giving play to the high energy efficiency advantages of the neuromorphic chip and the event camera.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technologies, and in particular, to a communication system, a communication method, a storage medium, and a program product. Background Art

[0002] With the gradual in-depth research on artificial intelligence algorithms and non-Von Neumann hardware architectures, many new types of hardware and systems have emerged in the academic field and industry, such as rapidly iterating and developing multi-core chips, many-core chips, deep learning accelerators, neuromorphic chips, general-purpose brain computing chips, etc. Different from traditional general-purpose processors such as central processing units (CPUs) and graphics processing units (GPUs) with centralized characteristics, neuromorphic chips with a many-core architecture have typical decentralized characteristics, which makes neuromorphic chips have important advantages such as low power consumption and low latency when used for brain simulation, neuroengineering modeling, and neural network acceleration compared with general-purpose processors.

[0003] An event camera is a new type of sensor. Different from traditional cameras that aim to capture a complete image, event cameras mainly collect the change amount of the target image. Therefore, its output has sparse characteristics, which makes event cameras have advantages such as low latency, high dynamic range, and low power consumption compared with traditional cameras. Since neuromorphic chips and event cameras have similar characteristics in terms of power consumption, latency, etc., how to fully combine neuromorphic chips and event cameras to achieve a communication system with low latency and low power consumption has become a research hotspot in this field. Summary of the Invention

[0004] In view of this, the present disclosure provides a communication system, a communication method, a storage medium, and a program product. The communication system provided by the present disclosure can make full use of the sparse characteristics of the event camera, reduce the communication bandwidth between the event camera and the neuromorphic chip, and can effectively improve the frame rate, thereby giving play to the high energy efficiency advantages of the neuromorphic chip and the event camera.

[0005] According to an aspect of the present disclosure, a communication system is provided, including an event camera, a preprocessing module, and a neuromorphic chip. The preprocessing module is connected between the event camera and the neuromorphic chip, and the preprocessing module is configured to: receive first data from the event camera; determine the storage location of second data on the neuromorphic chip according to the first data; process the first data according to the storage location to obtain a data packet including the second data; and output the data packet to the neuromorphic chip.

[0006] In a possible implementation, the event camera is configured to: generate events under a first image parameter, where the first data includes the position information of the pixel points where the events are generated, the time information of the generated events, and the event polarity information, and the position information of the pixel points where the events are generated is the position information of the pixel points under the first image parameter.

[0007] In a possible implementation, the neuromorphic chip is configured to: obtain image data under a second image parameter according to the data packet.

[0008] In a possible implementation, the neuromorphic chip includes a plurality of processor cores, each processor core corresponds to a storage area for storing the second data, and each processor core is configured to store the second data of at least one pixel point position under the second image parameter.

[0009] In a possible implementation, determining the storage location of the second data on the neuromorphic chip includes: mapping the position information of the pixel points where the events of the first data are generated to the position information of the pixel points under the second image parameter; determining the processor core on which the second data is stored and the storage address corresponding to the processor core according to the position information of the pixel points under the second image parameter; and determining the storage location of the second data on the neuromorphic chip as the storage address corresponding to the processor core.

[0010] In a possible implementation, processing the first data according to the storage location to obtain a data packet including the second data includes: processing the first data according to the storage address corresponding to the processor core in the storage location to obtain a data packet including the second data, such that the packet header of the data packet includes an identifier of the storage address.

[0011] In a possible implementation, the second data at least includes the position information of the pixel points under the second image parameter, the time information of the generated events, and the event polarity information.

[0012] In a possible implementation, the neuromorphic chip is further configured to store the second data to the storage address when receiving the data packet; obtaining the image data under the second image parameter according to the data packet includes: integrating the second data that meets a preset condition to obtain a frame of image data under the second image parameter, where the preset condition is that the time information of the generated events included in the second data is within a first time period.

[0013] In a possible implementation, the communication system further includes a fault tolerance module disposed between the preprocessing module and the neuromorphic chip, and the fault tolerance module is configured to: receive the data packet from the preprocessing module; when it is determined that the data packet has no error, control the preprocessing module to output the data packet to the neuromorphic chip; when it is determined that the data packet has an error, control the preprocessing module not to output the data packet.

[0014] In a possible implementation, outputting the data packet to the neuromorphic chip includes: outputting the data packet to the fault tolerance module; under the control of the fault tolerance module, outputting the data packet to the neuromorphic chip.

[0015] According to another aspect of the present disclosure, there is provided a communication method applied to a preprocessing module in a communication system. The communication system includes an event camera, the preprocessing module, and a neuromorphic chip. The preprocessing module is connected between the event camera and the neuromorphic chip. The method includes: receiving first data from the event camera; determining a storage location of second data on the neuromorphic chip according to the first data; processing the first data according to the storage location to obtain a data packet including the second data; and outputting the data packet to the neuromorphic chip.

[0016] According to another aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by the preprocessing module, the above method is implemented.

[0017] According to another aspect of the present disclosure, there is provided a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in the preprocessing module, the preprocessing module executes the above method.

[0018] A communication system according to an embodiment of the present disclosure receives first data from an event camera through a preprocessing module, determines a storage location of second data on a neuromorphic chip based on the first data, and can process the first data at the determined storage location to obtain a data packet including the second data and output it to the neuromorphic chip. Since the data packet is obtained by real-time processing of the first data from the event camera, it still has sparse characteristics, and the bandwidth required to transmit the data packet can be lower than the bandwidth required to transmit a whole frame of data, reducing the communication bandwidth between the event camera and the neuromorphic chip; and the data packet is obtained by processing based on the determined storage location of the second data on the neuromorphic chip, so it meets the requirements of the neuromorphic chip for data storage, enabling the data in the data packet to be used by the neuromorphic chip. In this way, the communication system according to the embodiment of the present disclosure makes full use of the sparse characteristics of the event camera to reduce the communication bandwidth between the event camera and the neuromorphic chip and effectively improve the frame rate while ensuring the operating stability of the neuromorphic chip, thus giving play to the high energy efficiency advantages of the neuromorphic chip and the event camera.

[0019] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are included in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure together with the specification and are used to explain the principles of the present disclosure.

[0021] Figure 1 FIG. shows an exemplary structural schematic diagram of a communication system according to an embodiment of the present disclosure.

[0022] Figure 2 FIG. shows an example of a storage area corresponding to a processor core on a neuromorphic chip 103 according to an embodiment of the present disclosure.

[0023] Figure 3 FIG. shows an exemplary structural schematic diagram of a communication system according to an embodiment of the present disclosure.

[0024] Figure 4 FIG. shows an exemplary workflow of a communication method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0026] As used herein, the term "exemplary" means "serving as an example, instance, or illustration". Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or better than other embodiments.

[0027] In addition, for a better illustration of the present disclosure, numerous specific details are given in the following detailed description. Those skilled in the art should understand that the present disclosure can be implemented without some of these specific details. In some instances, methods, means, elements, and circuits well-known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0028] In order to combine a neuromorphic chip and an event camera to implement a communication system with low latency and low power consumption, the prior art proposes to accumulate the data output by the event camera over time, restore it into a complete frame of image, and then transmit the restored image data to the neuromorphic chip for processing. Although this solution realizes the combination of the neuromorphic chip and the event camera, to a certain extent, it does not make good use of the sparsity of the data output by the event camera. In the case of high frame rate, it poses very high requirements on the data transmission bandwidth between the neuromorphic chip and the event camera. Usually, the existing neuromorphic chips can process data extremely fast (i.e., low latency), but if the data transmission bandwidth is insufficient, due to communication pressure, the entire communication system may not be able to operate in real time, with high throughput and high frame rate.

[0029] In view of this, the present disclosure proposes a communication system, a communication method, a storage medium, and a program product. The communication system proposed by the present disclosure can make full use of the sparse characteristics of the event camera, reduce the communication bandwidth between the event camera and the neuromorphic chip, and can effectively improve the frame rate, thereby giving play to the high energy efficiency advantages of the neuromorphic chip and the event camera.

[0030] Figure 1 Fig. shows an exemplary structural schematic diagram of a communication system according to an embodiment of the present disclosure.

[0031] As Figure 1 shown, in a possible implementation, the present disclosure proposes a communication system, including an event camera 101, a preprocessing module 102, and a neuromorphic chip 103. The preprocessing module 102 is connected between the event camera 101 and the neuromorphic chip 103, and the preprocessing module 102 is configured to:

[0032] Receive first data from the event camera 101;

[0033] Determine the storage location of second data on the neuromorphic chip 103 according to the first data;

[0034] Process the first data according to the storage location to obtain a data packet including the second data;

[0035] Output the data packet to the neuromorphic chip 103.

[0036] Among them, the event camera 101 can be used as a sensor of the communication system, the neuromorphic chip 103 can be used as the core computing unit of the communication system, and the preprocessing module 102 can be responsible for the data transmission from the event camera 101 to the neuromorphic chip 103. The first data can be the events obtained by the event camera 101, and the second data can be part of the information included in the events that needs to be used for data processing by the neuromorphic chip 103. If the second data is transmitted to the neuromorphic chip 103, it will be stored at a certain storage location on the neuromorphic chip 103. When the storage locations are different, the processing methods of the neuromorphic chip 103 for the second data may be different. Therefore, after receiving the first data from the event camera 101, the preprocessing module 102 can first determine the storage location of the second data on the neuromorphic chip 103, and then obtain a data packet including the second data according to the determined storage location and output it. The format of the data packet can be a data packet that conforms to the routing protocol of the communication system. The present disclosure does not limit the specific type of the routing protocol of the communication system.

[0037] According to the communication system of the embodiments of the present disclosure, by receiving the first data from the event camera through the preprocessing module and determining the storage location of the second data on the neuromorphic chip according to the first data, the first data can be processed at the determined storage location to obtain a data packet including the second data and output it to the neuromorphic chip. Since the data packet is obtained by real-time processing of the first data from the event camera, it still has the sparse characteristic. The bandwidth required to transmit the data packet can be lower than the bandwidth required to transmit the entire frame of data, so that the communication bandwidth between the event camera and the neuromorphic chip is reduced; and the data packet is obtained by processing according to the determined storage location of the second data on the neuromorphic chip, so it meets the requirements of the neuromorphic chip for data storage, so that the data in the data packet can be used by the neuromorphic chip. In this way, the communication system of the embodiments of the present disclosure makes full use of the sparse characteristic of the event camera to reduce the communication bandwidth between the event camera and the neuromorphic chip and effectively improve the frame rate, while ensuring the operation stability of the neuromorphic chip, so as to exert the high energy efficiency advantages of the neuromorphic chip and the event camera.

[0038] In a possible implementation manner, the event camera 101 is configured to: generate events under the first image parameters, the first data includes the pixel point position information of the generated events, the time information of the generated events, and the event polarity information, and the pixel point position information of the generated events is the pixel point position information under the first image parameters.

[0039] The event camera 101 of the embodiments of the present disclosure can be implemented based on the prior art. The working principle of the event camera 101 is briefly introduced below.

[0040] An event camera is an image sensor inspired by the biological retina. Different from traditional cameras that capture a complete image at a fixed frame rate, each pixel of an event camera independently detects changes in relative light intensity and only generates and outputs first data (i.e., events) when there are changes in relative light intensity. Therefore, it usually has the characteristics of sparsity and low redundancy.

[0041] The first data output by the event camera 101 can be the following quadruple:

[0042] (timestamp, polarity, x, y)

[0043] Among them, timestamp can be the time information when the event occurs, which can represent absolute time, and the resolution can reach 1 μs.

[0044] x and y can be the position information of the pixel points where the events occur. The number of pixel points included in the event camera 101 can be related to its parameters. For example, the first image parameter includes the number of pixel points a*b included in the event camera (a pixel points per row and b pixel points per column). When the event camera 101 generates an event under the first image parameter a*b, it can be considered that the event camera 101 includes a*b pixel points. If the pixel point located in the m-th row and the n-th column generates an event, then x and y can be equal to m and n respectively, where 0 ≤ m ≤ a and m is an integer, 0 ≤ n ≤ b and n is an integer.

[0045] polarity can be the event polarity information, which is used to indicate the change in the brightness of the pixel point where the event occurs at the time of the event compared to the brightness at an earlier time. When the brightness increase exceeds a certain threshold, the value of the event polarity information can be 1, and when the brightness decrease exceeds a certain threshold, the value of the event polarity information can be -1.

[0046] Therefore, when the event camera 101 generates an event under the first image parameter a*b, it can generate the first data when the brightness changes at a certain pixel point under the first image parameter a*b, so that the first data can include the position information of the pixel point where the event occurs, the time information when the event occurs, and the event polarity information. Since the event camera 101 generates an event under the first image parameter a*b, therefore, the position information of the pixel point where the event occurs can also be the pixel point position information under the first image parameter a*b.

[0047] In a possible implementation manner, the neuromorphic chip 103 is configured to: obtain image data under a second image parameter according to the data packet.

[0048] The neuromorphic chip 103 according to an embodiment of the present disclosure, as the core computing unit of a communication system, can be used to run some algorithms or models of the prior art, such as algorithms or models that can implement image recognition, video object detection, etc., for data processing. Generally, the parameters of the input data of the algorithms or models that the neuromorphic chip 103 can run are usually set in advance. For example, in the embodiment of the present disclosure, the input data parameters of the algorithms or models run by the neuromorphic chip 103 can be set in advance to the second image parameter c*d, where the second image parameter can represent the number of pixel points of the input data of the algorithms or models run by the neuromorphic chip 103. Since the data packet is sparse data and does not include the entire frame of data, the neuromorphic chip 103 is used to obtain the image data under the second image parameter according to the data packet, so as to obtain the entire frame of image data that can be input into the algorithms or models run by the neuromorphic chip 103. Its exemplary implementation manner is described below.

[0049] In a possible implementation manner, the neuromorphic chip 103 includes a plurality of processor cores, and each processor core corresponds to a storage area for storing the second data, and each processor core is used to store the second data at at least one pixel point position under the second image parameter.

[0050] For example, the neuromorphic chip 103 according to an embodiment of the present disclosure can be a chip with a many-core architecture, that is, the neuromorphic chip 103 can include a plurality of processor cores. Each processor core can correspond to a storage area for storing the second data, and the storage area can be located on the neuromorphic chip 103. Each processor core can be used to store the second data at at least one pixel point position under the second image parameter c*d. Optionally, which pixel point positions are included in the at least one pixel point position can be set in advance or determined in real time according to the application scenario requirements, the memory of the storage area corresponding to each processing core, etc. Its pre-setting method and real-time determination method can be implemented based on the prior art, and the present disclosure does not limit this. The storage address ranges of different storage areas can be different, and the storage addresses corresponding to each pixel point position on each processor core can also be different.

[0051] Figure 2 An example of the storage area corresponding to a processor core on the neuromorphic chip 103 according to an embodiment of the present disclosure is shown. As Figure 2 shown, the storage area includes a sub-region A for storing the second data during the time period corresponding to the current frame of image (for example, the first time period described below). In the sub-region A, corresponding storage addresses are respectively divided for at least one pixel point under the second image parameter c*d, and the storage addresses corresponding to different pixel points can be different. Figure 2 where (x’, y’) is the coordinate of the pixel point under the second image parameter c*d. Then Figure 2In the example, it is assumed that the second data precision is INT8. At this time, among at least one pixel point (such as (0, 0), (1, 0), (2, 0), (3, 0), (0, 1), (1, 1), (2, 1), etc.) under the second image parameter c*d, the data of each pixel point needs to occupy an address space of 16 Byte (8 Byte + 8 Byte). By setting the storage area corresponding to the processor core, it is more convenient to perform routing addressing of data packets and align the data of the neuromorphic chip and the event camera.

[0052] In this way, the storage area corresponding to each processor core can store the second data, making it more convenient for the neuromorphic chip to implement the functions of its many-core architecture and improving the data processing efficiency of the neuromorphic chip.

[0053] In one possible implementation manner, determining the storage location of the second data on the neuromorphic chip 103 includes:

[0054] Mapping the pixel point position information of the generation event of the first data to the pixel point position information under the second image parameter;

[0055] According to the pixel point position information under the second image parameter, determining the processor core on which the second data is stored on the neuromorphic chip 103 and the storage address corresponding to the processor core;

[0056] Determining the storage location of the second data on the neuromorphic chip 103 as storing to the storage address corresponding to the processor core.

[0057] For example, since each processor core corresponds to a storage area for storing the second data, the processing capabilities of multiple processor cores may be different, and the algorithms or models that can be run may also be different. Therefore, before the preprocessing module 102 obtains the data packet, it is necessary to determine the storage location of the second data on the neuromorphic chip 103. The storage location may include processor core information and storage address information, such as which storage area corresponding to the processor core to store in and which part of the storage address in the storage area to specifically store in. The second data that each processor core can store is limited. Therefore, according to the application scenario requirements and the size of the storage area corresponding to each processor core, when the data volume of a frame of image data under the second image parameter c*d is relatively small, one processor core may be able to process it, that is, the second data corresponding to different pixel point positions under the second image parameter c*d can be stored in the same processor core; when the data volume of a frame of image data under the second image parameter c*d is relatively large, multiple processor cores may need to cooperate for processing, that is, the second data corresponding to different pixel point positions under the second image parameter c*d may need to be stored in different processor cores.

[0058] The pixel position information of the generating event included in the first data is the pixel position under the first image parameter a*b, while the neuromorphic chip 103 obtains the image data under the second image parameter c*d according to the data packet. When the first image parameter a*b and the second image parameter c*d are different, it is necessary to map the pixel position information of the generating event (i.e., the pixel position under the first image parameter a*b) to make it the pixel position under the second image parameter c*d. The mapping relationship between the pixel position information under the first image parameter a*b and the pixel position information under the second image parameter c*d can be known. Based on this, according to the pixel position information of the generating event in the first data, for example, (x, y) = (0, 0), a pixel position under the second image parameter corresponding to this pixel position can be mapped, for example, (x', y') = (0, 0). Since it is preset or determined in each processor core which pixel positions are stored at which storage addresses, when the pixel positions under the second image parameter c*d corresponding to each processor core are known, the second data corresponding to the pixel position (0, 0) under this second image parameter can be uniquely determined to be stored in the storage area corresponding to which processor core, for example, processor core 1. And since the storage addresses corresponding to each pixel position on each processor core are different, according to a pixel position under the second image parameter corresponding to the pixel position of the generating event in the first data, the storage address of the second data corresponding to the pixel position under this second image parameter in the storage area corresponding to this processor core can be uniquely determined, for example, the 16-byte storage address p~q corresponding to (x', y') = (0, 0) in processor core 1.

[0059] On this basis, it can be determined that the storage position of the second data on the neuromorphic chip 103 is: the second data is stored in the storage address p~q corresponding to processor core 1.

[0060] In this way, it is possible to determine the storage position of the second data on the neuromorphic chip 103, so that after the data packet is output to the neuromorphic chip 103, when the second data included in it is stored at the determined storage position, the processor core can normally obtain the second data and run the corresponding algorithm or model to process the second data.

[0061] In a possible implementation manner, the processing the first data according to the storage position to obtain a data packet including the second data includes:

[0062] Processing the first data according to the storage address corresponding to the processor core in the storage position to obtain a data packet including the second data, so that the identification of the storage address is included in the packet header of the data packet.

[0063] For example, the storage location of the second data is determined by the preprocessing module 102. To enable the neuromorphic chip 103 to store the second data according to this storage location, the neuromorphic chip 103 also needs to obtain the relevant information of this storage location.

[0064] For example, the neuromorphic chip 103 may pre-store the identifiers corresponding to the storage addresses of the sub-regions corresponding to each pixel point in the storage areas corresponding to each processor core. Assuming that the determined storage location is to be stored at the storage address p-q corresponding to the processor core 1, when processing the first data according to the storage location to obtain a data packet including the second data, the identifier of the storage address p-q can be added to the packet header of the data packet. Table 1 shows an example of the information included in the packet header according to an embodiment of the present disclosure.

[0065] Table 1

[0066] Identification S T P Q X Y A Number of digits 1 1 1 1 8 8 12

[0067] Referring to Table 1, the number of bits of this packet header can be 32 bits, where S occupies one bit and is used to distinguish whether the information carried by the data packet is an instruction or data. When S = 0, it indicates that the data packet is a data packet including the second data, and when S = 1, it indicates that the data packet is an instruction packet including an instruction.

[0068] T occupies one bit and is used to indicate the data volume size of the data packet. When T = 0, it indicates that the data packet is a data packet with a small data volume, and when T = 1, it indicates that the data packet is a data packet with a large data volume.

[0069] P occupies one bit and is used to indicate whether the packet header data has been sent. When P = 0, it indicates that the packet header data has not been sent, and when P = 1, it indicates that the packet header data has been sent.

[0070] Q occupies one bit and is used to indicate whether the data packet is a relay / multicast data packet. When Q = 0, it indicates that the data packet is an ordinary data packet, and when Q = 1, it indicates that the data packet is a relay / multicast data packet. When the data packet is a relay / multicast data packet, after the data packet is sent to the corresponding processor core, the processor core can act as a transfer station to forward the data packet to other processor cores.

[0071] X and Y are used to indicate the coordinates of the processor core (destination cluster) to which the data packet is to be sent on the neuromorphic chip. Among them, X is used to indicate the horizontal relative distance of the destination cluster, and Y is used to indicate the vertical relative distance of the destination cluster. When the precision of the second data is int8, X and Y can each occupy 8 bits, covering the range of the horizontal relative distance [-128, 127] and the vertical relative distance [-128, 127].

[0072] A occupies 12 bits, which is the identifier of the storage address and is used to indicate the storage address where the second data will be stored. Since there is a corresponding relationship between the storage address and the pixel position information under the second image parameter c*d, A also indicates the pixel position information under the second image parameter c*d in the second data included in the data packet.

[0073] In this case, based on the identifier of the storage address, the neuromorphic chip 103 can determine the storage location of the second data, thereby meeting the requirement of storing the second data included in the data packet at a determined storage location.

[0074] Those skilled in the art should understand that the packet header of the data packet may also include more information. For example, when each processor core has a corresponding identifier, the packet header of the data packet can also include the identifier of the processor core where the second data is stored on the neuromorphic chip, so that the neuromorphic chip can more quickly determine which processor core processes the second data in the data packet according to the identifier of the processor core in the packet header of the data packet. The present disclosure places no restrictions on the specific information included in the packet header of the data packet.

[0075] In a possible implementation manner, the second data at least includes the pixel position information under the second image parameter, the time information of the generated event, and the event polarity information.

[0076] Since the neuromorphic chip 103 needs to obtain the image data under the second image parameter c*d and the second data is stored in the neuromorphic chip 103, the second data can at least include the pixel position information under the second image parameter c*d, the time information of the generated event, the event polarity information, and other information related to obtaining the image data under the second image parameter c*d. The method of obtaining the image data under the second image parameter c*d according to the pixel position information under the second image parameter c*d, the time information of the generated event, and the event polarity information can be implemented based on the prior art and will not be elaborated here.

[0077] In a possible implementation manner, the neuromorphic chip 103 is further configured to store the second data to the storage address when receiving the data packet;

[0078] The obtaining the image data under the second image parameter according to the data packet includes:

[0079] Integrating the second data that meets the preset conditions to obtain an image data under one frame of the second image parameter, where the preset condition is that the time information of the generated event included in the second data is within the first time period.

[0080] For example, since the packet header includes an identifier for the storage address, when the neuromorphic chip 103 receives a packet, it can store the second data at the storage address according to the identifier of the storage address. Thereafter, when the neuromorphic chip 103 needs to use the second data, it can obtain the corresponding second data from the storage address. For example, the neuromorphic chip 103 obtains the image data under the second image parameter c*d according to the packet, which may be to integrate the second data that meets the preset conditions to obtain a frame of image data under the second image parameter c*d.

[0081] Since this operation involves the integration operation of the second data within the first time period, and the obtained frame of image data under the second image parameter c*d may be input into the algorithms or models on the neuromorphic chip subsequently. That is to say, the second data within the first time period and a frame of image data under the second image parameter c*d may be used repeatedly. Therefore, optionally, as Figure 2 shown, in the storage area corresponding to a processor core, there may also be a sub-region B for storing the second data within the time period corresponding to the previous frame of image (such as the first time period), and a sub-region C for storing the integration of the second data within the time period corresponding to the previous frame of image (such as the first time period). By dividing the corresponding sub-regions to specifically store this part of the data, the data processing efficiency can be higher.

[0082] The above integration operation involves the addition of tensors. Therefore, preferably, it can be set to perform vector sum operation processing on the data at the corresponding pixel point positions within the processor core. Further, there may be a convolution operation on the data within the processor core. Since the convolution operation is performed in the manner of "first depth (referring to the value of the pixel point), then width (referring to the abscissa of the pixel point), and finally height (referring to the ordinate of the pixel point)", a corresponding storage position can also be set for a frame of image data under the second image parameter c*d, for example, set to store in the manner of "first depth, then width, and finally height". By making the storage position of the image data on the neuromorphic chip consistent with the usage mode when the data is used for operation, the data processing efficiency of the neuromorphic chip can be improved.

[0083] Those skilled in the art should understand that the storage area can also be divided in other ways, as long as each processor core can be used to store the second data at at least one pixel point position under the second image parameter. The present disclosure does not limit the specific division method of the storage area corresponding to a processor core of the neuromorphic chip.

[0084] Figure 3 FIG. shows an exemplary structural schematic diagram of a communication system according to an embodiment of the present disclosure.

[0085] In a possible implementation, the communication system further includes a fault tolerance module 104 disposed between the preprocessing module 102 and the neuromorphic chip 103, and the fault tolerance module 104 is configured to:

[0086] Receive the data packet from the preprocessing module 102;

[0087] When it is determined that the data packet has no error, control the preprocessing module 102 to output the data packet to the neuromorphic chip 103;

[0088] When it is determined that the data packet has an error, control the preprocessing module 102 not to output the data packet.

[0089] The fault tolerance module 104 can be used to prevent error accumulation. For example, when the preprocessing module 102 directly outputs the data packet to the neuromorphic chip 103, there is a risk of error accumulation. In this case, in the communication system, a fault tolerance module 104 can be provided, and the fault tolerance module 104 can be disposed between the preprocessing module 102 and the neuromorphic chip 103. The fault tolerance module 104 can receive the data packet from the preprocessing module 102 and determine whether the data packet has an error. When it is determined that the data packet has no error, control the preprocessing module 102 to output the data packet to the neuromorphic chip 103. When it is determined that the data packet has an error, control the preprocessing module 102 not to output the data packet. Optionally, when the frequency of the first data output is relatively high, it will also cause pressure on the communication bandwidth from the event camera to the neuromorphic chip. Therefore, when the fault tolerance module 104 controls the preprocessing module 102 to output the data packet to the neuromorphic chip 103, it can control the output frequency of the data packet to improve the stability of the communication bandwidth from the event camera to the neuromorphic chip.

[0090] In a possible implementation, the outputting the data packet to the neuromorphic chip 103 includes:

[0091] Output the data packet to the fault tolerance module 104;

[0092] Under the control of the fault tolerance module 104, output the data packet to the neuromorphic chip 103.

[0093] For example, when the communication system includes the fault tolerance module 104, the preprocessing module 102 outputs the data packet to the neuromorphic chip 103. It can first output the data packet to the fault tolerance module 104. The fault tolerance module 104 determines whether the data packet has an error. When the fault tolerance module 104 determines that the data packet has no error, the preprocessing module 102 outputs the data packet to the neuromorphic chip 103 under the control of the fault tolerance module 104.

[0094] In this way, the data packets output by the preprocessing module can be error-free data packets, reducing the impact of error accumulation on the accuracy of the neuromorphic chip in processing the second data. Moreover, by controlling the output of data packets through the fault tolerance module, the output frequency of data packets can be balanced, avoiding large fluctuations in the communication bandwidth.

[0095] An embodiment of the present disclosure also provides a communication method. Figure 4 An exemplary workflow of the communication method according to an embodiment of the present disclosure is shown. As Figure 4 shown, the method can be applied to the preprocessing module 102 in a communication system according to an embodiment of the present disclosure. The communication system includes an event camera 101, the preprocessing module 102, and a neuromorphic chip 103. The preprocessing module 102 is connected between the event camera 101 and the neuromorphic chip 103. The method includes:

[0096] Step S1, receiving first data from the event camera 101;

[0097] Step S2, determining the storage location of second data on the neuromorphic chip 103 according to the first data;

[0098] Step S3, processing the first data according to the storage location to obtain a data packet including the second data;

[0099] Step S4, outputting the data packet to the neuromorphic chip 103.

[0100] For an exemplary description of this method, reference can be made to the above, and details will not be repeated here.

[0101] In a possible implementation, the event camera is used to: generate events under first image parameters. The first data includes pixel point position information of the generated events, time information of the generated events, and event polarity information. The pixel point position information of the generated events is the pixel point position information under the first image parameters.

[0102] In a possible implementation, the neuromorphic chip is used to: obtain image data under second image parameters according to the data packet.

[0103] In a possible implementation, the neuromorphic chip includes multiple processor cores. Each processor core corresponds to a storage area for storing the second data. Each processor core is used to store the second data of at least one pixel point position under the second image parameters.

[0104] In a possible implementation, determining the storage location of the second data on the neuromorphic chip includes: mapping the pixel point position information of the generation event of the first data to the pixel point position information under the second image parameter; determining the processor core on which the second data is stored on the neuromorphic chip and the storage address corresponding to the processor core according to the pixel point position information under the second image parameter; and determining the storage location of the second data on the neuromorphic chip as the storage address corresponding to the processor core.

[0105] In a possible implementation, processing the first data according to the storage location to obtain a data packet including the second data includes: processing the first data according to the storage address corresponding to the processor core in the storage location to obtain a data packet including the second data, so that the identification of the storage address is included in the packet header of the data packet.

[0106] In a possible implementation, the second data at least includes the pixel point position information under the second image parameter, the time information of the generation event, and the event polarity information.

[0107] In a possible implementation, the neuromorphic chip is further configured to store the second data at the storage address when receiving the data packet; obtaining the image data under the second image parameter according to the data packet includes: integrating the second data that meets a preset condition to obtain a frame of image data under the second image parameter, where the preset condition is that the time information of the generation event included in the second data is within a first time period.

[0108] In a possible implementation, the communication system further includes a fault tolerance module disposed between the preprocessing module and the neuromorphic chip, and the fault tolerance module is configured to: receive the data packet from the preprocessing module; when it is determined that the data packet has no error, control the preprocessing module to output the data packet to the neuromorphic chip; and when it is determined that the data packet has an error, control the preprocessing module not to output the data packet.

[0109] In a possible implementation, outputting the data packet to the neuromorphic chip includes: outputting the data packet to the fault tolerance module; and outputting the data packet to the neuromorphic chip under the control of the fault tolerance module.

[0110] The embodiments of the present disclosure also propose a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0111] Embodiments of the present disclosure also provide a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0112] The present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing the preprocessing module 102 to implement various aspects of the present disclosure.

[0113] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punched card or raised structures in a groove having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not construed as an instantaneous signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0114] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0115] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present disclosure.

[0116] Aspects of the present disclosure are described herein with reference to the flowchart and / or block diagram of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer - readable program instructions.

[0117] These computer - readable program instructions can be provided to a processor of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data - processing apparatus, create a means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, which causes a computer, a programmable data - processing apparatus, and / or other devices to operate in a particular manner. Thus, the computer - readable medium storing the instructions includes a manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0118] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other devices to generate a computer-implemented process, such that the instructions executed on the computer, other programmable data processing apparatus, or other devices implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0119] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0120] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A communication system, characterized in that, It includes an event camera, a preprocessing module, and a neuromorphic chip. The preprocessing module is connected between the event camera and the neuromorphic chip, and the preprocessing module is used for: Receiving first data from the event camera; Determining the storage location of second data on the neuromorphic chip according to the first data; Processing the first data according to the storage location to obtain a data packet including the second data; Outputting the data packet to the neuromorphic chip; The event camera is used for: generating events under first image parameters. The first data includes the pixel point position information of the generated events, the time information of the generated events, and the event polarity information. The pixel point position information of the generated events is the pixel point position information under the first image parameters; The neuromorphic chip is used for: obtaining image data under second image parameters according to the data packet. The neuromorphic chip includes a plurality of processor cores, and each processor core corresponds to a storage area for storing the second data. Each processor core is used for storing the second data of at least one pixel point position under the second image parameters; The determining the storage location of the second data on the neuromorphic chip includes: mapping the pixel point position information of the generated events in the first data to the pixel point position information under the second image parameters; According to the pixel point position information under the second image parameters, determining the processor core on which the second data is stored on the neuromorphic chip and the storage address corresponding to the processor core; determining the storage location of the second data on the neuromorphic chip as the storage address corresponding to the processor core.

2. The communication system according to claim 1, characterized in that, The processing the first data according to the storage location to obtain a data packet including the second data includes: Processing the first data according to the storage address corresponding to the processor core in the storage location to obtain a data packet including the second data, so that the header of the data packet includes an identifier of the storage address.

3. The communication system according to claim 1 or 2, characterized in that, The second data at least includes the pixel point position information under the second image parameters, the time information of the generated events, and the event polarity information.

4. The communication system according to claim 1, characterized in that, The neuromorphic chip is further used for storing the second data to the storage address when receiving the data packet; The obtaining image data under the second image parameters according to the data packet includes: Integrating the second data that meets a preset condition to obtain a frame of image data under the second image parameters. The preset condition is that the time information of the generated events included in the second data is within a first time period.

5. The communication system according to claim 1, wherein The communication system further includes a fault tolerance module disposed between the preprocessing module and the neuromorphic chip. The fault tolerance module is used for: Receiving the data packet from the preprocessing module; When it is determined that the data packet has no error, controlling the preprocessing module to output the data packet to the neuromorphic chip; When it is determined that the data packet has an error, controlling the preprocessing module not to output the data packet.

6. The communication system according to claim 5, wherein The outputting the data packet to the neuromorphic chip includes: Outputting the data packet to the fault tolerance module; Under the control of the fault tolerance module, output the data packet to the neuromorphic chip.

7. A communication method, characterized in that, A preprocessing module applied to a communication system, the communication system includes an event camera, the preprocessing module and a neuromorphic chip, the preprocessing module is connected between the event camera and the neuromorphic chip, the method includes: Receive first data from the event camera; According to the first data, determine the storage location of the second data on the neuromorphic chip; Process the first data according to the storage location to obtain a data packet including the second data; Output the data packet to the neuromorphic chip; The event camera is configured to: generate events under first image parameters, the first data includes pixel point position information of the generated events, time information of the generated events, and event polarity information, and the pixel point position information of the generated events is pixel point position information under the first image parameters; The neuromorphic chip is configured to: obtain image data under second image parameters according to the data packet, the neuromorphic chip includes a plurality of processor cores, each processor core corresponds to a storage area for storing the second data, and each processor core is used to store the second data of at least one pixel point position under the second image parameters; The determining the storage location of the second data on the neuromorphic chip includes: mapping the pixel point position information of the generated events of the first data to the pixel point position information under the second image parameters; according to the pixel point position information under the second image parameters, determine the processor core on which the second data is stored on the neuromorphic chip and the storage address corresponding to the processor core; determine the storage location of the second data on the neuromorphic chip as the storage address corresponding to the processor core.

8. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the preprocessing module, the method described in claim 7 is implemented.

Images