Frame image spike conversion system

Abstract

To provide a self-adaptive spike generation method and a randomization method thereof, an apparatus, a brain chip, and an electronic device.SOLUTION: The method includes: preprocessing a frame image generated by a sensor in advance to reduce the number of events, and thereby performing frame image preprocessing to give sparsity to a generated target spike sequence; performing a frame differential operation on the frame image to obtain a frame differential pixel value for each pixel point; generating the number of spike events corresponding to pixel coordinates from the frame differential pixel value of each pixel point; determining whether a first condition is satisfied from at least the overall frame differential pixel value of the frame differential and / or the corresponding overall spike event number; if the first condition is satisfied, adjusting the frame image generation frame rate of the frame image sensor or adjusting the frame differential frequency of the frame differential operation; and randomizing the spike events.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This patent application claims priority from the invention patent applications with application numbers 202210381831.2 (Self - Adaptive Spike Generation Method, Apparatus, Brain - like Chip and Electronic Device) filed with the China National Intellectual Property Administration on April 13, 2022, and 202210387836.6 (Spike Sequence Randomization Method, Apparatus, Brain - like Chip and Electronic Device) filed with the China National Intellectual Property Administration on April 14, 2022, and incorporates and integrates all the content described therein into this patent application documents.

[0002] The present invention relates to a method, apparatus, brain - like chip and electronic device for converting a frame image into spike events, specifically to a method and apparatus for self - adaptive spike generation and its randomization, a brain - like chip, and an electronic device.

Background Art

[0003] Spiking Neural Network (SNN) can process complex, sparse, and noisy spatio - temporal information with excellent performance due to its rich neural dynamics characteristics and communication method of spike events. It is currently the neural network closest to the operation model of the human brain. Because of its ultra - low power consumption and high - level intelligence, it is a new research concern in the field of artificial intelligence.

[0004] Dynamic Vision Sensor (DVS) is a new type of sensor with an array of independent photosensitive pixels. When a pixel senses a change in light intensity, it independently and asynchronously transmits spike events to the subsequent system (such as a brain - like chip, etc.). Because of its ultra - high temporal resolution, it has attracted attention. However, at present, there is still a certain gap between its signal - to - noise ratio, low - light detection ability, consistency, and measurability and the actual application requirements.

[0005] Conventional image sensors based on frames (e.g., CMOS sensors, CCD sensors, etc., abbreviated as frame image sensors) are relatively mature image capture devices with a high signal-to-noise ratio. However, they cannot generate spike events (abbreviated as spikes or events, and several spikes constitute a spike sequence), and can only continuously generate multiple images in units of frames.

[0006] Generating a spike sequence (preferably a random spike sequence conforming to the Poisson distribution) that meets the requirements of a spike neural network processor (also called a brain-type chip) by a frame image sensor is an alternative to a DVS sensor. This alternative has the advantage that it can be directly integrated into an electronic device having a frame image sensor, and the existing device can be used as it is.

[0007] Therefore, it is required to meet the application needs of SNN processors with a spike generation technology that has a high signal-to-noise ratio, low power consumption, low cost, and is easy to implement. For the known literature on converting the images collected by a frame image sensor into spike events, specifically, the following can be referred to. Known Literature 1: CN111898737A Known Literature 2: CN111860786A Known Literature 3: EP3789909A1 Known Literature 4: CN112464807A

[0008] Public known documents 1 to 2 disclose means for converting numerical values / images into spike sequences. The core idea is to generate an initial spike sequence for a specific pixel value and randomly exchange or modify a certain number of spikes to obtain a target spike sequence. However, its drawback is that the number of randomly exchanged or modified spikes is necessarily finite, and the effect of randomization cannot be guaranteed (especially when the time step is long), and the cost for hardware implementation is relatively high. Also, obtaining contour information by the frame difference method is not mentioned. Only random spike sequence conversion is performed on each pixel value of a general frame image, and the recognition accuracy of the spike sequence obtained by such a method in a spiking neural network is extremely low.

[0009] Public known documents 3 to 4 disclose means for extracting regions of interest based on frame difference, which compare the differences between different images of two frames to generate a spike event sequence. However, there are problems such as high power consumption and insufficient real-time performance, and clear noise cannot be eliminated, so the network performance is not high. Also, means for randomizing the spike sequence are not shown, which has a very adverse effect on the inference accuracy of the SNN processor.

Summary of the Invention

Problems to be Solved by the Invention

[0010] The present invention provides an adaptive spike generation and its randomization method, apparatus, brain-type chip, and electronic device.

Means for Solving the Problems

[0011] The present invention attempts to solve or alleviate some or all of the above technical problems by the following technical means.

[0012] An adaptive spike generation method, comprising: comparing the differences between frame images to obtain the frame difference pixel values of each pixel coordinate point during frame difference; obtaining the number of spike events corresponding to the pixel coordinates from the frame difference pixel values; judging whether to adjust the generation frame rate or the frame difference frequency of the frame image from at least the overall frame difference pixel values of the frame difference or / and the corresponding overall number of spike events; including.

[0013] In one embodiment, when the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is smaller than a first threshold value, the frame rate or the frame difference frequency is decreased, and / or when it is larger than a second threshold value, the frame rate or the frame difference frequency is increased, or when the sum of the spike events corresponding to the frame difference, or the sum of the spike events satisfying a predetermined condition, or the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition is smaller than a first threshold value, the frame rate or the frame difference frequency is decreased, and / or when it is larger than a second threshold value, the frame rate or the frame difference frequency is increased.

[0014] In one embodiment, when the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is smaller than a first ratio, the frame rate or the frame difference frequency is decreased, and / or when it is larger than a second ratio, the frame rate or the frame difference frequency is increased, the first ratio is the ratio of the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition in the entire frame difference pixels, or when the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition is smaller than a first ratio, the frame rate or the frame difference frequency is decreased, and / or when it is larger than a second ratio, the frame rate or the frame difference frequency is increased, the second ratio is the ratio of the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition in the entire frame difference pixels.

[0015] In one embodiment, after reducing the frame rate or shortening the first time interval of the frame difference frequency, the frame rate or the frame difference frequency is restored to the default value, and / or within a second time interval, when the sum of the frame difference pixel values and / or the sum of the number of spike events are both smaller than a first threshold value, the frame rate or the frame difference frequency is reduced.

[0016] In one embodiment, the self-adaptive spike generation method includes: generating a randomized target spike sequence from the number of spike events; determining whether the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is greater than a third threshold value. If it is greater, the step of obtaining the number of spike events corresponding to the pixel coordinates from the frame difference pixel values is not further executed; or further including the step of determining whether the sum of the frame differences corresponding to the number of spike events, or the sum of the number of spike events satisfying a predetermined condition, or the count value of the number of frame difference pixels whose spike events satisfy a predetermined condition is greater than a third threshold value. If it is greater, the step of generating a randomized target spike sequence from the number of spike events is not further executed.

[0017] The self-adaptive spike generation device is used to generate a target spike sequence, and the self-adaptive spike generation device includes: a frame difference module that compares the differences between frame images and obtains the frame difference pixel values of each pixel coordinate point in the frame difference; a spike event number generation module that generates the number of spike events corresponding to the pixel coordinates from the frame difference pixel values; a randomization module that generates a randomized target spike sequence from the number of spike events; A first determination module that determines whether to adjust the generation frame rate or frame difference frequency of the frame image from at least the overall frame difference pixel value of the frame difference or / and the corresponding overall number of spike events; including.

[0018] In one embodiment, when the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is smaller than a first threshold value, the frame rate or the frame difference frequency is decreased, or / and when it is larger than a second threshold value, the frame rate or the frame difference frequency is increased, or, When the sum of the total number of spike events corresponding to the frame difference, or the sum of the spike events satisfying a predetermined condition, or the count value of the number of frame difference pixels whose spike event number satisfies a predetermined condition is smaller than a first threshold value, the frame rate or the frame difference frequency is decreased, or / and when it is larger than a second threshold value, the frame rate or the frame difference frequency is increased.

[0019] In one embodiment, the self - adaptive spike generation device further includes a second determination module that determines whether the sum of the frame difference pixel values and / or the sum of the spike event numbers is greater than a third threshold value, and when it is greater, ends the randomization operation for the spike events.

[0020] The spike sequence randomization method includes a step of obtaining frame difference pixel values between frame images and a step of calculating the number of spike events corresponding to the frame difference pixel values based on the frame difference pixel values, and the spike sequence randomization method further includes at least one of the following steps. i) Storing the pixel coordinates corresponding to the frame difference pixel values in all memory units where the number is the same as the number of spike events, reading the pixel coordinates stored in the memory unit corresponding to the currently acquired random number based on the currently acquired random number, and generating spike events in the target spike sequence based on at least the read pixel coordinate information; ii) Storing the number of spike events in the memory unit corresponding to the pixel coordinates of the frame difference pixel values, reading the number of spike events stored in the memory unit corresponding to the currently acquired random number based on the currently acquired random number, and when the read number of spike events is not zero, generating spike events in the target spike sequence based on at least the pixel coordinate information of the frame difference pixel value corresponding to the memory unit.

[0021] In one embodiment, the frame difference pixel value is the difference between the pixel values of two adjacent frame images or the absolute value of the difference between the pixel values of two adjacent frame images.

[0022] In one embodiment, the number of spike events corresponding to the frame difference pixel value is obtained based on one of the following methods. i) The ratio of the product of the frame difference pixel value and the maximum number of spike events that an allowed single pixel can emit to the theoretical maximum value of the pixels in the frame difference; ii) The ratio of the product of the frame difference pixel value and the maximum number of spike events that an allowed single pixel can emit to the maximum pixel value in the frame difference; iii) The ratio of the product of the frame difference pixel value and the maximum number of spike events that an allowed single pixel can emit to the larger of the maximum pixel value or the first lower limit value in the frame difference.

[0023] In one embodiment, after obtaining the frame difference pixel value between frame images, when the frame difference pixel value is smaller than the second lower limit value, the frame difference pixel value is set to zero.

[0024] In one embodiment, a random number generation module generates a random number sequence, and the currently acquired random number is derived from the random number sequence. The random number sequence does not generate the same random number within one cycle. The random number generation module includes at least one random number generator using a linear feedback shift register, or the cycle of the random number sequence is equal to the product of the number of frame difference pixels and the maximum number of spike events that an allowed single pixel can emit, or is equal to the number of frame difference pixels.

[0025] In one embodiment, when reading the number of spike events stored in the memory unit corresponding to the random number based on the currently acquired random number, the random number is derived from the random number sequence. The cycle of the random number sequence is equal to the number of frame difference pixels, and for each frame difference, the number of cycles that the random number sequence executes is equal to the maximum number of spike events that an allowed single pixel can emit.

[0026] The spike sequence randomization method includes the steps of acquiring a channel output signal and, based on the channel output signal, acquiring the number of spike events corresponding to the channel output signal, and further includes at least one of the following steps. i) Storing the channel address corresponding to the channel output signal in all memory units with the same number as the number of spike events, reading the channel address stored in the memory unit corresponding to the random number based on the currently acquired random number, and generating spike events in the target spike sequence based on at least the read channel address information. ii) Store the number of spike events in a memory unit corresponding to the channel address of the channel output signal, read the number of spike events stored in the memory unit corresponding to the current randomly acquired number based on the currently acquired random number, and if the read number of spike events is not zero, generate spike events in the target spike sequence based on at least the channel address information of the channel output signal corresponding to the memory unit.

[0027] In a spike sequence randomization device, A frame difference memory space and a spike event address memory space or a spike event number memory space, or A frame image memory space and a spike event number memory space, and including The spike sequence randomization device is configured to execute the spike sequence randomization method according to any one of the preceding items, The frame difference memory space stores frame difference pixel values, The memory units included in the spike event address memory space store the pixel coordinates corresponding to the frame difference pixel values, The memory units included in the spike event number memory space store the number of spike events corresponding to the frame difference pixel values, The memory units included in the frame image memory space store the latest pixel values acquired from the frame image sensor.

[0028] In a brain chip, the brain chip uses the self-adaptive spike generation method according to any one of the preceding items, or includes the self-adaptive spike generation device according to any one of the preceding items, or applies the spike sequence randomization method according to any one of the preceding items, or includes the aforementioned spike sequence randomization device.

[0029] In an electronic device, the electronic device includes the self-adaptive spike generation device according to any one of the above items, or includes the spike sequence randomization device described above, or includes the brain-like chip described above.

Advantages of the Invention

[0030] Some or all embodiments of the present invention have the following beneficial technical effects. 1) In the process of generating the target spike sequence, the present invention can self-adaptively adjust the frame rate or the frame difference frequency, and has high flexibility. It can also ensure low-power consumption operation in an environment with no change for a long time, and can quickly capture the change of motion when necessary. 2) The present invention can promptly end an unreasonable process (the total frame difference pixel value or the total number of spike events is too large), effectively remove noise, and at the same time save power consumption. By controlling the number of spikes in the target spike sequence, the present invention ensures that the SNN processor operates in real time, with high efficiency and stability. 3) The hardware of the present invention is user-friendly and realizes low cost. Compared with DVS, the existing camera in the electronic device can be used as it is, and it is not necessary to install a new imaging module and sensor. 4) Multiple steps of the present invention can be processed in parallel, saving resources and further reducing power consumption. 5) Considering macroscopically that the image becomes a randomized target spike sequence as a whole, the spiking neural network has a user-friendly randomization method, and there is no defect that the performance of the spiking neural network is significantly reduced by connecting the corresponding spike sequences of pixels one by one. Therefore, its randomization effect is high. 6) It consumes less hardware resources and less energy consumed in conversion. 7) The outline can be enhanced, and the network performance in a situation with dim light and weak shaking can be improved. 8) Suppress background noise and reduce power consumption during long-term standby. More beneficial effects will be further introduced in the preferred embodiments.

[0031] The purpose of disclosing the above technical means / features is to summarize the technical means and technical features described in the specific embodiment part. Therefore, the described scope may not be exactly the same. However, these new technical means disclosed in this part are also part of the many technical means disclosed in this patent document. The technical features disclosed in this part can be reasonably combined with the technical features disclosed in the subsequent specific embodiment part and some content in the drawings not clearly described in the specification to disclose more technical means.

[0032] The technical means combined with all the technical features disclosed at any position of the present invention are used to support the summary of the technical means, the modification of the patent document, and the disclosure of the technical means.

Brief Description of the Drawings

[0033]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Embodiments for Carrying Out the Invention

[0034] Since it is impossible to describe all kinds of alternatives, in the following, the drawings in the embodiments of the present invention are combined to clearly and completely describe the key points of the technical means in the embodiments of the present invention. For other technical means and details not disclosed in detail below, they generally fall within the technical goals or technical features that can be realized by ordinary means in this field. Due to the limitation of paper width, the details thereof are not introduced in the present invention.

[0035] Unless it is the meaning of division, " / " at any position in the present invention represents a logical OR (i.e., "or"). The numbers such as "first", "second", etc. at any position in the present invention are only used for descriptive classification notations, and do not imply an absolute order in time or space, nor do they imply that the terms with such numbers necessarily refer to different things from the same terms with other limitations.

[0036] The present invention describes key points for combination as various different specific embodiments, and these key points can be combined in various methods and products. In the present invention, even if it is a key point only described when introducing method / product means, it means that the corresponding product / method means also clearly includes the technical features.

[0037] When it is described that there is or includes a certain step, module, or feature at any position in the present invention, it does not imply that such existence is exclusive and the only existence. A person skilled in the art can completely supplement other technical means based on the technical means disclosed in the present invention to obtain other embodiments. Based on the key points described in the specific embodiments of the present invention, a person skilled in the art can completely obtain technical means that comply with the inventive concept of the present invention by adding means such as substitution, reduction, increase, combination, and order change to some technical features. These means that do not deviate from the inventive concept of the present invention are also within the protection scope of the present invention.

[0038] The frame image sensor of the present invention is a sensor capable of acquiring frame images such as CMOS sensors, CCD sensors, and grayscale sensors. The present invention is not limited to a specific type of image sensor, as long as it can acquire frame images.

[0039] FIG. 1 is a flowchart of a self-adaptive spike generation method in a preferred embodiment of the present invention, including the following steps.

[0040] Step S100: Frame image preprocessing.

[0041] The dataset of the spiking neural network consists of a spatio-temporal event stream and has sparse characteristics. On the other hand, the resolution of the conventional frame image sensor is high, and the number of spiking events generated after conversion may be large. Therefore, the frame image generated by the sensor may be pre-processed in advance to reduce the number of events and make the generated target spike sequence sparse.

[0042] Furthermore, as shown in FIG. 2, the pre-processing operation may include dimension reconstruction and grayscale conversion.

[0043] Step S101: Perform dimension reconstruction (reshape) on any frame image.

[0044] Let the resolution of the frame image sensor be (W×H), and the resolution of the frame image after reconstruction (also referred to as the target dimension) be (W’×H’), where W and W’ represent the width, and H and H’ represent the height.

[0045] In one embodiment, the method of dimension reconstruction is subsampling. The reconstruction of the frame image dimension by subsampling is easy to implement and suitable for hardware. Specifically, after adjusting any pixel coordinate I(x,y) in the original frame image, the pixel coordinate

Equation

Equation

[0046] In another embodiment, the method of dimension reconstruction is cropping. Based on the region of interest, the original frame image is cropped to obtain the target dimension. For example, any region of interest such as the upper left corner, the lower right corner, or the center of the original frame image is cropped. The cropping method of the present invention is not limited to this.

[0047] In other embodiments, the dimension reconstruction method may be any method such as bilinear interpolation, linear or non-linear zoom, taking values according to a ratio, etc. The present invention does not limit the dimension reconstruction method, and any means capable of arbitrarily adjusting the original frame image to the target dimension may be used.

[0048] Step S102: Grayscale the frame image.

[0049] Normally, a frame image is an RGB image and has three channels. The present invention converts an RGB image into a grayscale image, and there may be multiple grayscale conversion methods. The present invention does not limit a specific grayscale conversion method. For example, it may not be limited to extracting the value in any one of the three channels as the grayscale value, and the maximum value, average value or weighted average value of RGB may also be used as the grayscale value.

[0050] The present invention does not limit the order of performing the dimension reconstruction and grayscale conversion operations, and they may be performed in parallel or sequentially, or only one of the operations may be performed. Furthermore, the frame image preprocessing operation S100 of the present invention is not essential and is only performed when necessary.

[0051] Step S200: Perform a frame difference operation on the frame image to obtain the frame difference pixel value of each pixel point.

[0052] Specifically, at least two different frame images in the frame image sequence or the preprocessed frame image are compared to obtain a frame difference image of the image change between the two frames. The frame difference image reflects the change / motion information between frames.

[0053] In the frame difference process, the change value or the absolute value of the change value of the pixel value on each obtained pixel point (pixel coordinates), or a value obtained by performing operations such as noise filtering, is referred to as the frame difference pixel value. At a certain pixel point, the positive or negative of the frame difference pixel value may correspond to the polarity of the spike event. For example, if the frame difference pixel value is positive, it indicates pixel enhancement, and the polarity of the corresponding event is positive. If the frame difference pixel value is negative, it indicates pixel reduction, and the polarity of the corresponding event is negative. The reverse is also possible, and the present invention is not limited thereto.

[0054] In another several embodiments, when the change in the pixel value on a certain pixel point does not meet a predetermined condition (for example, being less than a certain threshold value, etc.), the frame difference pixel value may be discarded, that is, set to 0, which is useful for noise filtering.

[0055] For an image sequence including N frames, the difference operations may be performed sequentially or in parallel. When performed in parallel, power consumption and time can be extremely significantly saved, and the processing efficiency and real-time performance can be improved. Further, the difference between two adjacent frames may be performed sequentially or in parallel, and the difference between two frames separated by several frames may also be performed sequentially or in parallel. The present invention is not limited thereto.

[0056] In addition, the image sequence may be an original frame image set or a pre-processed frame image set collected by the sensor. Further, the order of step S100 and step S200 may be changed. Frame difference may be performed after pre-processing the frame image, or pre-processing operations may be performed after performing frame difference first.

[0057] Step S300: Generate the number of spike events corresponding to the pixel coordinates from the frame difference pixel values of each pixel point.

[0058] Based on the frame difference pixel values of each pixel point, the number of spike events corresponding to each pixel coordinate point is obtained. Any reasonable method for converting from pixel values to the number of spike events is executable, and the present invention does not limit which method is adopted to obtain the number of spike events corresponding to each pixel point after frame difference.

[0059] For example, the pixel value after frame difference is m, which generates the number of spike events after rounding up or rounding down, and the number of spike events is the value after rounding down of |m|

Number

Number

[0060] Step S400: Determine whether the first condition is satisfied from at least the overall frame difference pixel value of the frame difference or / and the corresponding overall number of spike events. If satisfied, adjust the frame image generation frame rate (abbreviated as frame rate) of the frame image sensor, or adjust the frame difference frequency in step S200. The specific adjustment method may be to adjust the frame rate / frame difference frequency upward or downward. If the condition is not satisfied, as shown in Figure 3, do not adjust the frame rate or the frame difference frequency.

[0061] For one frame difference, it includes several pixels, and each of these pixels corresponds to one frame difference pixel value or the number of spike events. For one frame difference, the overall frame difference pixel value or / and the corresponding overall number of spike events is the set constituted by the frame difference pixel values or the number of spike events corresponding to these several pixels. From the whole set, it is determined whether to adjust the frame rate or the frame difference frequency, and various conversion, aggregation, calculation and other methods are adopted for the set to obtain a value with a certain meaning (for example, the degree of intensity characterizing the movement of an object in front of the sensor, etc.), and from this value, it is determined whether to trigger the above adjustment operation.

[0062] Furthermore, for the embodiment including the preprocessing step S100, it may be adjusted by controlling step S100 or / and S200 or / and S300.

[0063] Furthermore, processes such as obtaining the sum of the number of spike events or the frame difference pixel values corresponding to each pixel position amount are performed. (Specifically, refer to step S200 or S300.) The process of obtaining the sum may be executed sequentially or in parallel. In hardware, it is easy to execute in parallel and has advantages in both power consumption and real-time performance.

[0064] The first condition includes one or more of the following cases.

[0065] When the sum of frame difference pixel values, or the sum of frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is less than a first threshold value, the frame rate or the frame difference frequency is decreased, and / or when it is greater than a second threshold value, the frame rate or the frame difference frequency is increased. Or, when the sum of the frame differences corresponding to the spike event numbers, or the sum of the spike event numbers satisfying a predetermined condition, or the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition is less than a first threshold value, the frame rate or the frame difference frequency is decreased, and / or when it is greater than a second threshold value, the frame rate or the frame difference frequency is increased.

[0066] Here, the sum of the frame difference pixel values / the sum of the spike event numbers corresponding to the frame differences is the sum obtained by directly summing each pixel value in the frame difference or the spike event numbers corresponding to each pixel. The sum of the frame difference pixel values satisfying a predetermined condition / the sum of the spike event numbers satisfying a predetermined condition is obtained by setting one condition (for example, pixel value ≧ 10, spike event number ≧ 2, etc.) and accumulating only the pixel values / spike event numbers that satisfy the condition to obtain the sum. The count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition / the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition is obtained by setting a condition (for example, pixel value ≧ 1 or 10, spike event number ≧ 1 or 2, etc.) and accumulating and counting only the pixels (coordinates) that satisfy the condition.

[0067] Here, in the case of downward adjustment, for example, when there is no change over a long period of time or the change is unclear in two frame images of frame difference, the sum of the frame difference pixel values of all pixels is small, and the sum of the corresponding spike generation numbers is small. Therefore, it is considered that there is no operation waiting for identification or the presence of a trigger, and the frame rate or the frame difference frequency is decreased to reduce power consumption. For example, the frame rate is decreased from 30 to 1. The decrease in the frame rate or the frame difference frequency may be stepwise. For example, it may be from 60 to 30 and then to 1. Correspondingly, at this time, the first threshold value includes a plurality of numerical values.

[0068] In the case of upward adjustment, for example, when the current frame rate is too low and cannot satisfy image collection for a rapidly moving object, but it is detected that the target object movement has started, by increasing the frame rate, the situation where the movement becomes ambiguous in the image after frame difference is reduced. For example, the frame rate is increased from 1 to 30. Similarly, the increase in the frame rate or the frame difference frequency may be stepwise. For example, it may be from 1 to 30 and then to 60. Correspondingly, at this time, the second threshold value includes a plurality of numerical values.

[0069] Here, the first threshold value and the second threshold value may be different or the same. In some embodiments, it is preferable to refer to the state of the current frame rate or the frame difference frequency before determining upward adjustment or downward adjustment.

[0070] In another embodiment, after shortening the first time interval of the frame rate or the frame difference frequency, the frame rate or the frame difference frequency is restored to the default value. Referring to the previous example, the default value may be 30 / 60 and 10 described above. This embodiment may allow a certain ratio of time width and increase the detection response sensitivity.

[0071] In another embodiment, the widths of the upward adjustment and the downward adjustment may be adjusted in proportion to the difference after comparison with a threshold value. For example, both the first threshold value and the second threshold value refer to the threshold value of the total number of spike events, and both are 1000. When the total number of spike events generated after frame difference is 100, the frame rate is adjusted from 60 to 6. When the total number of spike events generated after frame difference is 300, the frame rate is adjusted from 60 to 18, and so on.

[0072] Also, the first condition may be further extended as follows, that is, within a set time, for various frame differences, the sum of frame difference pixel values / sum of frame difference pixel values satisfying a predetermined condition, the count value of the number of frame difference pixels / sum of corresponding spike events satisfying a predetermined condition, the sum of spike events / sum of spike events satisfying a predetermined condition, and the count value of the number of frame difference pixels satisfying a predetermined condition are all smaller than the first threshold value, then it is considered to satisfy the condition, and the frame rate or the frame difference frequency is reduced. In such a case, it helps to prevent the response from being delayed due to the reduction of the frame rate or the frame difference frequency immediately when there is no movement of the target object, while reducing power consumption in a situation where there is no movement of the target object. Preferably, when adjusting upward, no similar time is set, or even if time is set, the time is extremely short, because setting such a time artificially delays the response when an immediate response is required.

[0073] As an equivalent substitution, in the foregoing embodiment, the judgment between the count value of the number of frame difference pixels satisfying a predetermined condition / the count value of the number of frame difference pixels satisfying a predetermined condition and the first threshold value / second threshold value is replaced by the judgment between the ratio of the count value in the total number of frame difference pixels and the first threshold value / second threshold value. Since the number of frame difference pixels is a constant, the difference lies only in whether the first threshold value / second threshold value is divided by the constant. Such an equivalent substitution is also within the scope protected by the present invention.

[0074] Preferably, the first condition is to reduce power consumption, especially when there is no moving object in the field of view of the frame image sensor for a long time, and at the same time, it also takes into account increasing the frame rate or frame difference frequency to meet the user's instantaneous response requirements. For this reason, it is possible to combine technical means such as the above-mentioned time, number of spike events, frame difference pixel values, ratios, and stepwise thresholds. All such easily conceivable combinations and conversions do not deviate from the concept of the present invention and should be within the scope protected by the present invention.

[0075] Step S500: Randomization of spike events.

[0076] For a spiking neural network (SNN) processor, its input data is a spatio-temporal event stream / spike sequence, and each event includes the coordinates and timestamps generated, such as an AER (Address Event Representation) event stream, SAER (Serial AER) event stream, etc. The SNN processor is suitable for receiving a random spike sequence that conforms to the Poisson distribution. Any reasonable randomization method is executable (see References 1 - 2 in the public domain), and the present invention is not limited thereto.

[0077] Due to the sparsity of the spatio-temporal event stream, the present invention randomizes the spike events to match the information processing characteristics of the SNN processor. In one embodiment, all the spike events corresponding to all pixels are randomized as a whole to obtain a target spike sequence. In another embodiment, the spike events corresponding to each pixel point are randomized respectively, and then a final target spike sequence is obtained. Also, the randomization process of the spike events on each pixel point may be executed in parallel or sequentially, and parallel means with the advantage of low latency is preferred. Hereinafter, the present invention further provides specific spike sequence randomization means considered based on the entire image.

[0078] Steps S400 and S500 of the present invention may be executed in parallel or in sequence.

[0079] As shown in FIG. 4, in one preferred embodiment, before the spike randomization step S500, it is determined whether each pixel coordinate satisfies a second condition based on the corresponding frame difference pixel value and / or the number of spike events. If it is satisfied, step S600 of ending the current process is further included. The ending method includes frame discarding (for example, resetting / clearing the current spike event set), or not performing the randomization step, and the present invention is not limited thereto.

[0080] Specifically, the second condition is to determine whether the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is greater than a third threshold. If it is greater, the step of obtaining the number of spike events corresponding to the pixel coordinates from the frame difference pixel values is no longer executed. Or, it is determined whether the sum of the frame differences corresponding to the number of spike events, or the sum of the number of spike events satisfying a predetermined condition, or the count value of the number of frame difference pixels whose spike event number satisfies a predetermined condition is greater than a third threshold. If it is greater, the step of generating a randomized target spike sequence from the number of spike events is no longer executed.

[0081] When the frame image sensor is shaken or tampered with violently, within a short time, the sum of the number of spike events corresponding to the frame difference / the sum of the frame difference pixel values becomes too large. As a result, the generated output spike sequence may not be valid input information for the subsequent SNN processor, and processing it may cause unnecessary result output. At the same time, because the number of spike events is too large, the processor is likely to stop moving or the power consumption increases. Therefore, before the randomization step, the frame image with too fast changes is filtered and removed in advance to ensure the processor performance and at the same time reduce the power consumption.

[0082] FIG. 5 is a self - adaptive spike generation device according to an embodiment of the present invention. The present invention further discloses a self - adaptive spike generation device for generating a target spike sequence, and the self - adaptive spike generation device includes: a frame difference module that compares differences between frame images and obtains frame difference pixel values for each pixel coordinate point in the frame difference; a spike event number generation module that generates the number of spike events corresponding to the pixel coordinates from the frame difference pixel values; a randomization module that generates a randomized target spike sequence from the number of spike events; a first determination module that determines whether to adjust the generation frame rate or frame difference frequency of the frame image based on at least the overall frame difference pixel values of the frame difference or / and the corresponding overall number of spike events; and includes.

[0083] In some embodiments, the self - adaptive spike generation device further includes a pre - processing module for pre - processing the frame image input to the frame difference module. The pre - processing operation may include dimension reconstruction or / and grayscale conversion.

[0084] In some embodiments, the self - adaptive spike generation device further includes a second determination module that determines whether each pixel coordinate meets a second condition based on the corresponding frame difference pixel value or / and the number of spike events, and if it is met, ends the current process.

[0085] For more adjustment solutions, reference may be made to the foregoing method - based embodiments, and they are incorporated herein by reference in the form of examples, and will not be described further herein.

[0086] FIG. 6 is a block diagram of the self - adaptive spike generation and processing system of the present invention, including a frame image sensor, a self - adaptive spike generation device, and a neural network processor that are connected in sequence. The frame image sensor collects frame images, the self - adaptive spike generation device is used for rapid generation of low - power - consumption and high - quality target spike sequences, and the neural network processor (such as an SNN processor) is used for inference and outputting classification results.

[0087] Here, the self - adaptive spike generation device may be installed independently or integrated into a processor or a frame image sensor. In some embodiments, the self - adaptive spike generation device is implemented as a frame image sensor interface. Further, the frame image sensor, the self - adaptive spike generation device, and the neural network processor are integrated on the same chip, and the chip simultaneously has sensing and computing capabilities.

[0088] FIG. 7 is another block diagram of the self - adaptive spike generation and processing system of the present invention, including a frame image sensor, a self - adaptive spike generation device, an event sensor (such as an event camera), and a neural network processor. The self - adaptive spike generation device converts the frame image collected by the frame image sensor into a first target spike sequence, the event camera outputs a second target spike sequence, and the neural network processor receives and processes the first target spike sequence or / and the second target spike sequence. Similarly, the frame image sensor or the event camera may be installed independently or integrated on the same chip together with the processor.

[0089] FIG. 8 is a block diagram of spike sequence randomization in the present invention. The frame image sensor continuously outputs image information captured from the environment in the form of frames, for example, outputs 30, 60, or 100 frames of images per second to the memory of the frame image sensor. For example, there are common RGB cameras and the like. The frame rate and the frame image sensor can be flexibly selected according to actual usage requirements, and the present invention is not limited to a specific form.

[0090] Since the frame images have sequence properties, the difference is taken between one adjacent frame or a plurality of adjacent frames before and after, and the difference image information between the two frame images, that is, the frame difference (Difference of Frames, DoF) is obtained. The frame difference may be the absolute value excluding the pixel difference and may leave valid sign (polarity) information. For example, if you wave the palm in front of the sensor, one frame image displays an image of the palm including the complete palm print, while the frame difference displays the contour information of the palm, not the palm print. This is similar to the result of the moving object contour captured by DVS.

[0091] Before performing the frame difference, steps such as an image subsampling operation, grayscale information extraction, region of interest extraction, etc. may be included, and then the spike sequence randomization step is performed. More frame difference means can refer to the known literature 3. Here, the image subsampling operation helps to directly use the existing cameras in the electronic device, and it is not necessarily required to attach a new imaging module and sensor, reducing costs.

[0092] FIG. 9 is a diagram showing spike sequence randomization in the first type of embodiment. The frame difference memory space stores frame difference pixel values. The memory unit included in the spike event address memory space stores the pixel coordinates corresponding to the frame difference pixel values. The number of times the spike event address (i.e., the pixel coordinates corresponding to the frame difference pixel value) is stored in the spike event address memory space is determined from the frame difference pixel values. The random number generation module generates a random number sequence, and based on the random number sequence, reads the spike event address in the spike event address memory space, that is, determines the address (x, y) of the spike event that the current time / timestamp t is about to emit, or further combines the polarity information p, that is, obtains a spike event including (x, y, p, t) information.

[0093] The spike event address is continuously read by the random number sequence to generate a target spike sequence consisting of spike events, thus completing the randomization process of the spike sequence. The spike sequence is sent into the subsequent system for processing. Preferably, the subsequent system here is an SNN processor, and of course, it may be other types of signal processing devices.

[0094] Regarding the SNN processor, when converting a frame difference or a general frame image into a random target spike sequence, it is converted into a short random spike sequence for each pixel, and then these short random spike sequences are connected in sequence and sent into the SNN processor for inference. After that, the obtained network performance is extremely low. This is because it does not conform to the spike neural network information processing characteristics at all. Known Document 1 does not show how to randomize an entire image, but only shows how to generate a randomized target spike sequence for a single pixel. On the other hand, the present invention discloses how to randomize the whole (specifically, the frame difference) of the image into a target spike sequence.

[0095] FIG. 10 is a diagram showing details of spike sequence randomization in the first type of embodiment. For example, if the image size used for frame difference is 128×128 (the product of horizontal and vertical resolutions, denoted as the number of frame difference pixels) and the width of each pixel value is 8 bits, the size of the frame difference memory space is at least 128×128×8 bits. Since it calculates the pixel difference between frame images separated by several frames (preferably, two adjacent frames), most of the pixel values of the frame difference are 0, and usually, only the pixels at the contour of the moving object are not zero. Without loss of generality, let a certain pixel value in the frame difference be denoted as I(x,y), and the maximum pixel value in the frame difference be max(I(x,y)). In the formula, the lowercase (x, y) represents the position of a certain pixel in the frame image sensor, and the uppercase (x,y) represents the set of coordinates of all pixels in the frame image sensor.

[0096] After obtaining the frame difference of the frame image sensor, at least from each pixel value in the frame difference, the spike event address corresponding to the pixel (i.e., the pixel coordinates of the pixel) is generated. As shown in FIG. 10, the pixel value I(0,2)=97 where the coordinates are (x,y)=(0,2), and I(2,2)=63, and the maximum pixel value in the frame difference is max(I(x,y))=158.

[0097] To code these pixel values in the frame difference memory space, it is necessary to obtain the corresponding number of spike events based on these pixel values. Also, let the number of spike events corresponding to the pixel at the coordinates (x,y) in the frame difference be denoted as R(x,y).

[0098] To obtain the number of spike events R(x,y), various means may be taken. In one embodiment,

Equation

Equation

[0099] In one embodiment,

Number

[0100] In one embodiment,

Number

[0101] In an alternative embodiment, all of the foregoing rounding is replaced by rounding up.

[0102] In an alternative embodiment, when the pixel value in the frame difference is less than a second lower limit value, the pixel value in the frame difference is directly set to zero, and preferably, this step is executed in the step of calculating the difference. For example, if the pixel value corresponding to the pixel with coordinates (0, 3) in FIG. 10 is 1 and is less than the second lower limit value of 5, the pixel value is set to zero. In this way, filtering of background noise can be realized.

[0103] In one embodiment, in the step of obtaining the frame difference, since the extreme information is retained, in the calculation of R(x, y), it is also necessary to consider meeting the realistic and objective requirements by taking the absolute value.

[0104] In most cases in the frame difference, that is, for pixels with a pixel value of 0, the corresponding number of spike events is usually all 0, which ensures the sparsity of the target spike sequence.

[0105] After obtaining the corresponding number of spike events R(x, y) for the pixel value in the frame difference, R(x, y) spike event addresses (x, y) are stored in the spike event address memory space. According to the length of the spike event address, the bit width of the memory space may be determined. For example, if the image size of the frame difference is 128×128, the memory space bit width is 14 bits. The address corresponding to the pixel in the frame difference is stored in the spike event address memory space in any reasonable manner. For example, it may be stored in the row / column order / reverse order, and the same address may be stored in the correct order / incorrect order / shifted order. The present invention is not limited to a specific form.

[0106] Since the maximum number of spike events emitted by an allowable single pixel is N (for example, 8), the theoretically required maximum length of the spike event address memory space is 128×128×N when taking the frame difference resolution of 128×128 as an example. However, frame difference is only the manifestation of the object's contour, and the frame difference data is sparse. Usually, unless there is an instantaneous switching between a pure black and a pure white environment, the maximum length is not fully utilized when actually storing addresses. Here, it is reasonable to assume that the actual length of the memory space is the number of frame difference pixels × K (K < N), for example, K = 3. Therefore, the spike event address memory space occupies the number of frame difference pixels × K × the memory space bit width, which is 128×128×K×14bit using the previous example.

[0107] The random number generation module generates random integers within the numerical range of 1 to the number of frame difference pixels × N (essentially pseudo-random numbers), or random numbers within another numerical range that corresponds one-to-one with the integers within this numerical range. For example, random numbers within the range of 0 to the number of frame difference pixels × N - 1, etc., to form a random number sequence. Also, after generating these numerical values within one cycle, these numerical values are cyclically generated within the next cycle. The purpose of generating these random numbers is to enable some of these random numbers to be mapped one by one to the numbers of each memory unit in the spike event address memory space. That is, for each generated random number, based on the mapping relationship between the random number and the number in the spike event address memory space, an address stored in the memory unit corresponding to the number in the spike event address memory space can be read once. For example, if the random number sequence is 9 - 5 - 2 - 7…, the addresses with numbers 9 - 5 - 2 - 7… in the spike event address memory space are read in sequence.

[0108] Preferably, the generation of the random number sequence is realized by a linear feedback shift register (LFSR) and an XOR gate. For example, by configuring it as a circuit that expresses a primitive polynomial, the generation of pseudo-random numbers is realized. This technology belongs to the technologies known in this field and will not be described further here.

[0109] The range of the pseudo-random numbers is from 1 to the number of frame difference pixels × N, and the number Kr (<K, which varies dynamically depending on the difference in frame differences) of the memory units (the memory units with the coordinates filled in in FIG. 10) in which the valid addresses due to the change in pixel values during frame difference are stored is not longer than the actual length of the spike event address memory space (the number of frame difference pixels × K), where K < N. Therefore, when the random number is mapped outside the range of 1 to Kr, the corresponding time / time step does not emit a spike event (also called non-spike). Therefore, the target spike sequence is usually a sparse interleaving spike sequence.

[0110] There may be more than one random number generation module. As described above, the first random number generation module generates pseudo-random numbers within a predetermined range, and the second random number generation module also generates pseudo-random numbers within a predetermined range. Both have the same predetermined range. When in use, both generate the target spike sequence for different frame differences in a certain order (the simplest one is alternately). For three or more random number generation modules, a similar method may be adopted for processing.

[0111] For example, a simple mapping means is that since 1 to Kr in the random numbers (from 1 to the number of frame difference pixels × N) and the numbers 1 to Kr of the memory units in the spike event address memory space are equivalent, they are corresponded one by one. Any reasonable and executable range of random number values and its mapping relationship with the memory unit can all be applied in the present invention and are not limited to a specific method here.

[0112] Preferably, since the target spike sequence is sent to and processed by the SNN processor, the present invention imparts the SNN processor with the ability to process environmental signals captured by the frame image sensor.

[0113] FIG. 11 shows a schematic diagram of spike sequence randomization in another embodiment of the present invention. Different from the means shown in FIG. 9, it is not to determine the number of times to store the spike event address from the pixel values in the frame difference memory space, but to generate the corresponding number of spike events R(x,y).

[0114] FIG. 12 shows a diagram illustrating the details of spike sequence randomization in another embodiment. In such an embodiment, for the sake of simplicity of description, the technical features and various technical symbols described in the previous embodiment are cited here.

[0115] After obtaining the frame difference pixel values and storing them in the frame difference memory space, the corresponding number of spike events R(x,y) is generated and stored in the corresponding memory unit of the spike event number memory space. The number of memory units in the spike event number memory space is the same as the number of pixels of the frame difference. By referring to the previous example, all are 128×128.

[0116] Similarly, a random number sequence is generated by the random number generation module. Based on the current random number, the corresponding memory unit is read. If R(x,y)>1 stored in the memory unit, the corresponding spike event, for example, a spike event including at least (x,y,t) information is emitted. t is a time stamp.

[0117] Also, such an embodiment further updates the corresponding R(x,y) value. For example, it subtracts 1 from itself. As shown in FIG. 12, the number of spike events of the frame difference pixel at the address (x,y)=(2,2) is updated from 3 to 2.

[0118] In this embodiment, the cycle of the random number sequence is equal to the number of frame-differenced pixels. In one cycle of the pseudo-random sequence, all non-zero pixels of R(x, y) emit one spike event. For one frame difference, the pseudo-random sequence circulates N times (the maximum number of spike events that an allowed single pixel can emit) until all are emitted, causing spike events. Such a means destroys the randomness of spike emission to a certain extent, but it has been found through actual measurement tests that there is no obvious weakness in the performance shown by the SNN. Compared with the previous embodiment, such an embodiment further reduces the occupancy of the memory space.

[0119] FIG. 13 is a diagram showing spike sequence randomization in a further embodiment. What is different from the previous embodiment is that after obtaining the latest pixel value from the frame image sensor, it is directly differenced from the pixel value of the corresponding coordinate stored in the frame image memory space to obtain the pixel value of the corresponding coordinate in the frame difference, and based on this pixel value, the corresponding number of spike events R(x, y) is calculated, and the number of spike events is stored in the corresponding memory unit in the spike event number memory space. In other words, in this further embodiment, since there is no frame difference memory space dedicated to storing frame difference pixel values, when calculating the aforementioned spike event R(x, y), max(I(x, y)) cannot be obtained, and therefore, it is only applicable to embodiments other than the contour enhancement type.

[0120] Also, the latest pixel value is stored in the corresponding memory unit in the frame image memory space. For example, in FIG. 13, the pixel value of the pixel with coordinates (x, y) obtained from the sensor is 105, and the pixel value of the corresponding coordinates previously stored in the frame image memory space is 1. After performing the difference, it is calculated that R(x, y) = 3. That is, the number of spike events 3 is stored in the corresponding memory unit in the spike event number memory space, and the pixel value of the corresponding coordinates stored in the frame image memory space is updated to 105. Other technical means are the same as those in the previous embodiment and are described herein by way of citation and will not be described further here.

[0121] Furthermore, the present invention further discloses a spike sequence randomization device, and the spike sequence randomization device includes a frame difference memory space and a spike event address memory space or a spike event number memory space, or a frame image memory space and a spike event number memory space and includes In addition, the spike sequence randomization device is configured to execute the aforementioned spike sequence randomization method (refer to FIGS. 8 to 13 and the corresponding description). Here, the frame difference memory space is used to store frame difference pixel values, the memory units included in the spike event address memory space are used to store the pixel coordinates corresponding to the frame difference pixel values, the memory units included in the spike event number memory space are used to store the spike event numbers corresponding to the frame difference pixel values, and the memory units included in the frame image memory space are used to store the latest pixel values obtained from the frame image sensor.

[0122] In one embodiment, the spike sequence randomization device is implemented as a frame image sensor interface.

[0123] FIG. 14 is a diagram showing chip-integrated sensor means. The sensor includes a DVS or / and a frame image sensor, and through a dynamic vision sensor interface or / and a frame image sensor interface, it sends the spike event sequence generated by the DVS or / and the generated randomized target spike sequence to the SNN processor for processing. The DVS or / and the frame image sensor and the SNN processor are located on different bare dies and integrated on the same chip, which may be called a brain-type chip and is a brain-type chip integrating detection and operation.

[0124] FIG. 15 is a diagram showing chip-external sensor integration means. The sensor includes a DVS or / and a frame image sensor, and through a dynamic vision sensor interface or / and a frame image sensor interface, it sends the spike event sequence generated by the DVS or / and the generated randomized target spike sequence to the SNN processor for processing. The DVS or / and the frame image sensor and the SNN processor establish a communication connection via a cable, for example, based on the USB protocol.

[0125] In one embodiment, the chip supports a dual-eye sensor. Preferably, one eye is a DVS and the other eye is a frame image sensor. Preferably, one eye is an in-chip integrated sensor and the other eye is an out-of-chip sensor. These in-chip / out-of-chip sensor types may be flexibly and freely combined as different embodiments.

[0126] FIG. 16 is a diagram showing details of spike sequence randomization in a further embodiment. In this embodiment, without being further limited to the above-mentioned visual signal collection, it relates to environmental signals such as audio, vibration, electrocardiogram signals, etc., and the present invention is not limited thereto.

[0127] The collected environmental signals are subjected to various software / hardware settings or processing. For example, audio signals are passed through bandpass filters of multiple different bands to obtain multiple channel output signals. These channel identifiers are also called channel addresses, such as CH-1, CH-6, etc., and correspond to the aforementioned pixel coordinates / spike event addresses.

[0128] Subsequently, each channel obtains the corresponding number of spike events and stores them in the corresponding memory unit in the spike event number memory space. Here, obtaining the corresponding number of spike events for each channel means mapping the channel output signal as the corresponding number of spike events, which can be any reasonable form of means, such as an increment value based on the channel output signal, various spike sparsification means, etc. The present invention is not limited to a specific means.

[0129] Please refer to the foregoing embodiments. According to the random number generated by the random number generation module, the memory unit corresponding to the random number is read. If a non-zero value is found, at least based on the channel address corresponding to the memory unit, a spike event is generated, and the number of spike events stored in the memory unit is updated, for example, by subtracting 1, etc. Several such spike events constitute a target spike sequence, and the target spike sequence is sent into the SNN processor. Of course, for the 0 value in the memory unit, there is no need to generate and emit spike events. Regarding other technical features, the technical features of the foregoing embodiments are cited here in the form of citation, and will not be described further here.

[0130] FIG. 17 is a diagram showing details of spike sequence randomization in such an embodiment. The difference from the embodiment shown in FIG. 16 is that after each channel obtains the corresponding number of spike events, the channel address is written into the same number of memory units in the spike event address memory space. As shown in FIG. 17, since the number of spike events of CH-1 is 2, the address is written twice, and for CH-2, since the corresponding number of spike events is 1, it is written once. Then, one random number is obtained from the random number generation module, and in the spike event address memory space, one address in the memory unit corresponding to the random number is read, and at least from the read channel address, a spike event is generated and emitted. However, when the mapping result of the random number exceeds the spike event address memory space, or even exceeds the range of the memory units in which the valid channel addresses are written, there is no need to generate and emit spike events. Some such spike events constitute the target spike sequence and send the target spike sequence into the SNN processor. For other technical features, the technical features of the foregoing embodiments are cited herein in the form of citation, and will not be described further herein.

[0131] The present invention discloses a chip, which is a brain-type chip or a neuromorphic chip. The chip includes the foregoing frame image sensor interface, or simultaneously includes a frame image sensor interface and an event sensor interface. The chip uses the self-adaptive spike generation method described in any one of the foregoing items, or includes the self-adaptive spike generation device described in any one of the foregoing items.

[0132] The present invention further discloses an electronic device, such as an electronic device like a smart home appliance or a smartphone, which includes the above-mentioned brain-type chip or spike sequence randomization device and is used to perform real-time monitoring and smart response to environmental signals. Due to the ultra-low power consumption characteristics of the brain-type chip, etc., it is possible to realize always-on smart inference. By configuring one or more specific inference capabilities in the SNN processor, the electronic device side can be endowed with smart information processing capabilities.

[0133] In addition, the present invention further discloses an electronic device including the aforementioned chip, which is used for low power consumption, real-time, high performance and responding to environmental signals. The power consumption of the means of the present invention is extremely low, suitable for edge smart computing, and applicable to fields such as smart home, IoT, autonomous driving, smart toys, etc.

[0134] The present invention has been described with reference to specific features and examples of the present invention, but various changes, combinations, and substitutions can be made without departing from the present invention. The protection scope of the present invention is not limited to the specific examples of the processes, devices, manufacturing, substance compositions, devices, methods, and steps described in the specification, and these methods and modules are intended to be related, interdependent, and cooperate with each other, and may be further implemented within one or more products and methods in the front / back stages.

[0135] Therefore, the specification and drawings should be regarded as merely an introduction to some embodiments of the technical means limited by the appended claims. Therefore, based on the principle of maximum reasonable interpretation, the appended claims should be interpreted, and it is intended to cover all changes, variations, combinations, or equivalents within the scope of the present invention disclosure as much as possible, and at the same time, a decoding method that does not conform to common sense should be avoided.

[0136] To achieve better technical effects or due to the requirements of some applications, those skilled in the art may further modify technical means based on the present invention. However, even if some of these improvements / designs have creativity and / or progressiveness, as long as they rely on the technical concept of the present invention and cover the technical features defined by the claims, such technical means should likewise fall within the protection scope of the present invention.

[0137] Among some of the technical features recited in the appended claims, there may exist alternative technical features, or the order of some technical processes and the order of material compositions can be reconfigured. After knowing the present invention, those skilled in the art can easily conceive of some of these substitution means, or change the order of technical processes and the order of material compositions, and then use substantially the same means to solve substantially the same technical problems and achieve substantially the same technical effects. Therefore, even if the above means and / or order are clearly defined in the claims, all such modifications, changes, and substitutions shall fall within the protection scope of the claims according to the principle of equivalence.

[0138] If each method step or module described in the embodiments disclosed in this specification is combined, it can be realized by hardware, software, or a combination of both. To clearly explain the compatibility between hardware and software, the steps and compositions of each embodiment have been generally described according to functions in the above description. Whether these functions are ultimately executed in the form of hardware or software is determined by the specific application of technical means or design constraints. Those skilled in the art can realize the functions described using different methods for each specific application, but such realization should not be considered outside the scope of protection sought by the present invention.

Claims

**Claim 1** An adaptive spike generation method, comprising: comparing differences between frame images to obtain frame difference pixel values of each pixel coordinate point in the frame difference; obtaining the number of spike events corresponding to the pixel coordinates from the frame difference pixel values; judging whether to adjust the generation frame rate or the frame difference frequency of the frame image based on at least the overall frame difference pixel values of the frame difference or / and the corresponding overall number of spike events; An adaptive spike generation method, characterized by comprising the above steps. **Claim 2** When the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is smaller than a first threshold, reducing the frame rate or the frame difference frequency, or / and when it is larger than a second threshold, increasing the frame rate or the frame difference frequency, or, When the sum of the spike events corresponding to the frame difference, or the sum of the spike events satisfying a predetermined condition, or the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition is smaller than a first threshold, reducing the frame rate or the frame difference frequency, or / and when it is larger than a second threshold, increasing the frame rate or the frame difference frequency. The adaptive spike generation method according to claim 1, characterized by the above. **Claim 3** When the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is smaller than a first ratio, reducing the frame rate or the frame difference frequency, or / and when it is larger than a second ratio, increasing the frame rate or the frame difference frequency. The first ratio is the ratio of the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition in the whole frame difference pixels, or, When the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition is smaller than a first ratio, reducing the frame rate or the frame difference frequency, or / and when it is larger than a second ratio, increasing the frame rate or the frame difference frequency. The second ratio is the ratio of the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition in the whole frame difference pixels. The adaptive spike generation method according to claim 1, characterized by the above. **Claim 4** After reducing the frame rate or shortening the first time interval of the frame difference frequency, restoring the frame rate or frame difference frequency to the default value, and / or when the sum of the frame difference pixel values and / or the sum of the number of spike events are both smaller than the first threshold within the second time interval, reducing the frame rate or frame difference frequency, the self-adaptive spike generation method according to claim 1, characterized in that.

5. Generating a randomized target spike sequence from the number of spike events; Determining whether the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is greater than a third threshold, and if it is greater, not executing the step of obtaining the number of spike events corresponding to the pixel coordinates from the frame difference pixel values any further, or Determining whether the sum of the frame differences corresponding to the total number of spike events, or the sum of the number of spike events satisfying a predetermined condition, or the count value of the number of frame difference pixels whose spike events satisfy a predetermined condition is greater than a third threshold, and if it is greater, not executing the step of generating a randomized target spike sequence from the number of spike events any further The self-adaptive spike generation method according to claim 1, further comprising the above.

6. A self-adaptive spike generation device for generating a target spike sequence, comprising: A frame difference module that compares the differences between frame images and obtains the frame difference pixel values of each pixel coordinate point in the frame difference; A spike event number generation module that generates the number of spike events corresponding to the pixel coordinates from the frame difference pixel values; A randomization module that generates a randomized target spike sequence from the number of spike events; A first determination module that determines whether to adjust the generation frame rate or frame difference frequency of the frame image from at least the total frame difference pixel values of the frame difference and / or the corresponding total number of spike events; A self-adaptive spike generation device, characterized by comprising the above.

7. When the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is less than a first threshold value, the frame rate or the frame difference frequency is decreased, and / or when it is greater than a second threshold value, the frame rate or the frame difference frequency is increased, or The self-adaptive spike generation device according to claim 6, characterized in that when the sum of the total number of spike events corresponding to the frame difference, or the sum of the number of spike events satisfying a predetermined condition, or the count value of the number of frame difference pixels whose spike event number satisfies a predetermined condition is less than a first threshold value, the frame rate or the frame difference frequency is decreased, and / or when it is greater than a second threshold value, the frame rate or the frame difference frequency is increased.

8. The self-adaptive spike generation device according to claim 6, further comprising a second determination module that determines whether the sum of the frame difference pixel values and / or the sum of the spike event numbers is greater than a third threshold value, and when it is greater, ends the randomization operation for the spike event.

9. A spike sequence randomization method including a step of obtaining frame difference pixel values between frame images, and a step of calculating the number of spike events corresponding to the frame difference pixel values based on the frame difference pixel values. This spike sequence randomization method i) storing the pixel coordinates corresponding to the frame difference pixel values in all memory units having the same number as the number of spike events, reading the pixel coordinates stored in the memory unit corresponding to the currently obtained random number based on the currently obtained random number, and generating a spike event in the target spike sequence based on at least the read pixel coordinate information; ii) storing the number of spike events in the memory unit corresponding to the pixel coordinates of the frame difference pixel values, reading the number of spike events stored in the memory unit corresponding to the currently obtained random number based on the currently obtained random number, and when the read number of spike events is not zero, generating a spike event in the target spike sequence based on at least the pixel coordinate information of the frame difference pixel values corresponding to the memory unit. A spike sequence randomization method, characterized by further including at least one of the above.

10. The frame difference pixel value is the difference between pixel values of two adjacent frame images or the absolute value of the difference between pixel values of two adjacent frame images, and the spike sequence randomization method according to claim 9 is characterized in that.

11. The number of spike events corresponding to the frame difference pixel value is i) the ratio of the product of the frame difference pixel value and the maximum number of spike events that an allowed single pixel can emit to the theoretical maximum value of pixels in the frame difference; ii) the ratio of the product of the frame difference pixel value and the maximum number of spike events that an allowed single pixel can emit to the maximum pixel value in the frame difference; iii) the ratio of the product of the frame difference pixel value and the maximum number of spike events that an allowed single pixel can emit to the larger of the maximum pixel value or the first lower limit value in the frame difference, and the spike sequence randomization method according to claim 9 is characterized in that it is obtained based on one of them.

12. After obtaining the frame difference pixel value between frame images, when the frame difference pixel value is smaller than the second lower limit value, the frame difference pixel value is set to zero, and the spike sequence randomization method according to claim 9 is characterized in that.

13. A random number sequence is generated by a random number generation module, and the currently acquired random number is derived from the random number sequence. The random number sequence does not generate the same random number within one cycle. The random number generation module includes at least one random number generator using a linear feedback shift register, or the cycle of the random number sequence is equal to the product of the number of frame difference pixels and the maximum number of spike events that an allowed single pixel can emit, or equal to the number of frame difference pixels, and the spike sequence randomization method according to claim 9 is characterized in that.

14. When reading the number of spike events stored in the memory unit corresponding to the random number based on the currently acquired random number, the random number is derived from the random number sequence, the cycle of the random number sequence is equal to the number of frame difference pixels, For each frame difference, the number of cycles executed by the random number sequence is equal to the maximum number of spike events that an allowed single pixel can emit. and the spike sequence randomization method according to claim 9 is characterized in that.

15. The step of obtaining a channel output signal Based on the channel output signal, obtaining the number of spike events corresponding to the channel output signal; In a spike sequence randomization method including: i) storing the channel address corresponding to the channel output signal in all memory units having the same number as the number of spike events, reading the channel address stored in the memory unit corresponding to the random number based on the currently obtained random number, and generating spike events in the target spike sequence based on at least the read channel address information; ii) storing the number of spike events in the memory unit corresponding to the channel address of the channel output signal, reading the number of spike events stored in the memory unit corresponding to the random number based on the currently obtained random number, and if the read number of spike events is not zero, generating spike events in the target spike sequence based on at least the channel address information of the channel output signal corresponding to the memory unit; A spike sequence randomization method, further comprising at least one of the above.

16. A frame difference memory space and a spike event address memory space or a spike event number memory space, or A frame image memory space and a spike event number memory space Including: A spike sequence randomization device is configured to execute the spike sequence randomization method according to any one of Claims 9 to 15, The frame difference memory space stores frame difference pixel values, The memory units included in the spike event address memory space store the pixel coordinates corresponding to the frame difference pixel values, The memory units included in the spike event number memory space store the number of spike events corresponding to the frame difference pixel values, The memory units included in the frame image memory space store the latest pixel values obtained from the frame image sensor A spike sequence randomization device characterized by the above.

17. A brain chip, using the self-adaptive spike generation method according to any one of Claims 1 to 5, or including the self-adaptive spike generation device according to any one of Claims 6 to 8, or applying the spike sequence randomization method according to any one of Claims 9 to 15, Or Including the spike sequence randomization device according to claim 16 A brain chip characterized by the above. **Claim 18** In an electronic device, Including the self-adaptive spike generation device according to any one of claims 6 to 8, or Including the spike sequence randomization device according to claim 16, or Including the brain chip according to claim 17 The electronic device characterized by the above.

Images