Frame Image Spike Conversion System

Abstract

The present invention discloses a frame image spike conversion system. In order to solve the technical problem of high power consumption when converting frame images into spike events, the present invention self-adaptively adjusts the frame rate or differential frame frequency according to the effective pixel change amount between frame images, thereby reducing the power consumption of the system and self-adaptively responding to the usage requirements. In order to solve the problem of low inference accuracy when the frame images are converted into spike events and then directly applied to the SNN processor, the present invention considers the image randomization process as a whole and proposes a global randomization means. The present invention discloses the entire process of applying a frame image sensor to an SNN processor, providing an alternative to the event camera. The present invention is suitable for the fields of brain-type chips and AIoT.

Description

Technical Field

[0001] This patent application claims priority from the invention patent applications with application numbers 202210381831.2 (Self - Adaptive Spike Generation Method, Device, Brain - Type Chip and Electronic Device) filed with the China National Intellectual Property Administration on April 13, 2022, and 202210387836.6 (Spike Sequence Randomization Method, Device, Brain - Type Chip and Electronic Device) filed with the China National Intellectual Property Administration on April 14, 2022, and incorporates and integrates all the content described therein into this patent application documents.

[0002] The present invention relates to a method, device, brain - type chip and electronic device for converting a frame image into spike events, and specifically relates to a self - adaptive spike generation and its randomization method, device, brain - type chip and electronic device.

Background Art

[0003] A spiking neural network (SNN) has excellent performance in efficiently processing complex, sparse, and noisy spatio - temporal information due to its rich neural dynamics characteristics and spike - event communication method, and is currently the neural network closest to the operation model of the human brain. Because of its ultra - low power consumption and high - level intelligence, it is a new research concern in the field of artificial intelligence.

[0004] A dynamic vision sensor (DVS) is a new type of sensor with an independent photosensitive pixel array. When a pixel senses a change in light intensity, it independently and asynchronously transmits spike events to the subsequent system (such as a brain - type chip, etc.). Because of its ultra - high temporal resolution, it has attracted attention. However, at present, there is still a certain gap between its signal - to - noise ratio, low - light detection ability, consistency, and measurability and the actual application requirements.

[0005] Conventional image sensors based on frames (e.g., CMOS sensors, CCD sensors, etc., abbreviated as frame image sensors) are relatively mature image capture devices with a high signal-to-noise ratio. However, they cannot generate spike events (abbreviated as spikes or events, and several spikes constitute a spike sequence), and can only continuously generate multiple images in units of frames.

[0006] Generating a spike sequence (preferably a random spike sequence that conforms to the Poisson distribution) that meets the requirements of a spike neural network processor (also called a brain-type chip) by a frame image sensor is an alternative to a DVS sensor. This alternative has the advantage that it can be directly integrated into an electronic device having a frame image sensor, and the existing device can be used as it is.

[0007] Therefore, it is required to meet the application demand of the SNN processor by a spike generation technology with a high signal-to-noise ratio, low power consumption, low cost, and easy implementation. For the known literature on converting the images collected by a frame image sensor into spike events, specifically, the following can be referred to. Known Literature 1: CN111898737A Known Literature 2: CN111860786A Known Literature 3: EP3789909A1 Known Literature 4: CN112464807A

[0008] The known documents 1 to 2 disclose a means for converting numerical values / images into spike sequences. The core idea is to generate an initial spike sequence for a specific pixel value and randomly exchange or modify a certain number of spikes to obtain a target spike sequence. However, its drawback is that the number of randomly exchanged or modified spikes is necessarily finite, and the effect of randomization cannot be guaranteed (especially when the time step is long), and the cost for hardware implementation is relatively high. In addition, obtaining contour information by the frame difference method is not mentioned. Only random spike sequence conversion is performed for each pixel value of a general frame image, and the discrimination accuracy of the spike sequence obtained by such a method in a spiking neural network is extremely low.

[0009] The known documents 3 to 4 disclose a region of interest extraction means based on frame difference, which compares the differences between different images of two frames to generate a spike event sequence. However, there are problems such as high power consumption and insufficient real-time performance, and clear noise cannot be eliminated, so the network performance is not high. In addition, a means for randomizing the spike sequence is not shown, which has a very adverse effect on the inference accuracy of the SNN processor.

Summary of the Invention

Problems to be Solved by the Invention

[0010] The present invention provides an adaptive spike generation and its randomization method, device, brain-type chip and electronic device.

Means for Solving the Problems

[0011] The present invention attempts to solve or alleviate some or all of the above technical problems by the following technical means.

[0012] An adaptive spike generation method, comprising: comparing the differences between frame images to obtain the frame difference pixel values of each pixel coordinate point in the frame difference; obtaining the number of spike events corresponding to the pixel coordinates from the frame difference pixel values; determining whether to adjust the frame rate or frame difference frequency of the frame image from at least the overall frame difference pixel values of the frame difference or / and the corresponding overall number of spike events; including.

[0013] In one embodiment, when the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is smaller than a first threshold value, the frame rate or frame difference frequency is decreased, or / and when it is larger than a second threshold value, the frame rate or frame difference frequency is increased, or when the sum of the spike events corresponding to the frame difference, or the sum of the spike events satisfying a predetermined condition, or the count value of the number of frame difference pixels whose spike event number satisfies a predetermined condition is smaller than a first threshold value, the frame rate or frame difference frequency is decreased, or / and when it is larger than a second threshold value, the frame rate or frame difference frequency is increased.

[0014] In one embodiment, when the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is smaller than a first ratio, the frame rate or frame difference frequency is decreased, or / and when it is larger than a second ratio, the frame rate or frame difference frequency is increased, the first ratio is the ratio of the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition in the whole frame difference pixels, or, when the count value of the number of frame difference pixels whose spike event number satisfies a predetermined condition is smaller than a first ratio, the frame rate or frame difference frequency is decreased, or / and when it is larger than a second ratio, the frame rate or frame difference frequency is increased, the second ratio is the ratio of the count value of the number of frame difference pixels whose spike event number satisfies a predetermined condition in the whole frame difference pixels.

[0015] In one embodiment, after reducing the frame rate or shortening the first time interval of the frame difference frequency, the frame rate or the frame difference frequency is restored to the default value, or / and within the second time interval, when the sum of the frame difference pixel values and / or the sum of the number of spike events are both smaller than the first threshold, the frame rate or the frame difference frequency is reduced.

[0016] In one embodiment, the self-adaptive spike generation method includes: generating a randomized target spike sequence from the number of spike events; determining whether the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is greater than a third threshold. If it is greater, no further step of obtaining the number of spike events corresponding to the pixel coordinates from the frame difference pixel values is executed; or further including determining whether the sum of the frame differences corresponding to the number of spike events, or the sum of the number of spike events satisfying a predetermined condition, or the count value of the number of frame difference pixels whose spike events satisfy a predetermined condition is greater than a third threshold. If it is greater, no further step of generating a randomized target spike sequence from the number of spike events is executed.

[0017] The self-adaptive spike generation device is used to generate a target spike sequence, and the self-adaptive spike generation device includes: a frame difference module that compares the differences between frame images and obtains the frame difference pixel values of each pixel coordinate point in the frame difference; a spike event number generation module that generates the number of spike events corresponding to the pixel coordinates from the frame difference pixel values; a randomization module that generates a randomized target spike sequence from the number of spike events; A first determination module that determines whether to adjust the generation frame rate or frame difference frequency of the frame image from at least the overall frame difference pixel value of the frame difference or / and the corresponding overall number of spike events; including.

[0018] In one embodiment, when the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is smaller than a first threshold value, the frame rate or the frame difference frequency is decreased, or / and when it is larger than a second threshold value, the frame rate or the frame difference frequency is increased, or, When the sum of the corresponding spike event numbers of the frame difference, or the sum of the spike event numbers that satisfy a predetermined condition, or the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition is smaller than a first threshold value, the frame rate or the frame difference frequency is decreased, or / and when it is larger than a second threshold value, the frame rate or the frame difference frequency is increased.

[0019] In one embodiment, the self-adaptive spike generation device further includes a second determination module that determines whether the sum of the frame difference pixel values and / or the sum of the spike event numbers is greater than a third threshold value, and when it is greater, ends the randomization operation for the spike event.

[0020] The spike sequence randomization method includes a step of obtaining a frame difference pixel value between frame images and a step of calculating the number of spike events corresponding to the frame difference pixel value based on the frame difference pixel value, and the spike sequence randomization method further includes at least one of the following steps. i) Storing the pixel coordinates corresponding to the frame difference pixel values in all memory units where the number is the same as the number of spike events, reading the pixel coordinates stored in the memory unit corresponding to the currently acquired random number based on the currently acquired random number, and generating spike events in the target spike sequence based on at least the read pixel coordinate information; ii) Storing the number of spike events in the memory unit corresponding to the pixel coordinates of the frame difference pixel values, reading the number of spike events stored in the memory unit corresponding to the currently acquired random number based on the currently acquired random number, and if the read number of spike events is not zero, generating spike events in the target spike sequence based on at least the pixel coordinate information of the frame difference pixel values corresponding to the memory unit.

[0021] In one embodiment, the frame difference pixel value is the difference between the pixel values of two adjacent frame images or the absolute value of the difference between the pixel values of two adjacent frame images.

[0022] In one embodiment, the number of spike events corresponding to the frame difference pixel value is obtained based on one of the following methods. i) The ratio of the product of the frame difference pixel value and the maximum number of spike events that an allowed single pixel can emit to the theoretical maximum value of the pixels in the frame difference; ii) The ratio of the product of the frame difference pixel value and the maximum number of spike events that an allowed single pixel can emit to the maximum pixel value in the frame difference; iii) The ratio of the product of the frame difference pixel value and the maximum number of spike events that an allowed single pixel can emit to the larger of the maximum pixel value or the first lower limit value in the frame difference.

[0023] In one embodiment, after obtaining the frame difference pixel value between frame images, if the frame difference pixel value is smaller than the second lower limit value, the frame difference pixel value is set to zero.

[0024] In one embodiment, a random number generation module generates a random number sequence, and the currently acquired random number is derived from the random number sequence. The random number sequence does not generate the same random number within one cycle. The random number generation module includes at least one random number generator using a linear feedback shift register, or the cycle of the random number sequence is equal to the product of the number of frame difference pixels and the maximum number of spike events that an allowed single pixel can emit, or is equal to the number of frame difference pixels.

[0025] In one embodiment, when reading the number of spike events stored in the memory unit corresponding to the random number based on the currently acquired random number, the random number is derived from the random number sequence. The cycle of the random number sequence is equal to the number of frame difference pixels, and for each frame difference, the number of cycles that the random number sequence executes is equal to the maximum number of spike events that an allowed single pixel can emit.

[0026] A spike sequence randomization method includes a step of acquiring a channel output signal and a step of acquiring the number of spike events corresponding to the channel output signal based on the channel output signal, and the spike sequence randomization method further includes at least one of the following steps. i) Storing the channel address corresponding to the channel output signal in all memory units with the same number as the number of spike events, reading the channel address stored in the memory unit corresponding to the random number based on the currently acquired random number, and generating spike events in the target spike sequence based on at least the read channel address information. ii) Store the number of spike events in a memory unit corresponding to the channel address of the channel output signal, read the number of spike events stored in the memory unit corresponding to the currently acquired random number based on the currently acquired random number, and if the read number of spike events is not zero, generate spike events in the target spike sequence based on at least the channel address information of the channel output signal corresponding to the memory unit.

[0027] In a spike sequence randomization device, a frame difference memory space and a spike event address memory space or a spike event number memory space, or a frame image memory space and a spike event number memory space, and including the spike sequence randomization device is configured to execute the spike sequence randomization method according to any one of the preceding items, the frame difference memory space stores frame difference pixel values, the memory units included in the spike event address memory space store the pixel coordinates corresponding to the frame difference pixel values, the memory units included in the spike event number memory space store the number of spike events corresponding to the frame difference pixel values, the memory units included in the frame image memory space store the latest pixel values acquired from the frame image sensor.

[0028] In a brain chip, the brain chip uses the self-adaptive spike generation method according to any one of the preceding items, or includes the self-adaptive spike generation device according to any one of the preceding items, or applies the spike sequence randomization method according to any one of the preceding items, or includes the spike sequence randomization device described above.

[0029] In an electronic device, the electronic device includes the self-adaptive spike generation device according to any one of the foregoing items, or includes the spike sequence randomization device described above, or includes the brain-type chip described above.

Advantages of the Invention

[0030] Some or all of the embodiments of the present invention have the following beneficial technical effects. 1) In the process of generating a target spike sequence, the present invention can self-adaptively adjust the frame rate or the frame difference frequency, and has high flexibility. It can also ensure low-power consumption operation in an environment with no change for a long time, and can quickly capture changes in motion when necessary. 2) The present invention can promptly end an unreasonable process (where the total frame difference pixel value or the total number of spike events is too large), effectively remove noise, and at the same time save power consumption. By controlling the number of spikes in the target spike sequence, the present invention ensures that the SNN processor operates in real time, with high efficiency and stability. 3) The hardware of the present invention is user-friendly and realizes low cost. Compared with DVS, the existing camera in the electronic device can be used as it is, and it is not necessarily required to install a new imaging module and sensor. 4) Multiple steps of the present invention can be processed in parallel, saving resources and further reducing power consumption. 5) Considering macroscopically that an image as a whole is randomized into a target spike sequence, the spiking neural network has a user-friendly randomization method, and there is no defect that the performance of the spiking neural network is significantly reduced by connecting the spike sequences corresponding to each pixel one by one. Therefore, its randomization effect is high. 6) It consumes less hardware resources and less energy consumed in conversion. 7) The outline can be enhanced, and the network performance in low-light and slightly shaking situations can be improved. 8) Suppress background noise and reduce power consumption during long-term standby. More beneficial effects will be further introduced in the preferred embodiments.

[0031] The purpose of disclosing the technical means / features above is to summarize the technical means and technical features described in the specific embodiment part. Therefore, the described scope may not be exactly the same. However, these new technical means disclosed in this part are also part of the numerous technical means disclosed in this patent document. The technical features disclosed in this part, in a reasonably combined form with the technical features disclosed in the subsequent specific embodiment part and some content in the drawings not clearly described in the specification, disclose more technical means.

[0032] All the technical features disclosed at any position of the present invention are combined to form technical means, which are used to support the summary of technical means, the modification of patent documents, and the disclosure of technical means.

Brief Description of the Drawings

[0033]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Modes for Carrying Out the Invention

[0034] Since it is impossible to describe all the various alternatives, the following will describe the present invention in conjunction with the drawings in the embodiments of the present invention to clearly and completely describe the key points of the technical means in the embodiments of the present invention. For other technical means and details not disclosed in detail below, they generally fall within the technical objectives or technical features that can be realized by ordinary means in this field. Due to the limitation of the paper width, the details are not introduced in the present invention.

[0035] Unless it is the meaning of division, " / " at any position in the present invention represents a logical OR (i.e., "or"). Numbers such as "first", "second", etc. at any position in the present invention are only used for descriptive classification and do not imply an absolute order in time or space, nor do they imply that terms with such numbers necessarily refer to different things from the same terms with other limitations.

[0036] The present invention describes key points for combination as various different specific embodiments, and these key points are combined in various methods and products. In the present invention, even if it is a key point described only when introducing method / product means, it means that the corresponding product / method means also clearly includes this technical feature.

[0037] When it is described that there is, or includes, a certain step, module, or feature at any position in the present invention, this kind of existence does not imply that it is exclusive and the only existence. A person skilled in the art can completely supplement other technical means based on the technical means disclosed in the present invention to obtain other embodiments. Based on the key points described in the specific embodiments of the present invention, a person skilled in the art can completely obtain technical means that comply with the inventive concept of the present invention by adding means such as substitution, reduction, increase, combination, and order change to some technical features. Means that do not deviate from these inventive concepts of the present invention are also within the protection scope of the present invention.

[0038] The frame image sensor of the present invention is a sensor capable of acquiring frame images such as a CMOS sensor, a CCD sensor, and a grayscale sensor. The present invention is not limited to a specific type of image sensor, as long as it can acquire a frame image.

[0039] Figure 1 is a flowchart of a self-adaptive spike generation method in a preferred embodiment of the present invention and includes the following steps.

[0040] Step S100: Frame image preprocessing.

[0041] The dataset of the spiking neural network consists of a spatio-temporal event stream and has sparse characteristics. On the other hand, the resolution of the conventional frame image sensor is high, and the number of spiking events generated after conversion may increase. Therefore, the frame image generated by the sensor may be pre-processed in advance to reduce the number of events and make the generated target spike sequence have sparsity.

[0042] Furthermore, as shown in FIG. 2, the pre-processing operation may include dimension reconstruction and grayscale conversion.

[0043] Step S101: Perform dimension reconstruction (reshape) on any frame image.

[0044] Let the resolution of the frame image sensor be (W×H), and the resolution of the frame image after reconstruction (also referred to as the target dimension) be (W’×H’), where W and W’ represent the width, and H and H’ represent the height.

[0045] In one embodiment, the method of dimension reconstruction is subsampling. The reconstruction of the frame image dimension by subsampling is easy to implement and suitable for hardware. Specifically, after adjusting any pixel coordinate I(x,y) in the original frame image, the pixel coordinate

Number

Number

[0046] In another embodiment, the method of dimension reconstruction is cropping. Based on the region of interest, the original frame image is cropped to obtain the target dimension. For example, crop any region of interest such as the upper left corner, the lower right corner, or the center of the original frame image. The cropping method of the present invention is not limited to this.

[0047] In other embodiments, the dimension reconstruction method may be any method such as bilinear interpolation, linear or non-linear zoom, taking values according to a ratio, etc. The present invention does not limit the dimension reconstruction method, and any means capable of arbitrarily adjusting the original frame image to the target dimension may be used.

[0048] Step S102: Convert the frame image to grayscale.

[0049] Normally, the frame image is an RGB image and has three channels. The present invention converts the RGB image into a grayscale image, and there may be multiple grayscale conversion methods. The present invention does not limit a specific grayscale conversion method. For example, it may not be limited to extracting the value in any one of the three channels as the grayscale value, and the maximum value, average value, or weighted average value of RGB may be used as the grayscale value, etc.

[0050] The present invention does not limit the order of performing the dimension reconstruction and grayscale conversion operations, and they may be performed in parallel or sequentially, or only one of the operations may be performed. Further, the frame image preprocessing operation S100 of the present invention is not essential and is only performed when necessary.

[0051] Step S200: Perform a frame difference operation on the frame image to obtain the frame difference pixel value of each pixel point.

[0052] Specifically, at least two different frame images in the frame image sequence or the preprocessed frame image are compared to obtain a frame difference image of the image change between the two frames. The frame difference image reflects the change / motion information between frames.

[0053] In the frame difference process, the change value or the absolute value of the change value of the pixel value on each obtained pixel point (pixel coordinates), or a value obtained by performing operations such as noise filtering, is referred to as the frame difference pixel value. At a certain pixel point, the positive or negative of the frame difference pixel value may correspond to the polarity of the spike event. For example, if the frame difference pixel value is positive, it indicates an enhancement of the pixel, and the polarity of the corresponding event is positive. If the frame difference pixel value is negative, it indicates a reduction of the pixel, and the polarity of the corresponding event is negative. The reverse is also possible, and the present invention is not limited thereto.

[0054] In another several embodiments, when the change in the pixel value on a certain pixel point does not satisfy a predetermined condition (for example, being less than a certain threshold value, etc.), the frame difference pixel value may be discarded, that is, set to 0, which is useful for noise filtering.

[0055] For an image sequence including N frames, the difference operation may be executed sequentially or in parallel. When executed in parallel, power consumption and time can be extremely significantly saved, and the processing efficiency and real-time performance can be improved. Further, the difference between two adjacent frames may be executed sequentially or in parallel, and the difference between two frames separated by several frames may also be executed sequentially or in parallel. The present invention is not limited thereto.

[0056] In addition, the image sequence may be a set of original frame images collected by the sensor or a set of frame images after preprocessing. Further, the order of step S100 and step S200 may be interchanged. Frame difference may be performed after preprocessing the frame image, or preprocessing operations may be performed after performing frame difference first.

[0057] Step S300: Generate the number of spike events corresponding to the pixel coordinates from the frame difference pixel values of each pixel point.

[0058] Based on the frame difference pixel values of each pixel point, the number of spike events corresponding to each pixel coordinate point is obtained. Any reasonable conversion method from pixel values to the number of spike events is executable, and the present invention does not limit which method is adopted to obtain the number of spike events corresponding to each pixel point after frame difference.

[0059] For example, the pixel value after frame difference is m, which generates the number of spike events after rounding up or rounding down, and the number of spike events is the value after rounding down of |m|

Number

Number

[0060] Step S400: Determine whether the first condition is satisfied based on at least the overall frame difference pixel value of the frame difference or / and the corresponding overall number of spike events. If satisfied, adjust the frame image generation frame rate (abbreviated as frame rate) of the frame image sensor, or adjust the frame difference frequency in step S200. The specific adjustment method may be to adjust the frame rate / frame difference frequency upward or downward. If the condition is not satisfied, as shown in FIG. 3, do not adjust the frame rate or the frame difference frequency.

[0061] For one frame difference, it includes several pixels, and each of these pixels corresponds to one frame difference pixel value or the number of spike events. For one frame difference, the overall frame difference pixel value or / and the corresponding overall number of spike events is the set constituted by the frame difference pixel values or the number of spike events corresponding to these several pixels. From the whole set, it is determined whether to adjust the frame rate or the frame difference frequency, and various conversion, aggregation, calculation and other methods are adopted for the set to obtain a value with a certain meaning (for example, the degree of intensity characterizing the movement of an object in front of the sensor, etc.), and from this value, it is determined whether to trigger the above adjustment operation.

[0062] Furthermore, for the embodiment including the preprocessing step S100, it may be adjusted by controlling step S100 or / and S200 or / and S300.

[0063] Furthermore, processes such as obtaining the sum of the number of spike events or the frame difference pixel values corresponding to each pixel position amount are performed. (Specifically, refer to step S200 or S300.) The step of obtaining the sum may be executed sequentially or in parallel. In hardware, it is easy to execute in parallel and has advantages in both power consumption and real-time performance.

[0064] The first condition includes one or more of the following cases.

[0065] When the sum of frame difference pixel values, or the sum of frame difference pixel values where the pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels where the pixel values satisfy a predetermined condition is less than the first threshold value, the frame rate or the frame difference frequency is decreased, and / or when it is greater than the second threshold value, the frame rate or the frame difference frequency is increased. Or, when the sum of the total number of spike events corresponding to the frame difference, or the sum of the number of spike events that satisfy a predetermined condition, or the count value of the number of frame difference pixels where the number of spike events satisfies a predetermined condition is less than the first threshold value, the frame rate or the frame difference frequency is decreased, and / or when it is greater than the second threshold value, the frame rate or the frame difference frequency is increased.

[0066] Here, the sum of frame difference pixel values / the sum of the total number of spike events corresponding to the frame difference is obtained by directly summing the respective pixel values in the frame difference or the number of spike events corresponding to each pixel. The sum of frame difference pixel values where the pixel values satisfy a predetermined condition / the sum of the number of spike events that satisfy a predetermined condition is obtained by setting one condition (for example, pixel value ≥ 10, number of spike events ≥ 2, etc.) and accumulating and summing only the pixel values / number of spike events that satisfy the condition. The count value of the number of frame difference pixels where the pixel values satisfy a predetermined condition / the count value of the number of frame difference pixels where the number of spike events satisfies a predetermined condition is obtained by setting a condition (for example, pixel value ≥ 1 or 10, number of spike events ≥ 1 or 2, etc.) and accumulating and counting only the pixels (coordinates) that satisfy the condition.

[0067] Here, in the case of downward adjustment, for example, when there is no change over a long period of time or the change is unclear in two frame images of frame difference, the sum of the frame difference pixel values of all pixels is small, and the sum of the corresponding spike generation numbers is small. Therefore, it is considered that there is no operation waiting for identification or the presence of a trigger, and the frame rate or the frame difference frequency is decreased to reduce power consumption. For example, the frame rate is decreased from 30 to 1. The decrease in the frame rate or the frame difference frequency may be stepwise. For example, it may be from 60 to 30 and then to 1. Correspondingly, at this time, the first threshold value includes a plurality of numerical values.

[0068] In the case of upward adjustment, for example, when the current frame rate is too low and cannot satisfy the image collection for a rapidly moving object, but it is detected that the target object movement has started, by increasing the frame rate, the situation where the movement becomes ambiguous in the image after frame difference is reduced. For example, the frame rate is increased from 1 to 30. Similarly, the increase in the frame rate or the frame difference frequency may be stepwise. For example, it may be from 1 to 30 and then to 60. Correspondingly, at this time, the second threshold value includes a plurality of numerical values.

[0069] Here, the first threshold value and the second threshold value may be different or the same. In some embodiments, it is preferable to refer to the state of the current frame rate or the frame difference frequency before determining the upward adjustment or the downward adjustment.

[0070] In another embodiment, after shortening the first time interval of the frame rate or the frame difference frequency, the frame rate or the frame difference frequency is restored to the default value. Referring to the previous example, the default value may be 30 / 60 and 10 as described above. This embodiment may allow a certain ratio of time width and increase the detection response sensitivity.

[0071] In another embodiment, the width of the upward adjustment and the downward adjustment may be adjusted in proportion to the difference after comparison with a threshold value. For example, the first threshold value and the second threshold value both refer to the threshold value of the total number of spike events, and both are 1000. When the total number of spike events generated after frame difference is 100, the frame rate is adjusted from 60 to 6. When the total number of spike events generated after frame difference is 300, the frame rate is adjusted from 60 to 18, and so on.

[0072] Also, the first condition may be further extended as follows, that is, within a set time, for various frame differences, the sum of frame difference pixel values / sum of frame difference pixel values satisfying a predetermined condition, the count value of the number of frame difference pixels / sum of corresponding spike events satisfying the predetermined condition, the sum of spike events / sum of spike events satisfying the predetermined condition, and the count value of the number of frame difference pixels satisfying the predetermined condition are all smaller than the first threshold value. In this case, it is considered that the condition is satisfied, and the frame rate or the frame difference frequency is reduced. In such a case, it is useful to reduce power consumption in a situation where there is no target object movement, and at the same time, to prevent the response from being delayed due to the immediate reduction of the frame rate or the frame difference frequency caused by the lack of operation of the user for a while. Preferably, when adjusting upward, no similar time is set, or even if time is set, the time is extremely short, because setting such time artificially delays the response when an immediate response is required.

[0073] As an equivalent substitution, in the above-mentioned embodiment, the judgment between the count value of the number of frame difference pixels satisfying a predetermined condition / spike event and the first threshold value / second threshold value of the count value of the number of frame difference pixels satisfying the predetermined condition is replaced by the judgment between the ratio of the count value in the total number of frame difference pixels and the first threshold value / second threshold value. Since the number of frame difference pixels is a constant, the difference lies only in whether the first threshold value / second threshold value is divided by the constant. Such an equivalent substitution is also within the scope protected by the present invention.

[0074] Preferably, the first condition is to reduce power consumption, especially when there is no moving object in the field of view of the frame image sensor for a long time, and at the same time consider increasing the frame rate or frame difference frequency to meet the user's instantaneous response requirements. Therefore, it is possible to combine the above-mentioned technical means such as time, number of spike events, frame difference pixel value, ratio, stepwise threshold, etc. All such easily conceivable combinations and conversions do not deviate from the concept of the present invention and should be within the scope protected by the present invention.

[0075] Step S500: Randomization of spike events.

[0076] For a spiking neural network (SNN) processor, its input data is a spatio-temporal event stream / spike sequence, and each event includes the coordinates, timestamp generated, for example, an AER (Address Event Representation) event stream, a SAER (Serial AER) event stream, etc. The SNN processor is suitable for receiving a random spike sequence that conforms to the Poisson distribution, and any reasonable randomization method is executable (see References 1 and 2 in the public literature), and the present invention is not limited thereto.

[0077] Due to the sparsity of the spatio-temporal event stream, the present invention randomizes the spike events to match the information processing characteristics of the SNN processor. In one embodiment, all pixels corresponding to the spike events are randomized as a whole to obtain a target spike sequence. In another embodiment, the spike events corresponding to each pixel point are randomized respectively, and then a final target spike sequence is obtained. Also, the randomization process of the spike events on each pixel point may be executed in parallel or sequentially, and the parallel means with the advantage of low latency is preferred. Hereinafter, the present invention further provides specific spike sequence randomization means based on the entire image.

[0078] Steps S400 and S500 of the present invention may be executed in parallel or in sequence.

[0079] As shown in FIG. 4, in a preferred embodiment, before the spike randomization step S500, it is determined whether each pixel coordinate satisfies a second condition based on the corresponding frame difference pixel value and / or the number of spike events. If it is satisfied, step S600 of ending the current process is further included. The ending method includes frame discarding (for example, resetting / clearing the current set of spike events, etc.) or not performing the randomization step, and the present invention is not limited thereto.

[0080] Specifically, the second condition is to determine whether the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is greater than a third threshold. If it is greater, the step of obtaining the number of spike events corresponding to the pixel coordinates from the frame difference pixel values is not executed any further. Or, it is determined whether the sum of the frame differences corresponding to the number of spike events, or the sum of the number of spike events that satisfy a predetermined condition, or the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition is greater than a third threshold. If it is greater, the step of generating a randomized target spike sequence from the number of spike events is not executed any further.

[0081] When the frame image sensor is shaken or tampered with violently, within a short period of time, the sum of the number of spike events corresponding to the frame difference / the sum of the frame difference pixel values becomes too large. As a result, the generated output spike sequence may not be valid input information for the subsequent SNN processor, and processing it may cause unnecessary result output. At the same time, because the number of spike events is too large, the processor is likely to become immobile or the power consumption increases. Therefore, before the randomization step, the frame image with too fast changes is filtered and removed in advance to ensure the processor performance and at the same time reduce the power consumption.

[0082] FIG. 5 is a self - adaptive spike generation device according to an embodiment of the present invention. The present invention further discloses a self - adaptive spike generation device for generating a target spike sequence, and the self - adaptive spike generation device includes: a frame difference module that compares the differences between frame images and obtains the frame difference pixel values of each pixel coordinate point in the frame difference; a spike event number generation module that generates the number of spike events corresponding to the pixel coordinates from the frame difference pixel values; a randomization module that generates a randomized target spike sequence from the number of spike events; a first judgment module that determines whether to adjust the generation frame rate or the frame difference frequency of the frame image from at least the overall frame difference pixel values of the frame difference or / and the corresponding overall number of spike events; and includes.

[0083] In some embodiments, the self - adaptive spike generation device further includes a pre - processing module for pre - processing the frame images input to the frame difference module. The pre - processing operation may include dimension reconstruction or / and grayscale conversion.

[0084] In some embodiments, the self - adaptive spike generation device further includes a second judgment module that determines whether each pixel coordinate meets a second condition from the corresponding frame difference pixel value or / and the number of spike events, and if it meets, ends the current process.

[0085] For more adjustment solutions, reference may be made to the above - mentioned method - based embodiments, and they are incorporated into these embodiments in the form of citations here and will not be described further here.

[0086] FIG. 6 is a block diagram of the self - adaptive spike generation and processing system of the present invention, including a frame image sensor, a self - adaptive spike generation device, and a neural network processor connected in sequence. The frame image sensor collects frame images, the self - adaptive spike generation device is used for rapid generation of low - power - consumption, high - quality target spike sequences, and the neural network processor (such as an SNN processor, etc.) is used for inference and outputting classification results.

[0087] Here, the self - adaptive spike generation device may be installed independently or integrated into a processor or a frame image sensor. In some embodiments, the self - adaptive spike generation device is implemented as a frame image sensor interface. Further, the frame image sensor, the self - adaptive spike generation device, and the neural network processor are integrated on the same chip, and the chip simultaneously has sensing and computing capabilities.

[0088] FIG. 7 is another block diagram of the self - adaptive spike generation and processing system of the present invention, including a frame image sensor, a self - adaptive spike generation device, an event sensor (such as an event camera), and a neural network processor. The self - adaptive spike generation device converts the frame image collected by the frame image sensor into a first target spike sequence, the event camera outputs a second target spike sequence, and the neural network processor receives and processes the first target spike sequence or / and the second target spike sequence. Similarly, the frame image sensor or the event camera may be installed independently or integrated on the same chip together with the processor.

[0089] Figure 8 is a block diagram of spike sequence randomization in the present invention. The frame image sensor continuously outputs image information captured from the environment in the form of frames, for example, outputs 30, 60, or 100 frames of images per second to the memory of the frame image sensor. For example, there are common RGB cameras and the like. The frame rate and the frame image sensor can be flexibly selected according to actual usage requirements, and the present invention is not limited to a specific form.

[0090] Since the frame images have sequence properties, the difference is taken between one adjacent frame or a plurality of adjacent frames before and after, and the difference image information between the two frame images, that is, the frame difference (Difference of Frames, DoF) is obtained. The frame difference may be the absolute value excluding the pixel difference and may retain valid sign (polarity) information. For example, if you wave the palm in front of the sensor, one frame image shows an image of the palm including the complete palmprint, while the frame difference shows the contour information of the palm, not the palmprint. This is similar to the result of the moving object contour captured by DVS.

[0091] Before performing the frame difference, steps such as image subsampling operation, grayscale information extraction, region of interest extraction, etc. may be included, and then the spike sequence randomization step is performed. More frame difference means can refer to the known literature 3. Here, the image subsampling operation helps to directly use the existing cameras in the electronic device, and it is not necessarily required to install a new imaging module and sensor, thus reducing costs.

[0092] FIG. 9 is a diagram showing spike sequence randomization in the first type of embodiment. The frame difference memory space stores frame difference pixel values. The memory unit included in the spike event address memory space stores the pixel coordinates corresponding to the frame difference pixel values. The number of times the spike event address (i.e., the pixel coordinates corresponding to the frame difference pixel value) is stored in the spike event address memory space is determined from the frame difference pixel value. The random number generation module generates a random number sequence, and based on the random number sequence, reads the spike event address in the spike event address memory space, that is, determines the address (x, y) of the spike event that the current time / timestamp t is about to emit, or further combines the polarity information p, that is, obtains a spike event including (x, y, p, t) information.

[0093] The random number sequence continuously reads the spike event address to generate a target spike sequence consisting of spike events, thus completing the randomization process of the spike sequence. The spike sequence is sent into the subsequent system for processing. Preferably, the subsequent system here is an SNN processor, and of course, it may also be other types of signal processing devices.

[0094] Regarding the SNN processor, when converting a frame difference or a general frame image into a random target spike sequence, it is converted into a short random spike sequence for each pixel, and then these short random spike sequences are connected in sequence and sent into the SNN processor for inference. After that, the obtained network performance is extremely low. This is because it does not conform to the spike neural network information processing characteristics at all. Known Document 1 does not show how to randomize an entire image, but only shows how to generate a randomized target spike sequence for a single pixel. On the other hand, the present invention discloses how to randomize the whole (specifically, the frame difference) of the image to form a target spike sequence.

[0095] Figure 10 shows the details of spike sequence randomization in the first type of embodiment. For example, if the image size used for frame difference is 128×128 (the product of the horizontal and vertical resolutions, denoted as the number of frame difference pixels) and the width of each pixel value is 8 bits, the size of the frame difference memory space is at least 128×128×8 bits. Since it calculates the pixel difference between frame images separated by several frames (preferably two adjacent frames), most of the pixel values of the frame difference are 0, and usually only the pixels at the contour of the moving object are not zero. Without loss of generality, let a certain pixel value in the frame difference be denoted as I(x,y), and the maximum pixel value in the frame difference be max(I(x,y)). In the formula, the lowercase (x, y) represents the position of a certain pixel in the frame image sensor, and the uppercase (x,y) represents the set consisting of the coordinates of all pixels in the frame image sensor.

[0096] After obtaining the frame difference of the frame image sensor, at least from each pixel value in the frame difference, the spike event address corresponding to the pixel (that is, the pixel coordinates of the pixel) is generated. As shown in Figure 10, the pixel value I(0,2)=97 where the coordinates are (x,y)=(0,2), and I(2,2)=63, and the maximum pixel value in the frame difference is max(I(x,y))=158.

[0097] To code these pixel values in the frame difference memory space, it is necessary to obtain the corresponding number of spike events based on these pixel values. Also, let the number of spike events corresponding to the pixel at the coordinates (x,y) in the frame difference be denoted as R(x,y).

[0098] To obtain the number of spike events R(x,y), various means may be taken. In one embodiment,

Equation

Equation

[0099] In one embodiment,

Number

[0100] In one embodiment,

Number

[0101] In an alternative embodiment, all of the foregoing rounding-down operations are replaced with rounding-up operations.

[0102] In an alternative embodiment, when the pixel value in the frame difference is smaller than the second lower limit value, the pixel value in the frame difference is directly set to zero, and preferably, this step is executed in the step of calculating the difference. For example, if the pixel value corresponding to the pixel with coordinates (0, 3) in FIG. 10 is 1 and is smaller than the second lower limit value of 5, the pixel value is set to zero. In this way, filtering of background noise can be realized.

[0103] In one embodiment, in the step of obtaining the frame difference, since the extreme information is retained, in the calculation of R(x, y), it is also necessary to consider meeting the realistic and objective requirements by taking the absolute value.

[0104] In most cases in the frame difference, that is, for pixels with a pixel value of 0, the corresponding number of spike events is usually all 0, which ensures the sparsity of the target spike sequence.

[0105] After obtaining the number of spike events R(x, y) corresponding to the pixel value in the frame difference, R(x, y) spike event addresses (x, y) are stored in the spike event address memory space. According to the length of the spike event address, the bit width of the memory space may be determined. For example, if the image size of the frame difference is 128×128, the memory space bit width is 14 bits. The addresses corresponding to the pixels in the frame difference are stored in the spike event address memory space in any reasonable manner. For example, they may be stored in row / column order / reverse order, and the same address may be stored in the correct order / incorrect order / shifted order. The present invention is not limited to a specific form.

[0106] Since the maximum number of spike events emitted by an allowable single pixel is N (for example, 8), the theoretically required maximum length of the spike event address memory space is 128×128×N when taking the frame difference resolution of 128×128 as an example. However, frame difference is only the manifestation of the object's contour, and the data of frame difference is sparse. Usually, unless there is an instantaneous switching between a pure black and a pure white environment, the maximum length is not fully utilized when actually storing addresses. Here, it is reasonable to assume that the actual length of the memory space is the number of frame difference pixels × K (K < N), for example, K = 3. Therefore, the spike event address memory space occupies the number of frame difference pixels × K × the memory space bit width, which is 128×128×K×14bit by referring to the previous example.

[0107] The random number generation module generates random integers within the numerical range of 1 to the number of frame difference pixels × N (essentially pseudo-random numbers), or random numbers within another numerical range that corresponds one-to-one with the integers within this numerical range. For example, random numbers within the range of 0 to the number of frame difference pixels × N - 1, etc., to form a random number sequence. Also, after generating these numerical values within one cycle, these numerical values are cyclically generated within the next cycle. The purpose of generating these random numbers is to enable some of these random numbers to be mapped one by one to the numbers of each memory unit in the spike event address memory space. That is, for each generated random number, based on the mapping relationship between the random number and the number in the spike event address memory space, an address stored in the memory unit corresponding to the number in the spike event address memory space can be read once. For example, if the random number sequence is 9 - 5 - 2 - 7…, the addresses with numbers 9 - 5 - 2 - 7… in the spike event address memory space are read in sequence.

[0108] Preferably, the generation of the random number sequence is realized by a linear feedback shift register (LFSR) and an XOR gate. For example, by configuring it as a circuit that expresses a primitive polynomial, the generation of pseudo-random numbers is realized. This technology belongs to the technology known in this field and will not be described further here.

[0109] The range of the pseudo-random numbers is from 1 to the number of frame difference pixels × N, and the number Kr (<K, which varies dynamically depending on the difference in frame difference) of the memory units (the memory units with the coordinates filled in in FIG. 10) in which the valid addresses due to the change of the pixel values in the frame difference are stored is not longer than the actual length of the spike event address memory space (the number of frame difference pixels × K), where K < N. Therefore, when the random number is mapped outside the range of 1 to Kr, the corresponding time / time step does not emit a spike event (also called non-spike). Therefore, the target spike sequence is usually a sparse interleaving spike sequence.

[0110] There may be more than one random number generation module. As described above, the first random number generation module generates pseudo-random numbers within a predetermined range, and the second random number generation module also generates pseudo-random numbers within a predetermined range. Both have the same predetermined range, and when in use, they generate the target spike sequence for different frame differences in a certain order (the simplest one is alternately). For three or more random number generation modules, a similar method may be adopted for processing.

[0111] For example, a simple mapping means is that since 1 to Kr in the random numbers (1 to the number of frame difference pixels × N) and the numbers 1 to Kr of the memory units in the spike event address memory space are equivalent, they are corresponding one by one. Any reasonable and executable range of random number values and the mapping relationship between them and the memory units can all be applied in the present invention and are not limited to a specific method here.

[0112] Preferably, since the target spike sequence is sent to and processed by the SNN processor, the present invention endows the SNN processor with the ability to process the environmental signals captured by the frame image sensor.

[0113] FIG. 11 shows a schematic diagram of spike sequence randomization in another embodiment of the present invention. Different from the means shown in FIG. 9, it is not to determine the number of times to store the spike event address from the pixel values in the frame difference memory space, but to generate the corresponding number of spike events R(x, y).

[0114] FIG. 12 shows a diagram illustrating the details of spike sequence randomization in another embodiment. In such an embodiment, for the sake of simplicity of description, the technical features and various technical symbols described in the previous embodiment are cited here.

[0115] After obtaining the frame difference pixel values and storing them in the frame difference memory space, the corresponding number of spike events R(x, y) is generated and stored in the corresponding memory unit of the spike event number memory space. The number of memory units in the spike event number memory space is the same as the number of pixels of the frame difference. By referring to the previous example, all are 128×128.

[0116] Similarly, a random number sequence is generated by the random number generation module. Based on the current random number, the corresponding memory unit is read. If R(x, y)>1 stored in the memory unit, the corresponding spike event, for example, a spike event including at least (x, y, t) information, is emitted. t is a time stamp.

[0117] Also, such an embodiment further updates the corresponding R(x, y) value. For example, subtract 1 from itself. As shown in FIG. 12, the number of spike events of the frame difference pixel at the address (x, y)=(2, 2) is updated from 3 to 2.

[0118] In this embodiment, the cycle of the random number sequence is equal to the number of frame-differenced pixels. In one cycle of the pseudo-random sequence, all non-zero pixels of R(x,y) emit one spike event. For one frame difference, the pseudo-random sequence circulates N times (the maximum number of spike events emitted by an allowed single pixel) until all are emitted, causing spike events. Such means destroys the randomness of spike emission to a certain extent, but actual measurement tests have shown that there is no obvious weakness in the performance shown by the SNN. Compared with the previous embodiment, such an embodiment further reduces the occupation of memory space.

[0119] Figure 13 is a diagram showing spike sequence randomization in a further embodiment. Different from the previous embodiment, after obtaining the latest pixel value from the frame image sensor, it is directly differentiated from the pixel value of the corresponding coordinate stored in the frame image memory space to obtain the pixel value of the corresponding coordinate during frame difference. Based on this pixel value, the corresponding number of spike events R(x,y) is calculated, and the number of spike events is stored in the corresponding memory unit in the spike event number memory space. In other words, since there is no frame difference memory space dedicated to storing frame difference pixel values in this further embodiment, max(I(x,y)) cannot be obtained when calculating the aforementioned spike event R(x,y). Therefore, it is only applicable to embodiments other than the contour enhancement type.

[0120] Also, the latest pixel value is stored in the corresponding memory unit in the frame image memory space. For example, in Figure 13, the pixel value of the pixel with coordinates (x, y) obtained from the sensor is 105, and the pixel value of the corresponding coordinate previously stored in the frame image memory space is 1. After differentiation, it is calculated that R(x,y) = 3. That is, the number of spike events 3 is stored in the corresponding memory unit in the spike event number memory space, and the pixel value of the corresponding coordinate stored in the frame image memory space is updated to 105. Other technical means are the same as those in the previous embodiment and are described here by citation method without further elaboration.

[0121] The present invention further discloses a spike sequence randomizer, the spike sequence randomizer comprising: A frame difference memory space and a spike event address memory space or a spike event number memory space, or Frame image memory space and spike event count memory space and, Including, The spike sequence randomizer is also configured to perform the aforementioned spike sequence randomization method (see Figures 8-13 and corresponding description), wherein the frame difference memory space is used for storing frame difference pixel values, the memory units included in the spike event address memory space are used for storing pixel coordinates to which the frame difference pixel values ​​correspond, the memory units included in the spike event number memory space are used for storing spike event numbers to which the frame difference pixel values ​​correspond, and the memory units included in the frame image memory space are used for storing the latest pixel values ​​obtained from the frame image sensor.

[0122] In one embodiment, the spike sequence randomizer is implemented as a frame image sensor interface.

[0123] 14 is a diagram showing a sensor integration means in a chip. The sensor includes a DVS or / and a frame image sensor, and the spike event sequence generated by the DVS or / and the target spike sequence generated after randomization are sent to the SNN processor for processing through a dynamic visual sensor interface or / and a frame image sensor interface, and the DVS or / and the frame image sensor and the SNN processor are located on different bare dies and integrated into the same chip, which can be called a brain-type chip, and is a brain-type chip integrated with sensing and computing.

[0124] FIG. 15 is a diagram showing a means for integrating sensors outside the chip. The sensors include a DVS or / and a frame image sensor, and through a dynamic vision sensor interface or / and a frame image sensor interface, a spike event sequence generated by the DVS or / and a target spike sequence after randomization is sent to an SNN processor for processing. The DVS or / and the frame image sensor and the SNN processor establish a communication connection via a cable, for example, based on the USB protocol.

[0125] In one embodiment, the chip supports a dual-eye sensor. Preferably, one eye is a DVS and the other eye is a frame image sensor. Preferably, one eye is an in-chip integrated sensor and the other eye is an out-of-chip sensor. These types of in-chip / out-of-chip sensors may be flexibly and freely combined as different embodiments.

[0126] FIG. 16 is a diagram showing details of spike sequence randomization in a further embodiment. In this embodiment, without being limited to the above-described visual signal collection, it relates to environmental signals such as audio, vibration, electrocardiogram signals, etc., and the present invention is not limited thereto.

[0127] The collected environmental signals are subjected to various software / hardware settings or processing. For example, an audio signal is passed through band-pass filters of a plurality of different bands to obtain a plurality of channel output signals. These channel identifiers are also called channel addresses, for example, CH-1, CH-6, etc., and correspond to the pixel coordinates / spike event addresses described above.

[0128] Thereafter, each channel obtains the corresponding number of spike events and stores them in the corresponding memory unit in the spike event number memory space. Here, obtaining the corresponding number of spike events for each channel means mapping the channel output signal as the corresponding number of spike events, and it may be any reasonable form of means, for example, an increment value based on the channel output signal, various spike sparsification means, etc. The present invention is not limited to a specific means.

[0129] Refer to the foregoing embodiments. According to the random number generated by the random number generation module, the memory unit corresponding to the random number is read. If a non-zero value is found, at least based on the channel address corresponding to the memory unit, a spike event is generated, and the number of spike events stored in the memory unit is updated, for example, by subtracting 1. Some of these spike events constitute a target spike sequence, and the target spike sequence is sent to the SNN processor. Of course, for the 0 value in the memory unit, there is no need to generate and emit spike events. For other technical features, the technical features of the foregoing embodiments are cited here in the form of citation, and will not be described further here.

[0130] FIG. 17 is a diagram showing the details of spike sequence randomization in such an embodiment. The difference from the embodiment shown in FIG. 16 is that after obtaining the number of spike events corresponding to each channel, the channel address is written into the same number of memory units in the spike event address memory space. As shown in FIG. 17, since the number of spike events of CH-1 is 2, it is written twice at the address, and for CH-2, since the corresponding number of spike events is 1, it is written once. Then, one random number is obtained from the random number generation module, and one address in the memory unit corresponding to the random number is read in the spike event address memory space. At least based on the read channel address, a spike event is generated and emitted. However, if the mapping result of the random number exceeds the spike event address memory space, or even exceeds the range of the memory units in which the valid channel addresses are written, there is no need to generate and emit spike events. Some of these spike events constitute a target spike sequence, and the target spike sequence is sent to the SNN processor. For other technical features, the technical features of the foregoing embodiments are cited here in the form of citation, and will not be described further here.

[0131] The present invention discloses a chip, which is a brain chip or a neuromorphic chip. The chip includes the aforementioned frame image sensor interface, or simultaneously includes a frame image sensor interface and an event sensor interface. The chip uses the self-adaptive spike generation method described in any one of the foregoing items, or includes the self-adaptive spike generation device described in any one of the foregoing items.

[0132] The present invention further discloses an electronic device, such as an electronic device such as a smart home appliance or a smartphone. It includes the aforementioned brain chip or spike sequence randomization device and is used to perform real-time monitoring and smart response to environmental signals. Due to the ultra-low power consumption characteristics of the brain chip and the like, it is possible to realize always-on smart inference. By configuring one or more specific inference capabilities in the SNN processor, the electronic device side can be given smart information processing capabilities.

[0133] In addition, the present invention further discloses an electronic device including the aforementioned chip, which is used for low power consumption, real-time, high performance, and responding to environmental signals. The power consumption of the means of the present invention is extremely low, suitable for edge smart computing, and applicable to fields such as smart home, IoT, autonomous driving, and smart toys.

[0134] The present invention has been described with reference to specific features and examples of the present invention. However, various changes, combinations, and substitutions can be made without departing from the present invention. The protection scope of the present invention is not limited to the specific examples of the steps, devices, manufacturing, substance compositions, devices, methods, and steps described in the specification. It is intended that these methods and modules are related, interdependent, cooperate with each other, and may be further implemented within one or more products and methods in the front / back stages.

[0135] Therefore, the specification and the drawings should be regarded as merely introducing some embodiments of the technical means limited by the appended claims. Accordingly, the appended claims should be interpreted based on the principle of maximum reasonable interpretation, intending to cover all changes, variations, combinations or equivalents within the scope of the present invention disclosure as much as possible, and at the same time, a way of interpretation that is inconsistent with common sense should be avoided.

[0136] To achieve better technical effects or due to the needs of some applications, those skilled in the art may further modify the technical means based on the present invention. However, even if some of these improvements / designs have creativity and / or progressiveness, as long as they rely on the technical concept of the present invention and cover the technical features defined by the claims, such technical means should also fall within the protection scope of the present invention.

[0137] There may be alternative technical features for some of the technical features mentioned in the appended claims, or the order of some technical processes and the order of material compositions can be reconfigured. After knowing the present invention, those of ordinary skill in the art can easily conceive of some of these substitution means, or change the order of technical processes and the order of material compositions, and then use substantially the same means to solve substantially the same technical problems and achieve substantially the same technical effects. Therefore, even if the above means and / or order are clearly limited in the claims, all of these modifications, changes and substitutions should fall within the protection scope of the claims according to the principle of equivalence.

[0138] If each method step or module described in the embodiments disclosed in this specification is combined, it can be implemented by hardware, software, or a combination of both. To clearly explain the compatibility between hardware and software, the steps and compositions of each embodiment have been generally described according to functions in the above description. Whether these functions are ultimately executed in the form of hardware or software is determined by the specific application of technical means or design constraints. A person of ordinary skill in the art can use different methods to implement the functions described, but such implementation should not be considered outside the scope of protection sought by the present invention.

Claims

1. A method for converting a frame image into a spike event, comprising: comparing the differences between frame images to obtain frame difference pixel values for each pixel coordinate point in the frame difference; obtaining the number of spike events corresponding to the pixel coordinates from the frame difference pixel values; generating a randomized target spike sequence from the number of spike events; determining whether to adjust the generation frame rate or frame difference frequency of the frame image based on at least the overall frame difference pixel values of the frame difference or / and the corresponding overall number of spike events; The method as claimed in claim 1, characterized by comprising the above steps.

2. When the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is less than a first threshold, the frame rate or frame difference frequency is decreased, or / and when it is greater than a second threshold, the frame rate or frame difference frequency is increased, or When the sum of the number of spike events corresponding to the frame difference, or the sum of the number of spike events satisfying a predetermined condition, or the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition is less than a first threshold, the frame rate or frame difference frequency is decreased, or / and when it is greater than a second threshold, the frame rate or frame difference frequency is increased. The method according to claim 1, characterized by the above.

3. When the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is less than a first ratio, the frame rate or frame difference frequency is decreased, or / and when it is greater than a second ratio, the frame rate or frame difference frequency is increased. The first ratio and the second ratio are respectively the ratios of the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition in the whole frame difference pixels, or When the count value of the number of frame difference pixels for which the number of spike events satisfies a predetermined condition is smaller than a first ratio, the frame rate or the frame difference frequency is decreased, and / or when it is larger than a second ratio, the frame rate or the frame difference frequency is increased, wherein the first ratio and the second ratio are each a ratio of the count value of the number of frame difference pixels for which the number of spike events satisfies a predetermined condition to the whole number of frame difference pixels. The method according to claim 1.

4. After decreasing the frame rate or shortening a first time interval of the frame difference frequency, the frame rate or the frame difference frequency is restored to a default value, and / or within a second time interval, when the sum of the frame difference pixel values and / or the sum of the number of spike events are both smaller than a first threshold value, the frame rate or the frame difference frequency is decreased. The method according to claim 1.

5. Determine whether the sum of the frame difference pixel values, or the sum of the frame difference pixel values for which the pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels for which the pixel values satisfy a predetermined condition is larger than a third threshold value. If it is larger, the step of obtaining the number of spike events corresponding to the pixel coordinates from the frame difference pixel values is not further executed, or Determine whether the sum of the frame differences corresponding to the number of spike events, or the sum of the number of spike events that satisfy a predetermined condition, or the count value of the number of frame difference pixels for which the number of spike events satisfies a predetermined condition is larger than a third threshold value. If it is larger, the step of generating a randomized target spike sequence from the number of spike events is not further executed. The method according to claim 1, further comprising.

6. An apparatus for converting a frame image into spike events, A frame difference module that compares the differences between frame images and obtains the frame difference pixel values of each pixel coordinate point in the frame difference; and A spike event number generation module that generates the number of spike events corresponding to the pixel coordinates from the frame difference pixel values. A randomization module that generates a randomized target spike sequence from the number of spike events, A first determination module that determines whether to adjust the generation frame rate or frame difference frequency of the frame image from at least the overall frame difference pixel value of the frame difference or / and the corresponding overall number of spike events, The apparatus, characterized by including the above.

7. When the sum of the frame difference pixel values, or the sum of the frame difference pixel values whose pixel values satisfy a predetermined condition, or the count value of the number of frame difference pixels whose pixel values satisfy a predetermined condition is less than a first threshold value, the frame rate or frame difference frequency is decreased, or / and when it is greater than a second threshold value, the frame rate or frame difference frequency is increased, or, When the sum of the frame differences corresponding to the number of spike events, or the sum of the number of spike events that satisfy a predetermined condition, or the count value of the number of frame difference pixels whose spike event numbers satisfy a predetermined condition is less than a first threshold value, the frame rate or frame difference frequency is decreased, or / and when it is greater than a second threshold value, the frame rate or frame difference frequency is increased. The apparatus according to claim 6, characterized by the above.

8. The apparatus according to claim 6, further including a second determination module that determines whether the sum of the frame difference pixel values or / and the sum of the number of spike events is greater than a third threshold value, and when it is greater, ends the randomization operation for the spike events.

9. A brain-type chip, Using the method according to any one of claims 1 to 5, or, Including the apparatus according to any one of claims 6 to 8 The brain-type chip, characterized by the above.

10. In an electronic device, Including the apparatus according to any one of claims 6 to 8, or, Including the brain chip according to claim 9 The electronic device characterized by this.

Images