Shadow data enhancement method and device, training method, storage medium and chip

Abstract

This invention discloses a shadow data enhancement method and apparatus, training method, storage medium, and chip. To address the issue of unwanted shadow data generated in image sensors due to shadow noise in real-world environments, this invention adds shadow data to sample data before training a spiking neural network. This includes determining pixel regions that meet preset conditions based on the sample data to obtain a target range; and selecting at least one pixel outside the target range as the shadow center point for shadow data enhancement. Through this method, diverse training data containing shadow noise is obtained, enabling the spiking neural network to adapt to complex and variable real-world working environments, exhibiting better accuracy and stability. This invention is suitable for event cameras or neuromorphic computing fields.

Description

Technical Field

[0001] This invention relates to a shadow data augmentation method and apparatus, training method, storage medium and chip, specifically to a data augmentation method, training method, storage medium and chip for making spiking neural networks more accurate, robust and universal. Background Technology

[0002] Spiking neural networks (SNNs), hailed as the third generation of neural networks, mimic the workings of the brain, exhibiting event-driven characteristics and rich spatiotemporal dynamics, while boasting low computational cost and low power consumption. It's worth noting that neuromorphic hardware or neuromorphic chips utilize a non-von Neumann architecture, meaning they do not perform traditional mathematical / program function calculations based on computer programs.

[0003] Event-based imaging devices are a novel type of biomimetic visual sensor, such as event cameras, dynamic vision sensors (DVS, DAVIS), and neuromorphic vision sensors based on event imaging fusion. The following discussion will focus on event cameras, but is not limited to them. Unlike traditional frame image sensors (such as APS sensors), event cameras do not capture images at a fixed rate. Each pixel operates independently, and based on the perceived changes in light intensity, when the change exceeds a certain threshold, it outputs an ON event (increased light intensity) or an OFF event (decreased light intensity). For details, please refer to existing technology 1: EP3731516A1.

[0004] Event cameras capture changes / motion information in a scene. The output typically includes a timestamp of the event (accurate to µs / ns), the pixel coordinates (x, y) of the event, and the polarity of the event (whether the light intensity brightens or darkens, or the pixel's photovoltage value, i.e., grayscale value). The polarity of the event can be ignored in some cases. Since event cameras generate events based on changes in light intensity, their output only shows positive or negative values, not intensity. For artificial neural networks and deep neural networks, this makes it impossible to analyze the cause of the event, potentially providing incorrect feature information and affecting training results.

[0005] A perception and computing scheme combining event cameras (or pulse sequences obtained by interpolating frames) and spiking neural networks (SNNs) offers a low-power (down to milliwatts) and high-real-time (down to microseconds) integrated sensing and computing solution for edge computing, IoT, and other terminal scenarios, enabling terminal intelligence without internet connectivity. Event streams generated by event / spike-based imaging devices are currently the most suitable datasets for SNN applications. However, current event camera datasets, such as Neuromorphic-MNIST and DVS-Gesture, are relatively small in scale and have limited application scenarios. Developing large-scale SNN training datasets that fit real-world usage scenarios remains a challenge for the industry.

[0006] The imaging principle of an event camera is that each pixel receives light from the corresponding spatial location of an entity, generating event data based on changes in light. It is highly sensitive to changes in light, thus easily capturing minute external disturbances as noise events. In particular, when a light source shines on a user, if a shadow is cast on the ground where the user is standing or on a nearby table / wall / mirror / glass, and this shadow is within the event camera's field of view, when the user performs an action, the shadow on the ground or wall will also generate event data. The position and size of the shadow are affected by the distance and angle between the user, the light source, and the sensor (also known as the lens). In practical applications, because shadows are so frequently seen within the field of view, the stability and robustness of inference accuracy in neuromorphic hardware deployed with network configuration parameters obtained from the training dataset are not ideal, easily leading to inference errors. Furthermore, due to the complexity and variability of the environment, those skilled in the art expect neuromorphic hardware to have the same accuracy as the trained model in the actual environment, as well as better environmental adaptability. The recorded training data cannot cover all possible shadow positions and sizes. Therefore, it is necessary to perform data augmentation on the training data regarding shadows to obtain diverse training data containing shadows, so that the model has better robustness and environmental adaptability. Summary of the Invention

[0007] To solve or alleviate some or all of the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A shadow data augmentation method, considering the influence of shadow noise, adds shadow data to sample data, including the following steps:

[0009] Based on sample data, determine the pixel regions that meet the preset conditions to obtain the target range;

[0010] Select at least one pixel outside the target area as the shadow center point for shadow data enhancement;

[0011] The preset condition is one of the following:

[0012] i) Within a continuous region, the pixel value of any pixel unit or the number of pulse events generated is greater than or equal to the first threshold.

[0013] ii) Within a continuous region, the number of pixel units whose pixel value or the number of generated pulse events is greater than or equal to the first threshold is greater than or equal to the second threshold.

[0014] In some embodiments, the pixel region that generates shadow noise is obtained based on a preset shadow size and / or shape, with the shadow center point as the center.

[0015] The pixel values ​​of all pixel units or the number of shadow events generated within the pixel region that generates shadow noise are set to a first value, and the pixel region that generates shadow noise is rotated by a preset angle to obtain shadow data.

[0016] In some embodiments, the pixel region that generates shadow noise is obtained based on a preset shadow size and / or shape, with the shadow center point as the center.

[0017] The pixel values ​​of more than half of the pixel units in the pixel area that generates shadow noise or the number of shadow events generated are randomly set to a first value, and the pixel area that generates shadow noise is rotated by a preset angle to obtain shadow data.

[0018] In some embodiments, the first value is less than the first threshold.

[0019] In some embodiments, the first value is greater than or equal to a third threshold.

[0020] In some embodiments, if the sample data is not in event frame format, the sample data is preprocessed to convert it into an event frame; the target range is obtained based on the pixel value of the pixel unit in the event frame or the number of pulse events generated.

[0021] In some embodiments, the shadow size is adjusted in one of the following ways to adjust the pixel region that generates shadow noise:

[0022] i) The size of the pixel region that generates shadow noise is inversely proportional to the distance between the shadow and the event camera; the distance is the horizontal and / or vertical coordinate distance.

[0023] ii) The size of the pixel region that produces shadow noise is inversely proportional to the size of the light source;

[0024] iii) The size of the pixel region that produces shadow noise is inversely proportional to the perpendicularity of the user, action, or object to the reflective surface.

[0025] In some embodiments, the magnitude of the first value is proportional to the speed of movement of the user, action, or object.

[0026] In some embodiments, within the pixel region where shadow noise is generated, the number of pixel units set to a first value is proportional to the speed of movement of the user, action, or object.

[0027] In some embodiments, shadow data is added to the event frame data corresponding to the sample data to complete shadow data enhancement, or the pixel value of the pixel unit in the sample data corresponding to the shadow data coordinates is directly updated based on the shadow data.

[0028] In some embodiments, the shape is circular or rectangular.

[0029] In some embodiments, within the field of view, at least one pixel outside the target area on the left, right, or bottom side is selected as the shadow center point.

[0030] An apparatus for data augmentation using shadows, comprising:

[0031] The judgment module determines the pixel regions that meet preset conditions based on the sample dataset, thereby obtaining the target range;

[0032] The selection module, coupled to the judgment module, selects a pixel outside the target range as the center point of the shadow, and selects the size and shape of the shadow;

[0033] An enhancement module, coupled to the selection module, enhances shadow data based on parameters determined by the selection module, with the shadow center point as the center.

[0034] The preset condition is one of the following:

[0035] i) Within a continuous region, the pixel value of any pixel unit or the number of pulse events generated is greater than or equal to the first threshold.

[0036] ii) Within a continuous region, the number of pixel units whose pixel value or the number of generated pulse events is greater than or equal to the first threshold is greater than or equal to the second threshold.

[0037] In some embodiments, the enhancement module, based on parameters determined by the selection module, obtains the pixel region that generates shadow noise, centered on the shadow center point.

[0038] The pixel values ​​of more than half of the pixel units in the pixel region that generates shadow noise or the number of shadow events generated are set to a first value, and the pixel region that generates shadow noise is rotated by a preset angle to obtain shadow data.

[0039] In some embodiments, the first value is less than the first threshold.

[0040] In some embodiments, the first value is greater than or equal to a third threshold.

[0041] In some embodiments, the magnitude of the first value is proportional to the speed of movement of the user, action, or object.

[0042] In some embodiments, within the pixel region where shadow noise is generated, the number of pixel units set to a first value is proportional to the speed of movement of the user, action, or object.

[0043] In some embodiments, if the sample data is not in event frame format, the apparatus for data augmentation using shadows further includes a preprocessing module;

[0044] The preprocessing module is coupled between the sample data and the judgment module and is used to convert the sample data into event frames.

[0045] A spiking neural network training method, based on the shadow data augmentation method described above, performs at least one data augmentation on the sample data of the spiking neural network training set; and completes the training of the spiking neural network based on the augmented training set.

[0046] A storage medium storing computer code, by executing the computer code, to implement the shadow data augmentation method as described above.

[0047] A chip includes a spiking neural network processor, the spiking neural network processor being configured with optimal configuration parameters obtained using the spiking neural network training method as described above.

[0048] In some embodiments, the chip is a neuromorphic chip or a neuromorphic chip, and has an event triggering mechanism.

[0049] In some embodiments, the chip includes an image sensor that is integrated with or coupled to the spiking neural network processor via an interface.

[0050] An electronic product having a chip as described above.

[0051] Some or all of the embodiments of the present invention have the following beneficial technical effects:

[0052] 1) This invention augments the training data with respect to shadows to obtain diverse training data containing shadows, enabling the spiking neural network to adapt to complex and ever-changing real-world working environments and achieve better accuracy and stability.

[0053] 2) Based on the situation where shadows exist within the sensor's field of view in practical application scenarios, this invention specifically enhances the data related to shadows. The method is simple, easy to implement, and effectively solves the corresponding problems.

[0054] 3) This invention has advantages when the amount of training data is small. Based on the relative position of the user and the event camera, the relative position (distance and angle) of the user, the light source and the reflective surface, and the action speed, rich and diverse training data can be obtained.

[0055] Further beneficial effects will be described in the preferred embodiments.

[0056] The technical solutions / features disclosed above are intended to summarize the technical solutions and features described in the Detailed Embodiments section, and therefore the scope of the description may not be entirely the same. However, these new technical solutions disclosed in this section are also part of the numerous technical solutions disclosed in this invention document. The technical features disclosed in this section, together with the technical features disclosed in the subsequent Detailed Embodiments section and some contents in the drawings not explicitly described in the specification, disclose more technical solutions in a reasonable combination.

[0057] The technical solution formed by combining all the technical features disclosed at any position in this invention is used to support the summary of the technical solution, the modification of the patent document, and the disclosure of the technical solution. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the shadow phenomenon in a certain embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of spiking neural network training and parameter deployment mapping in a certain embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of shadow data under different conditions in a certain embodiment of the present invention;

[0061] Figure 4 This is a schematic diagram of shadow data in another embodiment of the present invention;

[0062] Figure 5 This is a flowchart of shadow data enhancement in a certain embodiment of the present invention;

[0063] Figure 6 These are several examples of shadow data augmentation in a certain embodiment of the present invention;

[0064] Figure 7 This is a schematic diagram of shadow data enhancement in a preferred embodiment of the present invention;

[0065] Figure 8 This is a block diagram of shadow data enhancement in a certain embodiment of the present invention. Detailed Implementation

[0066] Since it is impossible to exhaustively describe all alternative solutions, the key points of the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Other technical solutions and details not disclosed in detail below generally belong to technical objectives or features that can be achieved by conventional means in the art, and due to space limitations, they will not be described in detail here.

[0067] Unless it refers to division, the " / " in any position in this invention represents logical "OR". The serial numbers "first", "second", etc., in any position in this invention are merely descriptive distinguishing marks and do not imply an absolute temporal or spatial order, nor do they imply that terms prefixed with such serial numbers necessarily refer to different things than the same terms prefixed with other modifiers.

[0068] This invention describes various key points used to combine into various specific embodiments, which will be incorporated into various methods and products. In this invention, even if a key point is described only when introducing a method / product solution, it means that the corresponding product / method solution also explicitly includes that technical feature.

[0069] The description of the existence or inclusion of a step, module, or feature at any location in this invention does not imply that such existence is exclusive or unique. Those skilled in the art can obtain other embodiments by supplementing the technical solutions disclosed in this invention with other technical means. The embodiments disclosed in this invention are generally for the purpose of disclosing preferred embodiments, but this does not imply that opposite embodiments of the preferred embodiments are excluded by this invention. As long as such opposite embodiments at least solve one of the technical problems of this invention, they are intended to be covered by this invention. Based on the key points described in the specific embodiments of this invention, those skilled in the art can substitute, delete, add, combine, or change the order of certain technical features to obtain a technical solution that still follows the concept of this invention. These solutions that do not depart from the technical concept of this invention are also within the protection scope of this invention.

[0070] Spiking neural network (SNN): A type of event-driven neuromorphic chip, it is a third-generation artificial neural network with rich spatiotemporal dynamics, diverse encoding mechanisms, event-driven characteristics, low computational cost, and low power consumption. This invention does not specifically limit the type of spiking neural network; any neural network driven by pulse signals or events can be applied to the sound source localization method provided in this invention. The spiking neural network can be built according to the actual application scenario, such as spiking convolutional neural network (SCNN) or spiking recurrent neural network (SRNN).

[0071] Event cameras, also known as event-driven image sensors or dynamic vision sensors (DVS), are based on this principle. Some technical solutions fuse these cameras with traditional frame image pixels, resulting in sensors that can output both events and pixel brightness. Examples include DAVIS and ATIS sensors. These event-based sensors (EBS) are collectively referred to as event imaging devices in this invention, and they belong to the category of sensors. This invention uses an event camera as an example to disclose a shadow data enhancement scheme.

[0072] Figure 1 This is a schematic diagram of the shadow phenomenon in a certain embodiment of the present invention, including sample data and real-time data obtained by signal acquisition in actual environments where shadows exist. Sample data is typically insufficient and recorded in specific scenarios; the data is typical, and the image quality is usually good. However, in actual indoor environments, the shadows of user movements are ubiquitous. The presence of shadows causes the event camera to generate unwanted events (or pulses), as shown in the shadow data. These unwanted events present false "motion" or result in extremely poor image quality, thus affecting the difficulty and capability of subsequent information processing.

[0073] Because neuromorphic datasets for SNN training are still in their early stages, current SNN sample datasets are not only small but also only geared towards a few single tasks. Furthermore, datasets recorded for current tasks are even more limited by sample size, funding, and time constraints. This invention, based on shadow data augmentation, expands the training data to obtain a richer dataset. Moreover, the resulting dataset closely matches real-world usage scenarios (actions and light sources may overlap). Configuration parameters trained on this dataset can not only handle complex task scenarios and have better universality but also improve the inference accuracy and stability of SNNs or neuromorphic hardware.

[0074] Figure 2This is a schematic diagram of spiking neural network training and parameter deployment mapping in a certain embodiment of the present invention. The original sample data is augmented with shadow data to obtain a dataset with shadow data. The spiking neural network is trained using this dataset. Since the position and size of the shadow are related to the action, the relative position of the light source and the sensor, the present invention performs shadow data augmentation for various situations, obtaining multiple datasets, and using one or more of these datasets to train the spiking neural network. Training involves obtaining the optimal network configuration parameters for a given set of sample data (training set or test set). Based on these optimal network configuration parameters, the SNN can output a result matching the input sample for any given input. Note that chips or neuromorphic hardware configured with spiking neural networks do not follow the traditional "von Neumann architecture," and therefore do not have the concept of "instructions" (computer programs). Typically, SNN training and inference execution are performed on different devices. Network training is performed on a training device (such as a high-performance CPU or GPU), and the SNN on the training device is a simulation of the SNN on the chip. The configuration parameters obtained during training are mapped to the SNN on the chip. Furthermore, SNN training and execution can be performed on the same device; for example, the neuromorphic hardware includes training and inference modes, performing on-chip learning.

[0075] Extensive testing using actual chips reveals that shadow events can be broadly categorized into two shapes: circular (including elliptical) or rectangular (including square or oblong).

[0076] Typically, the positions of the light source and the reflecting surface are fixed, assuming that both the action and the shadow are within the field of view of the event camera. For small light sources (or point light sources), the action may completely block the light source, resulting in a large and relatively blurry shadow. This means that the pixel area generating shadow noise events is large, but the number of shadow noise events generated per pixel is small. As the distance between the action and the light source increases, the shadow gradually decreases in size and its sharpness increases, meaning that the pixel area generating shadow noise events is small, but the number of shadow noise events generated per pixel is large. For large light sources, the probability of shadows appearing is lower than that of small light sources. Under the same conditions (e.g., the hand and the light source at the same distance and angle), the range and sharpness of shadow imaging under large light sources are smaller than those under small light sources.

[0077] Furthermore, experiments have confirmed that shadow phenomena also exist for event frames or pulse sequences converted from difference frame images generated by traditional frame image sensors. This is because the shadow phenomenon originates from the image formation of the action on the reflective surface, which changes the light intensity at the corresponding position on the reflective surface and affects the pixel value of the pixel at the shadow position on the reflective surface.

[0078] Figure 3 This is a schematic diagram of shadow data under different conditions in a certain embodiment of the present invention, wherein, Figure 3(a) in the image shows the imaging effect after the event generated by the event camera is compressed into frames when there is no shadow. Figure 3 (b) and Figure 3 (c) shows the image effect after event frame compression generated by the event camera when the user's arm is waving, with the light source in front of the user and the shadow formed on the wall behind the user. It can be clearly seen that the shadow on the wall causes the event camera to generate shadow noise events or shadow data. These unwanted shadow data result in extremely poor image quality, which in turn affects the difficulty and capability of subsequent information processing. Figure 3 (b) and Figure 3 (c) shows the image effect after frame compression produced by the event camera when the user swings their arm at different speeds under the same environment (the relative positions (distance and angle) of the light source, user, and reflective surface are basically the same), with the light source in front of the user and casting a shadow on the wall behind the user. Figure 3 (b) shows the image effect of the wall shadow after frame compression of the shadow noise event generated by the event camera when the speed is slower. Figure 3 (c) in the figure shows the image effect of the wall shadow after frame compression of the shadow noise event generated by the event camera when the speed is relatively fast.

[0079] Figure 4 This is a schematic diagram of shadow data in another embodiment of the present invention. Shadow data, also known as shadow noise data, includes information such as the size and shape of the pixel region that generates the shadow noise event, and the amount of shadow noise generated within that pixel region. Shadow data is limited not only by the relative distance between the shadow and the event camera, but also by factors such as the size of the light source, the relative distance and angle between the hand, the light source, and the reflecting surface, and the speed / intensity of movement. Under otherwise identical conditions, this document exemplarily describes the influence of various environmental factors on shadow data as follows:

[0080] Similar to how the relative distance between the user / action and the event camera affects the image quality after frame compression of an event triggered by an action, within the event camera's field of view, the greater the relative distance between the shadow and the event camera, the smaller the shadow noise in the frame compression image, and vice versa. Here, the relative distance refers to the distance on the horizontal and / or vertical axes.

[0081] The farther the user / action is from the light source, the smaller and clearer the shadow on the reflective surface. Correspondingly, the pixel range of shadow noise generated by the event camera is smaller, and the events within the noise area are more concentrated (i.e., the more shadow noise events generated by pixel units within the noise area). Conversely, the closer the user is to the light source, the larger the range of shadow noise and the sparser the events within the noise area.

[0082] The larger the light source, the smaller the pixel range that produces shadow noise, and vice versa.

[0083] The greater the range of shadow noise, the more parallel the user's hand is to the reflective surface. The greater the perpendicularity of the action to the reflective surface, the smaller the range of shadow noise.

[0084] The faster the movement, the faster the light changes, and the more and denser the events generated within the noise area, such as... Figure 3 As shown in (c), as the motion slows down, the degree of change in light decreases, and events become fewer and sparser within the noise area, as... Figure 3 As shown in (b) of the diagram.

[0085] Furthermore, tests show that the number of noise events or pixel values ​​generated by pixels in the pixel area corresponding to the shadow are basically the same, and less than the number of valid events or pixel values ​​generated by pixels in the pixel area corresponding to the real action.

[0086] Figure 5 This is a flowchart of shadow data enhancement in a certain embodiment of the present invention, including the following steps:

[0087] S101, Preprocessing.

[0088] The sample dataset can be a set of difference frames, a set of event frames, or an event stream. If the sample data is an event stream, the event stream data is compressed into frames. If the sample data is in the form of difference frames, the difference frames are converted into event frame data. If the sample data is event frame data, this step is skipped.

[0089] For event streams directly generated by event cameras, events within a time window are compressed to generate event frames. An event frame is a collection of events within a time window. Event frames can be generated in single-channel or dual-channel manner. A single-channel event frame involves superimposing all timestamps of events at each pixel within the time window, ignoring event polarity, or selectively superimposing ON (increased light intensity) or OFF (decreased light intensity) events at each pixel's timestamps. A dual-channel event frame involves superimposing all timestamps of events at each pixel within the time window according to their respective polarities. This invention does not limit the methods for generating various event frames.

[0090] S102. Determine the pixel region that meets the preset conditions to obtain the target range. For example, search for a set of pixels that meet the preset conditions on event frame data based on a sample dataset.

[0091] In one embodiment, the target range is determined based on the pixel values ​​of each pixel. On the event frame data, a continuous region is searched where the pixel value of a pixel unit or the number of events generated is greater than or equal to a first threshold to obtain the target range. This target range is the pixel region corresponding to the actual action captured by the event camera.

[0092] In a preferred embodiment, the target range is defined as the set of pixels in an area where the number of pixels whose pixel values ​​are greater than or equal to a first threshold is greater than or equal to a second threshold.

[0093] S103. Based on the target range, find the region to add shadow data and perform shadow data enhancement.

[0094] The pixel at any coordinate on the left, right, or bottom of the target area is designated as the center point for shadow data augmentation. Furthermore, to increase the amount of sample data, one or more pixel units at any location outside the target area can be selected as the center point for shadow data augmentation. This embodiment uses a single shadow center point as an example, but is not limited to this.

[0095] Centered on the center point of the shadow, and based on a preset shadow size and / or shape, the pixel values ​​or the number of shadow events generated by all pixel units within the pixel region corresponding to the shadow (hereinafter referred to as the shadow region) are set to a first value. Then, the shadow region is rotated by a preset angle to generate a set of shadow data. In other embodiments, the pixel values ​​or the number of pulse events generated by more than half of the pixel units within the pixel region corresponding to the shadow are randomly set to the first value, and the pixel region corresponding to the shadow is rotated by a preset angle to obtain the shadow data.

[0096] The shadow data can be stored first, and then added to the sample data graph to complete the shadow data enhancement graph. Alternatively, the sample data graph can be updated directly to obtain the enhanced dataset. This invention does not limit the scope of the invention.

[0097] By adjusting the size or shape parameters of the shadow noise, a new set of shadow data is obtained. The training set sample data is then augmented using one or more sets of shadow data corresponding to different shadow sizes or shapes to obtain one or more new sample datasets.

[0098] Since the distance between the shadow and the event camera is inversely proportional to the size of the shadow data, the farther the shadow is from the event camera, the smaller the pixel area of ​​the shadow noise data, and vice versa. Furthermore, factors such as the size of the light source, the relative distance and angle between the hand and the light source during hand movements, and the extent to which the light source is obstructed all affect the shape, size, and clarity of the shadow. Therefore, this invention randomly selects a center point and / or sets the shadow size and / or shape to simulate various different usage scenarios.

[0099] The shadow shape can be any shape, such as a circle (including a near-circular shape) or a rectangle (square or elongated shape). To simplify operation, the shadow shape of this invention is preferably a square or a rectangle, but this is not a limitation.

[0100] In some embodiments, the shadow is defined as a rectangle. The width and height of the rectangle are randomly selected from preset width and height parameters. The pixel values ​​of at least half of the pixel units within the rectangular area or the number of shadow noise events generated are randomly set to be the same. The rectangle is then rotated by a preset angle to generate shadow data. Furthermore, this invention does not limit the order of the steps "setting the pixel values ​​of the pixels within the rectangular area or the number of shadow noise events generated to a first value" and "rotating the rectangle"; they can be performed sequentially or in parallel.

[0101] Based on sample data Figure 3 As shown in (a), if a larger first threshold is selected, the area of ​​the arm with obvious movement, i.e., the target range, is determined based on the first threshold. A position is selected below the target range as the center point for shadow data enhancement. The size of the rectangular shadow is set, the pixel value of all pixels within the rectangular area is set to the first value, and the rectangle is rotated by a preset angle to generate shadow data. Similarly, if a smaller first threshold is selected, the target range determined based on the first threshold is the entire area of ​​the human figure. Likewise, a position is selected to the lower right of the target range as the center point for shadow data enhancement. The size of the rectangular shadow is set, the pixel value of all pixels within the rectangular area is set to the first value, and the rectangle is rotated by a preset angle to generate shadow data. In a preferred embodiment, the first value is less than the first threshold.

[0102] By adjusting the size of the rectangle, scenes corresponding to shadows of different sizes are simulated; by adjusting the value of the first value, scenes corresponding to shadows of different intensities are simulated. In some embodiments, the first value is greater than or equal to a third threshold. Preferably, the first value is greater than or equal to the third threshold, and the first value is less than the first threshold.

[0103] Figure 6 These are several examples of shadow data enhancement in a certain embodiment of the present invention. The present invention determines a set of pixels that meet preset conditions, i.e., a target range. One or more pixel units at any location within the target range are selected as the center point for shadow data enhancement. Based on the relative position (distance and angle) between the action and the light source, and the distance between the shadow and the event camera, the size and shape of the shadow noise are set. With the shadow center point as the center, all pixels within the shadow shape (e.g., a rectangle) area are set to the same pixel value, and the shadow area is rotated by an angle to generate shadow data. Figure 6 (a) to Figure 6 (d) in the figure shows the shadow data augmentation performed under different conditions. The training data after shadow augmentation can not only match the shadow phenomenon that exists in actual use, but also cover a variety of possible situations, improve the diversity of training data, and make the training network have better robustness and universality.

[0104] Figure 7 This is a schematic diagram of shadow data enhancement in a preferred embodiment of the present invention. The event generated by the event camera includes coordinate information and time information, namely, the pixel coordinates of the event and the timestamp of the event. The two-bit coordinates (x, y) indicate that the sensor is a two-dimensional sensor; it can also be a one-dimensional sensor that generates the event, such as an audio sensor, a vibration sensor, or a more multi-dimensional sensor; the present invention is not limited to these. Specifically, the first pixel with coordinates (x, y) generates event e at time t, and event e-1 is generated at time t-1. The target range is obtained based on preset conditions, such as... Figure 7 As shown in (a) in the figure. The set of pixels whose number of events generated by each pixel unit is greater than or equal to the first threshold (5) is the target area. A pixel outside the target area is selected as the center point for shadow data augmentation, and the data is then processed as follows. Figure 7 The number of noise events generated by all pixels in the rectangular area shown in (b) is set to 4. The rectangular area is rotated to obtain shadow data, and the generated shadow data is added to the sample data to complete the shadow data enhancement.

[0105] The present invention also relates to a method for training a spiking neural network, which uses the data augmentation method described above to perform at least one data augmentation on the sample data of the spiking neural network training set; and completes the training of the spiking neural network based on the augmented training set.

[0106] Figure 8 This is a block diagram of shadow data enhancement in a certain embodiment of the present invention, including a judgment module, a selection module, and an enhancement module coupled in sequence. The judgment module determines pixel regions that meet preset conditions based on the sample dataset, obtaining a target range, for example, a continuous region where the pixel value of a search pixel unit is greater than or equal to a first threshold. The selection module selects a pixel point within the target range as the center point of the shadow noise and selects the shadow size and shape. The enhancement module, based on the parameters determined by the selection module, sets the same pixel value or number of shadow events for all pixels within the pixel region corresponding to the preset shadow size and shape around the center point of the shadow noise, rotates the pixel region generating the shadow noise by a preset angle to generate shadow data, and adds the generated shadow data to the sample data to complete the shadow data enhancement.

[0107] In a preferred embodiment, a preprocessing module is further included, coupled between the sample dataset and the judgment module, for converting the sample data into event frame data. If the sample data is event frame data, this processing is skipped.

[0108] The present invention also relates to a chip including a spiking neural network processor, the spiking neural network processor being configured with optimal configuration parameters obtained using the spiking neural network training method as described above.

[0109] In a preferred embodiment, the chip is a neuromorphic chip or a neuromorphic chip, which has event-triggered characteristics.

[0110] In another preferred embodiment, the chip includes an event-based imaging sensor and a spiking neural network processor, which are integrated together or coupled via an interface. The chip is equipped with network configuration parameters obtained by training on multiple sets (50 sets) of different shadow data augmented datasets. Using this chip to perceive and compute data in real-world application scenarios results in better chip performance, stability, and adaptability.

[0111] In another preferred embodiment, the chip includes a frame image sensor, which is integrated with or coupled to the spiking neural network processor via an interface.

[0112] The present invention also relates to an electronic product having the chip described above.

[0113] Although the invention has been described with reference to specific features and embodiments, various modifications, combinations, and substitutions can be made therein without departing from the invention. The scope of protection of this invention is not limited to the specific embodiments of processes, machines, manufactures, material compositions, apparatuses, methods, and steps described in the specification, and these methods and modules may also be implemented in one or more related, interdependent, cooperative, or upstream / downstream products or methods.

[0114] Therefore, the specification and drawings should be simply regarded as a description of some embodiments of the technical solutions defined by the appended claims, and thus the appended claims should be interpreted in accordance with the principle of the greatest reasonable interpretation, and are intended to cover as much as possible all modifications, variations, combinations or equivalents within the scope of the invention, while avoiding unreasonable interpretations.

[0115] To achieve better technical effects or for the needs of certain applications, those skilled in the art may make further improvements to the technical solution based on this invention. However, even if such improvements / designs are inventive and / or progressive, as long as they rely on the technical concept of this invention and cover the technical features defined in the claims, the technical solution should also fall within the protection scope of this invention.

[0116] The technical features mentioned in the appended claims may have alternative technical features, or the order of certain technical processes or material organization may be rearranged. Those skilled in the art, upon learning of this invention, will readily conceive of these alternative means, or alter the order of the technical processes or material organization, and then employ substantially the same means to solve substantially the same technical problems and achieve substantially the same technical effects. Therefore, even if the claims explicitly define the aforementioned means and / or order, these modifications, alterations, and substitutions should all fall within the scope of protection of the claims based on the principle of equivalents.

[0117] The method steps or modules described in the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application or design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered outside the scope of protection claimed by this invention.

Claims

1. A shadow data augmentation method, characterized in that: To account for the impact of shadow noise, shadow data is added to the sample data, including the following steps: Based on sample data, determine the pixel regions that meet the preset conditions to obtain the target range; Select at least one pixel outside the target area as the shadow center point for shadow data enhancement; The preset condition is one of the following: i) Within a continuous region, the pixel value of any pixel unit or the number of pulse events generated is greater than or equal to the first threshold. ii) Within a continuous region, the number of pixels whose pixel value or the number of generated pulse events is greater than or equal to the first threshold is greater than or equal to the second threshold.

2. The shadow data augmentation method according to claim 1, characterized in that: Centered on the shadow center point, and based on the preset shadow size and / or shape, obtain the pixel region that generates shadow noise; The pixel values ​​of all pixel units or the number of pulse events generated within the pixel region that generates shadow noise are set to a first value, and the pixel region that generates shadow noise is rotated by a preset angle to obtain shadow data.

3. The shadow data augmentation method according to claim 1, characterized in that: Centered on the shadow center point, and based on the preset shadow size and / or shape, obtain the pixel region that generates shadow noise; The pixel values ​​of more than half of the pixel units in the pixel area that generates shadow noise or the number of pulse events generated are randomly set to a first value, and the pixel area that generates shadow noise is rotated by a preset angle to obtain shadow data.

4. The shadow data augmentation method according to any one of claims 2 or 3, characterized in that: The first value is less than the first threshold.

5. The shadow data augmentation method according to claim 4, characterized in that: The first value is greater than or equal to the third threshold.

6. The shadow data augmentation method according to any one of claims 2 or 3, characterized in that: If the sample data is not in event frame format, preprocess the sample data to convert it into an event frame; obtain the target range based on the pixel value of the pixel unit in the event frame or the number of pulse events generated.

7. The shadow data augmentation method according to any one of claims 2 or 3, characterized in that, Adjust the shadow size using one of the following methods to adjust the pixel area that generates shadow noise: i) The size of the pixel region that generates shadow noise is inversely proportional to the distance between the shadow and the event camera; the distance is the horizontal and / or vertical coordinate distance. ii) The size of the pixel region that produces shadow noise is inversely proportional to the size of the light source; iii) The size of the pixel region that produces shadow noise is inversely proportional to the perpendicularity of the user, action, or object to the reflective surface.

8. The shadow data augmentation method according to claim 4, characterized in that: The magnitude of the first value is proportional to the speed of the user, action, or object.

9. The shadow data augmentation method according to claim 3, characterized in that: Within the pixel region where shadow noise is generated, the number of pixel units set to a first value is proportional to the speed of the user, action, or object's movement.

10. The shadow data augmentation method according to claim 6, characterized in that: The shadow data is added to the event frame data corresponding to the sample data to complete the shadow data enhancement, or the pixel value of the pixel unit in the sample data corresponding to the shadow data coordinates is directly updated based on the shadow data.

11. The shadow data augmentation method according to any one of claims 2 or 3, characterized in that: The shape is either circular or rectangular.

12. The shadow data augmentation method according to any one of claims 2 or 3, characterized in that: Within the field of view, select at least one pixel outside the target area on the left, right, or bottom as the shadow center point.

13. A shadow data augmentation device, characterized in that, include: The judgment module determines the pixel regions that meet preset conditions based on the sample dataset, thereby obtaining the target range; The selection module, coupled to the judgment module, selects a pixel outside the target range as the center point of the shadow, and selects the size and shape of the shadow; An enhancement module, coupled to the selection module, enhances shadow data based on parameters determined by the selection module, with the shadow center point as the center. The preset condition is one of the following: i) Within a continuous region, the pixel value of any pixel unit or the number of pulse events generated is greater than or equal to the first threshold. ii) Within a continuous region, the number of pixels whose pixel value or the number of generated pulse events is greater than or equal to the first threshold is greater than or equal to the second threshold.

14. The shadow data enhancement apparatus according to claim 13, characterized in that: The enhancement module, based on the parameters determined by the selection module, obtains the pixel region that generates shadow noise, centered on the shadow center point. The pixel values ​​of more than half of the pixel units in the pixel region that generates shadow noise or the number of pulse events generated are set to a first value, and the pixel region that generates shadow noise is rotated by a preset angle to obtain shadow data.

15. The shadow data enhancement apparatus according to claim 14, characterized in that: The first value is less than the first threshold.

16. The shadow data enhancement apparatus according to claim 15, characterized in that: The first value is greater than or equal to the third threshold.

17. The shadow data enhancement apparatus according to any one of claims 14 to 16, characterized in that: The magnitude of the first value is proportional to the speed of the user, action, or object.

18. The shadow data enhancement apparatus according to any one of claims 14 to 16, characterized in that: Within the pixel region where shadow noise is generated, the number of pixel units set to a first value is proportional to the speed of the user, action, or object's movement.

19. The shadow data enhancement apparatus according to any one of claims 13 to 16, characterized in that: If the sample data is not in event frame format, the shadow data enhancement device further includes a preprocessing module; The preprocessing module is coupled between the sample data and the judgment module and is used to convert the sample data into event frames.

20. A method for training a spiking neural network, characterized in that: Based on the shadow data augmentation method as described in any one of claims 1 to 12, at least one data augmentation is performed on the sample data of the spiking neural network training set; The spiking neural network was trained using the enhanced training set.

21. A storage medium, characterized in that: The storage medium stores computer code, characterized in that: by executing the computer code, the shadow data augmentation method as described in any one of claims 1 to 12 is implemented.

22. A chip, characterized in that: It includes a spiking neural network processor, which is deployed with optimal configuration parameters obtained using the spiking neural network training method as described in claim 20.

23. The chip according to claim 22, characterized in that: The chip is a neuromorphic chip or a neuromorphic chip, and has an event triggering mechanism.

24. The chip according to claim 22, characterized in that: The chip includes an image sensor, which is integrated with or coupled to the spiking neural network processor via an interface.

25. An electronic product, characterized in that: The electronic product is equipped with a chip as described in any one of claims 22 to 24.

Images