Data processing method and apparatus, electronic device, computer readable medium

Abstract

The present disclosure provides a data processing method, comprising: performing enhancement processing on original data collected by an event information collection device according to at least one data enhancement transformation, to obtain at least one piece of enhanced data, wherein each piece of the enhanced data corresponds to at least one data enhancement transformation; obtaining at least one piece of initial prediction result corresponding to at least one piece of to-be-predicted data according to the at least one piece of to-be-predicted data, wherein the at least one piece of to-be-predicted data comprises the at least one piece of enhanced data, or one piece of the original data and the at least one piece of enhanced data; and obtaining a target prediction result according to the at least one piece of initial prediction result. The present disclosure also provides a data processing device, an electronic device and a computer readable medium.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data processing method, a data processing apparatus, an electronic device, and a computer-readable medium. Background Technology

[0002] An event information acquisition device is an event-driven sensor that can output a data stream reflecting changes in the scene when an event occurs.

[0003] In some related technologies, the inference accuracy based on data collected by event information acquisition devices is not yet ideal. Summary of the Invention

[0004] This disclosure provides a data processing method, a data processing apparatus, an electronic device, and a computer-readable medium.

[0005] In a first aspect, embodiments of this disclosure provide a data processing method, including:

[0006] The raw data collected by the event information acquisition device is enhanced according to at least one data enhancement transformation to obtain at least one enhanced data channel, wherein each enhanced data channel corresponds to at least one data enhancement transformation.

[0007] At least one initial prediction result is obtained based on at least one source of data to be predicted, which corresponds one-to-one with the at least one source of data to be predicted. The at least one source of data to be predicted includes the at least one source of enhanced data, or one source of the original data and the at least one source of enhanced data.

[0008] The target prediction result is obtained based on at least one initial prediction result.

[0009] In some embodiments, the raw data includes multiple raw event data, each raw event data corresponding to an event; the raw event data includes coordinate components, time components, and polarity components, wherein the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event; the step of enhancing the raw data collected by the event information acquisition device according to at least one data enhancement transformation to obtain at least one channel of enhanced data includes:

[0010] Based on at least one combination of various data augmentation transformations, including at least one data augmentation transformation that transforms the coordinate components of the event data, at least one data augmentation transformation that filters the event data, and at least one data augmentation transformation that transforms the polarity components of the event data, each of the original event data in the original data is processed to obtain the at least one channel of augmented data, wherein each combination of augmentation transformations includes at least one data augmentation transformation.

[0011] In some embodiments, processing each of the original event data in the original data according to a data augmentation transformation that transforms the polarity components of the event data includes:

[0012] The polarity components of each of the original event data are transformed to obtain each of the enhanced event data constituting the enhanced data, wherein the polarity of the events corresponding to the enhanced event data is opposite to the polarity of the events corresponding to the original event data.

[0013] In some embodiments, processing each of the original event data in the original data according to a data augmentation transformation that transforms the coordinate components of the event data includes:

[0014] The coordinate components of each of the original event data in the original data are scaled and transformed; or

[0015] Perform a shift transformation on the coordinate components of each of the original event data in the original data; or

[0016] The coordinate components of each of the original event data in the original data are rotated and transformed.

[0017] In some embodiments, for any one of the original event data in the original data, scaling and transforming the coordinate components of each of the original event data in the original data includes:

[0018] The coordinate components of the original event data are scaled and transformed according to a preset ratio to obtain an enhanced event data that constitutes the enhanced data.

[0019] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0020] In some embodiments, for any one of the original event data in the original data, performing a shift transformation on the coordinate components of each of the original event data in the original data includes:

[0021] The coordinate components of the original event data are shifted and transformed according to the shift vector to obtain an enhanced event data that constitutes the enhanced data.

[0022] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0023] In some embodiments, for any one of the original event data in the original data, performing a rotation transformation on the coordinate components of each of the original event data in the original data includes:

[0024] The coordinate components of the original event data are rotated and transformed relative to the center coordinates according to a preset rotation angle to obtain an enhanced event data that constitutes the enhanced data.

[0025] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0026] In some embodiments, processing each of the original event data in the original data according to a data augmentation transformation that filters the event data includes:

[0027] Denoising processing is performed based on the time and coordinate components of each original event data in the original data; or

[0028] Redundancy removal is performed based on the time and coordinate components of each original event data in the original data; or

[0029] Event supplementation processing is performed based on the time and coordinate components of each original event data in the original data.

[0030] In some embodiments, for any one of the original event data in the original data, denoising processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0031] Detect whether there is event data falling within a preset spatiotemporal range centered on the original event data;

[0032] If no event data falls within a preset spatiotemporal range centered on the original event data, the original event data is deleted.

[0033] In some embodiments, for any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes:

[0034] Detect whether there are duplicate event data within a preset time range whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data;

[0035] If duplicate event data exists within the preset time range, the duplicate event data is deleted.

[0036] In some embodiments, for any one of the original event data in the original data, event supplementation processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0037] Supplementary event data is inserted at predetermined steps within a time interval centered on the time component of the original event data.

[0038] In some embodiments, inserting supplementary event data with a predetermined step size within a time interval centered on the time component of the original event data includes: detecting whether there is first original event data and second original event data; the first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment;

[0039] When the first original event data and the second original event data exist, at least one supplementary event data is inserted between the first original event data and the second original event data at a predetermined time step. The coordinate components and polarity components of the supplementary event data and the original event data are the same.

[0040] In some embodiments, the step of obtaining at least one initial prediction result corresponding one-to-one with at least one data to be predicted includes:

[0041] Generate the prediction frame data corresponding to each of the prediction data streams based on each prediction data stream.

[0042] The initial prediction results for each channel are obtained based on the frame data to be predicted for each channel.

[0043] In some embodiments, the step of generating predictable frame data corresponding to each of the predictable data streams includes:

[0044] The event stream composed of each path of data to be predicted is aggregated to generate event frame data;

[0045] The frame data to be predicted is generated by superimposing multiple sequentially adjacent event frame data.

[0046] In some embodiments, the step of obtaining the target prediction result based on the at least one initial prediction result includes:

[0047] The initial prediction results from each path are weighted and summed to obtain the target prediction result.

[0048] In some embodiments, prior to the step of obtaining the target prediction result based on the at least one initial prediction result, the data processing method further includes:

[0049] For at least one of the initial prediction results, perform the inverse transformation of the corresponding data augmentation transformation.

[0050] In a second aspect, embodiments of this disclosure provide a data processing apparatus, comprising:

[0051] A data augmentation module is used to augment the raw data collected by the event information acquisition device according to at least one data augmentation transformation to obtain at least one channel of augmented data, wherein each channel of augmented data corresponds to at least one of the data augmentation transformations.

[0052] The prediction module is used to obtain at least one initial prediction result corresponding to at least one prediction data according to at least one prediction data, wherein the at least one prediction data includes at least one augmented data, or one original data and at least one augmented data;

[0053] The fusion module is used to obtain the target prediction result based on the at least one initial prediction result.

[0054] In some embodiments, the raw data includes multiple raw event data, each of which corresponds to an event; the raw event data includes coordinate components, time components, and polarity components, wherein the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event.

[0055] The data augmentation module is used to process each of the original event data in the original data according to at least one combination of various data augmentation transformations, including at least one data augmentation transformation that transforms the coordinate components of the event data, at least one data augmentation transformation that filters the event data, and at least one data augmentation transformation that transforms the polarity components of the event data, to obtain the at least one channel of augmented data. Each combination of augmentation transformations includes at least one data augmentation transformation.

[0056] In some embodiments, in the data augmentation module, processing each of the original event data in the original data according to the data augmentation transformation that transforms the polarity components of the event data includes:

[0057] The polarity components of each of the original event data are transformed to obtain each of the enhanced event data constituting the enhanced data, wherein the polarity of the events corresponding to the enhanced event data is opposite to the polarity of the events corresponding to the original event data.

[0058] In some embodiments, in the data augmentation module, processing each of the original event data in the original data according to the data augmentation transformation that transforms the coordinate components of the event data includes:

[0059] The coordinate components of each of the original event data in the original data are scaled and transformed; or

[0060] Perform a shift transformation on the coordinate components of each of the original event data in the original data; or

[0061] The coordinate components of each of the original event data in the original data are rotated and transformed.

[0062] In some embodiments, in the data enhancement module, for any one of the original event data in the original data, scaling and transforming the coordinate components of each of the original event data in the original data includes:

[0063] The coordinate components of the original event data are scaled and transformed according to a preset ratio to obtain an enhanced event data that constitutes the enhanced data.

[0064] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0065] In some embodiments, in the data enhancement module, for any one of the original event data in the original data, performing a shift transformation on the coordinate components of each of the original event data in the original data includes:

[0066] The coordinate components of the original event data are shifted and transformed according to the shift vector to obtain an enhanced event data that constitutes the enhanced data.

[0067] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0068] In some embodiments, in the data enhancement module, for any one of the original event data in the original data, performing a rotation transformation on the coordinate components of each of the original event data in the original data includes:

[0069] The coordinate components of the original event data are rotated and transformed relative to the center coordinates according to a preset rotation angle to obtain an enhanced event data that constitutes the enhanced data.

[0070] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0071] In some embodiments, in the data augmentation module, processing each of the original event data in the original data according to the data augmentation transformation that filters the event data includes:

[0072] Denoising processing is performed based on the time and coordinate components of each original event data in the original data; or

[0073] Redundancy removal is performed based on the time and coordinate components of each original event data in the original data; or

[0074] Event supplementation processing is performed based on the time and coordinate components of each original event data in the original data.

[0075] In some embodiments, in the data enhancement module, for any one of the original event data in the original data, denoising processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0076] Detect whether there is event data falling within a preset spatiotemporal range centered on the original event data;

[0077] If no event data falls within a preset spatiotemporal range centered on the original event data, the original event data is deleted.

[0078] In some embodiments, in the data enhancement module, for any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes:

[0079] Detect whether there are duplicate event data within a preset time range whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data;

[0080] If duplicate event data exists within the preset time range, the duplicate event data is deleted.

[0081] In some embodiments, in the data enhancement module, for any one of the original event data in the original data, event supplementation processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0082] Supplementary event data is inserted at predetermined steps within a time interval centered on the time component of the original event data.

[0083] In some embodiments, in the data augmentation module, inserting supplementary event data with a predetermined step size within a time interval centered on the time component of the original event data includes: detecting whether there is first original event data and second original event data; the first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment;

[0084] When the first original event data and the second original event data exist, at least one supplementary event data is inserted between the first original event data and the second original event data at a predetermined time step. The coordinate components and polarity components of the supplementary event data and the original event data are the same.

[0085] In some embodiments, the prediction module includes:

[0086] A framing unit is used to generate predictable frame data corresponding to each channel of predictable data based on each channel of predictable data.

[0087] The prediction unit is used to obtain the initial prediction result of each channel based on the data of each channel of the frame to be predicted.

[0088] In some embodiments, the framing unit is used to aggregate the event stream composed of each of the data to be predicted to generate event frame data; and to superimpose multiple sequentially adjacent event frame data to generate the data to be predicted.

[0089] In some embodiments, the fusion module is used to perform a weighted summation of the initial prediction results from each path to obtain the target prediction result.

[0090] In some embodiments, the data processing apparatus further includes:

[0091] The inverse transformation module is used to perform the inverse transformation of the corresponding data augmentation transformation on at least one of the at least one initial prediction results.

[0092] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0093] One or more processors;

[0094] A memory having stored one or more programs, which, when executed by one or more processors, cause the one or more processors to implement any of the data processing methods described in the first aspect of the present disclosure.

[0095] One or more I / O interfaces are connected between the processor and the memory and configured to enable information interaction between the processor and the memory.

[0096] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements any of the data processing methods described in the first aspect of this disclosure.

[0097] In the data processing method provided in this embodiment, the raw data acquired by the event information acquisition device is subjected to data augmentation transformation to obtain at least one augmented data, and inference prediction is performed based on at least one augmented data and / or the raw data to obtain at least one initial prediction result. Finally, the target prediction result is obtained based on at least one initial prediction result. Through data augmentation transformation, augmented data under different conditions can be obtained from the raw data, so that the inference has better generalization performance, and / or the influence of redundant data in the raw data can be eliminated, the influence of noise in the raw data can be eliminated, and the data missing in the raw data can be compensated to improve the continuity in the time dimension. Thus, the target prediction result that integrates the various initial prediction results is more accurate, that is, the inference accuracy based on the data acquired by the event information acquisition device is improved.

[0098] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0099] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0100] Figure 1 This is a flowchart of a data processing method according to an embodiment of this disclosure;

[0101] Figure 2 This is a schematic diagram of a data processing architecture according to an embodiment of this disclosure;

[0102] Figure 3 This is a flowchart of some steps in another data processing method according to an embodiment of this disclosure;

[0103] Figure 4 This is a flowchart of some steps in another data processing method according to an embodiment of this disclosure;

[0104] Figure 5 This is a flowchart of some steps in another data processing method according to an embodiment of this disclosure;

[0105] Figure 6 This is a flowchart of some steps in another data processing method according to an embodiment of this disclosure;

[0106] Figure 7 This is a flowchart of some steps in another data processing method according to an embodiment of this disclosure;

[0107] Figure 8 This is a flowchart of some steps in another data processing method according to an embodiment of this disclosure;

[0108] Figure 9 This is a flowchart of some steps in another data processing method according to an embodiment of this disclosure;

[0109] Figure 10 This is a block diagram of a data processing apparatus according to an embodiment of the present disclosure;

[0110] Figure 11 This is a block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0111] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0112] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0113] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0114] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0115] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0116] Firstly, referring to Figure 1 This disclosure provides a data processing method, including:

[0117] In step S100, the raw data collected by the event information acquisition device is enhanced according to at least one data enhancement transformation to obtain at least one enhanced data channel, wherein each enhanced data channel corresponds to at least one data enhancement transformation.

[0118] In step S200, at least one initial prediction result corresponding to at least one prediction data is obtained based on at least one prediction data, wherein the at least one prediction data includes at least one enhanced data, or one original data and at least one enhanced data;

[0119] In step S300, the target prediction result is obtained based on the at least one initial prediction result.

[0120] In some embodiments, the event information acquisition device includes a dynamic vision sensor (DVS). A DVS is an event-driven sensor that outputs a data stream reflecting changes in pixel brightness at a specific location within a scene when an event occurs, and has become an important component of computer vision. In some embodiments, the dynamic vision sensor is an event-driven camera.

[0121] Figure 2 This is a schematic diagram of a data processing architecture according to an embodiment of this disclosure. Figure 2The data processing architecture includes multiple branches, each corresponding to at least one data augmentation transformation. For example, such as... Figure 2 As shown, the model training architecture includes branches corresponding to data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n, and also includes branches that concatenate at least two of data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n. Concatenating at least two of data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n means that at least two of data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n are sequentially performed on the original data in that branch to obtain one path of augmented data. When the original data collected by the event information acquisition device passes through each branch, the corresponding data augmentation transformation of that branch is executed to obtain each path of augmented data. It should be noted that, in this embodiment of the disclosure, the data processing architecture may also include only one branch, which corresponds to one type of data augmentation transformation or multiple concatenated data augmentation transformations.

[0122] In this embodiment of the disclosure, after the original data is augmented, the original data can be retained as one channel of data to be predicted; alternatively, the original data can be discarded, and only the augmented data obtained can be used as the data to be predicted. This embodiment of the disclosure does not impose any special limitations on this.

[0123] In this embodiment, data augmentation transformation enhances the original data, enabling the generation of augmented data under different conditions. The augmented data, compared to the original data, more comprehensively reflects various aspects of the events acquired by the event information acquisition device. For example, through data augmentation transformation, at least one of the following can be obtained: augmented data with different event polarity changes, augmented data with different event coverage sizes, augmented data with different event locations, and augmented data after event coverage rotation and / or mirror flipping. Furthermore, the obtained augmented data can eliminate the influence of redundant data in the original data, and / or eliminate the influence of noise in the original data, and / or compensate for data gaps in the original data, thereby improving continuity in the time dimension. This embodiment does not impose any specific limitations on these aspects.

[0124] like Figure 2 As shown, in step S200, the network layer performs inference on the data to be predicted for each path to obtain the initial prediction results for each path. The network layer can be configured according to the application scenario, function, etc., and this embodiment does not impose any special limitations on it. For example, the network layer can be a neural network capable of target recognition; the network layer can also be a neural network capable of target detection; the network layer can also be a neural network capable of target counting. In some embodiments, the network layer is a spiking neural network.

[0125] In this embodiment, the network layer used for inference on the data to be predicted can be a neural network trained using any training method, and this embodiment does not impose any special limitations on it. For example, it can be trained using augmented data obtained by augmenting the original training data collected by the event information acquisition device. The augmentation process can include any augmentation transformation method in this embodiment, and this disclosure does not limit it. Since the original training data has been augmented, the resulting training data has high complexity. Training the neural network with high-complexity training data can solve the problem of overfitting during training. Furthermore, the trained neural network can correspond to more possible scenarios during inference, thereby enabling the trained neural network to have better processing accuracy.

[0126] In the data processing method provided in this embodiment, the raw data acquired by the event information acquisition device is subjected to data augmentation transformation to obtain at least one augmented data, and inference prediction is performed based on at least one augmented data and / or the raw data to obtain at least one initial prediction result. Finally, the target prediction result is obtained based on at least one initial prediction result. Through data augmentation transformation, augmented data under different conditions can be obtained from the raw data, so that the inference has better generalization performance, and / or the influence of redundant data in the raw data can be eliminated, the influence of noise in the raw data can be eliminated, and the data missing in the raw data can be compensated to improve the continuity in the time dimension. Thus, the target prediction result that integrates the various initial prediction results is more accurate, that is, the inference accuracy based on the data acquired by the event information acquisition device is improved.

[0127] The following description uses an event information acquisition device as an example of a dynamic visual sensor to illustrate the embodiments of this disclosure.

[0128] In some embodiments, the raw data acquired by the dynamic vision sensor consists of multiple event data in the form of (x, y, t, p) quadruples, with each event data corresponding to one event. Here, the coordinate component (x, y) represents the coordinates of the event; the time component t represents the time of the event's occurrence; and the polarity component p represents the polarity of the event, i.e., whether the illumination is enhanced or weakened; for example, 0 indicates weakened illumination, and 1 indicates enhanced illumination.

[0129] Accordingly, in some embodiments, the raw data includes multiple raw event data, each raw event data corresponding to one event; the raw event data includes coordinate components, time components, and polarity components, wherein the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event; refer to Figure 3 Step S100 includes:

[0130] In step S110, each of the original event data in the original data is processed according to at least one combination of various data augmentation transformations, including at least one data augmentation transformation that transforms the coordinate components of the event data, at least one data augmentation transformation that filters the event data, and at least one data augmentation transformation that transforms the polarity components of the event data, to obtain the at least one channel of augmented data. Each combination of augmentation transformations includes at least one data augmentation transformation.

[0131] It should be noted that the data augmentation transformations that make up different combinations of augmentation transformations are not all the same.

[0132] In this embodiment of the disclosure, processing each of the original event data in the original data according to a data augmentation transformation that transforms the polarity component of the event data can obtain augmented data with different changes in illumination intensity corresponding to the target; processing each of the original event data in the original data according to at least one data augmentation transformation that transforms the coordinate component of the event data can obtain at least one of multiple augmented data such as augmented data with different sizes of the target within the visual range, augmented data with different positions of the target within the visual range, and augmented data after the target is rotated and / or mirrored within the visual range; processing each of the original event data in the original data according to at least one data augmentation transformation that filters the event data can eliminate the influence of redundant data in the original data, and / or eliminate the influence of noise in the original data, and / or compensate for data missing data in the original data, thereby improving the continuity in the time dimension.

[0133] This disclosure does not impose any special limitations on how the data augmentation transformation, which transforms the polarity components of the event data, is applied to process each of the original event data in the original data.

[0134] In some embodiments, refer to Figure 4 In step S110, the data augmentation transformation that transforms the polarity components of the event data to process each of the original event data in the original data includes:

[0135] In step S111, the polarity components of each of the original event data are transformed to obtain each of the enhanced event data constituting the enhanced data, wherein the polarity of the events corresponding to the enhanced event data is opposite to the polarity of the events corresponding to the original event data.

[0136] It should be noted that, in this embodiment of the disclosure, step S111 is performed on each original event data in the original data to obtain multiple enhanced event data.

[0137] In some embodiments, step S111 can be performed as follows:

[0138] i. Read raw event data (x, y, t, p) from the raw data;

[0139] ii. If the polar component p of the original event data (x, y, t, p) is 0, it is changed to 1 to obtain the enhanced event data corresponding to the original event data; if the polar component p of the original event data (x, y, t, p) is 1, it is changed to 0 to obtain the enhanced event data corresponding to the original event data. The coordinate components of the enhanced event data are the same as those of the original event data, and the time components of the enhanced event data are the same as those of the original event data.

[0140] The above polarity component transformation is performed sequentially on all the original event data of the original data.

[0141] This disclosure does not specifically limit how each of the original event data in the original data is processed according to a data augmentation transformation that transforms at least one coordinate component of the event data.

[0142] In some embodiments, refer to Figure 5 In step S110, the data augmentation transformation that transforms the coordinate components of the event data to process each of the original event data in the original data includes:

[0143] In step S112, the coordinate components of each of the original event data in the original data are scaled and transformed; or

[0144] In step S113, the coordinate components of each of the original event data in the original data are shifted and transformed; or

[0145] In step S114, the coordinate components of each of the original event data in the original data are rotated and transformed.

[0146] In some embodiments, the range of amplitude of the enhancement processing of the original data through at least one data augmentation change in steps S112, S113, and S114 can be controlled. For example, when adjusting the size of the target within the visual range through data augmentation transformation, the range of target size change can be controlled; when changing the position of the target within the visual range through data augmentation transformation, the range of target displacement can be controlled; when rotating the target within the visual range through data augmentation transformation, the rotation angle of the target can be controlled. In embodiments of this disclosure, the range of amplitude of the enhancement processing is controlled using an enhancement amplitude vector.

[0147] This disclosure does not impose any special limitations on how to scale and transform the coordinate components of each of the original event data in the original data.

[0148] In some embodiments, for any one of the original event data in the original data, scaling the coordinate components of each of the original event data in the original data includes: scaling the coordinate components of the original event data according to a preset ratio to obtain an enhanced event data constituting the enhanced data; and deleting the enhanced event data if the coordinate components of the enhanced event data exceed a preset data boundary.

[0149] In some embodiments, scaling transformation of the coordinate components of any one of the original event data in the original data can be performed as follows:

[0150] i. Determine the preset scale α and the preset data boundary (X1, Y1). The preset data boundary represents the size of the raw data acquired by the dynamic vision sensor.

[0151] ii. Read raw event data (x, y, t, p) from the raw data;

[0152] iii. Multiply the coordinate components (x, y) sequentially by a preset ratio α, and round up to obtain the new coordinate components. This results in enhanced event data. The time component of the enhanced event data is the same as the time component of the original event data, and the polarity component of the enhanced event data is the same as the polarity component of the original event data.

[0153] iv. If and / or Delete the enhanced event data.

[0154] The above coordinate scaling transformation is performed sequentially on all the original event data of the original data to obtain enhanced event data with multiple coordinate components that do not exceed the preset data boundary.

[0155] This disclosure does not impose any special limitations on how to perform shift transformations on the coordinate components of each of the original event data in the original data.

[0156] In some embodiments, for any one of the original event data in the original data, shifting and transforming the coordinate components of each of the original event data in the original data includes: shifting and transforming the coordinate components of the original event data according to a shift vector to obtain an enhanced event data constituting the enhanced data; and deleting the enhanced event data if the coordinate components of the enhanced event data exceed a preset data boundary.

[0157] In some embodiments, shifting and transforming the coordinate components of any one of the original event data in the original data can be performed as follows:

[0158] i. Determine the shift vector (Δx2, Δy2) and the preset data boundary (X2, Y2). The preset data boundary represents the size of the raw data acquired by the dynamic vision sensor.

[0159] ii. Read raw event data (x, y, t, p) from the raw data;

[0160] iii. Add the coordinate components (x, y) of the original event data and the shift vector (Δx2, Δy2) to obtain the coordinate components (x+Δx2, y+Δy2) of the enhanced event data;

[0161] iv. If x + Δx² > X² and / or y + Δy² > Y², delete this enhanced event data.

[0162] The above coordinate shift transformation is performed sequentially on all the original event data of the original data to obtain multiple enhanced event data whose coordinate components do not exceed the preset data boundary.

[0163] This disclosure does not impose any special limitations on how to perform rotation transformation on the coordinate components of each of the original event data in the original data.

[0164] In some embodiments, for any one of the original event data in the original data, rotating and transforming the coordinate components of each of the original event data in the original data includes: rotating and transforming the coordinate components of the original event data relative to the center coordinate by a preset rotation angle to obtain an enhanced event data constituting the enhanced data; and deleting the enhanced event data if the coordinate components of the enhanced event data exceed the preset data boundary.

[0165] In some embodiments, performing a rotation transformation on the coordinate components of any one of the original event data in the original data can be performed as follows:

[0166] i. Determine the preset rotation angle Determine the shift vector (Δx3, Δy3) to determine the preset boundary (X3, Y3) and center coordinates.

[0167] ii. Read raw event data (x, y, t, p) from the raw data;

[0168] iii. Calculate the coordinate components (x, y) of the original event data about the center coordinates using angle calculations. The angle θ and the distance R will As the rotation angle, the coordinate components (x1, y1) of the enhanced event data are determined using the rotation angle and the distance R;

[0169] iv. If x1 > X3 and / or y1 > Y3, delete this enhanced event data.

[0170] The above rotation transformation is performed sequentially on all the original event data of the original data to obtain enhanced event data with multiple coordinate components that do not exceed the preset data boundary.

[0171] This disclosure does not specifically limit how each of the original event data in the original data is processed according to at least one data augmentation transformation that filters event data.

[0172] In some embodiments, refer to Figure 6 In step S110, processing each of the original event data in the original data according to the data augmentation transformation that filters the event data includes:

[0173] In step S115, denoising processing is performed based on the time and coordinate components of each of the original event data in the original data; or

[0174] In step S116, redundancy removal is performed based on the time and coordinate components of each of the original event data in the original data; or

[0175] In step S117, event supplementation processing is performed based on the time and coordinate components of each of the original event data in the original data.

[0176] In this embodiment of the disclosure, denoising processing enables the augmented data to eliminate the influence of noise compared to the original data, thereby improving inference accuracy. This embodiment of the disclosure does not specifically limit how denoising processing is performed based on the time and coordinate components of each of the original event data in the original data.

[0177] In some embodiments, for any one of the original event data in the original data, denoising processing based on the time component and coordinate component of each of the original event data in the original data includes: detecting whether there is event data falling within a preset spatiotemporal range centered on the original event data; and deleting the original event data if there is no event data falling within the preset spatiotemporal range centered on the original event data.

[0178] It should be noted that, in this embodiment of the disclosure, event data falling within a preset time and space range means that the time component of the event data satisfies a preset time interval and the coordinate component satisfies a preset coordinate interval.

[0179] In some embodiments, denoising based on the time and coordinate components of any one of the original event data in the original data can be performed as follows:

[0180] i. Determine (Δx4, Δy4) and ΔT4;

[0181] ii. Read a raw event data (x, y, t, p) from the raw data, and use (x-Δx4, y-Δy4, t-ΔT4) to (x+Δx4, y+Δy4, t+ΔT4) to represent the preset spatiotemporal range centered on the raw event data;

[0182] iii. Detect whether there is event data falling within the spatiotemporal range of (x-Δx4, y-Δy4, y-ΔT4) to (x+Δx4, y+Δy4, t+ΔT4);

[0183] iv. If there is no event data falling within the spatiotemporal range of (x-Δx4, y-Δy4, t-ΔT4) to (x+Δx4, y+Δy4, t+ΔT4), delete the original event data.

[0184] The above denoising process is performed sequentially on all raw event data in the original data to remove the raw event data that serves as noise.

[0185] In this embodiment of the disclosure, the redundancy removal process enables the augmented data to eliminate the influence of redundant data compared to the original data, thereby improving inference accuracy. This embodiment of the disclosure does not specifically limit how the redundancy removal process is performed based on the time and coordinate components of each of the original event data in the original data.

[0186] In some embodiments, for any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes: detecting whether there is duplicate event data within a preset time range whose coordinate component is the same as the coordinate component of the original event data and whose polarity component is the same as the polarity component of the original event; and deleting the duplicate event data if the duplicate event data exists within the preset time range.

[0187] In some embodiments, redundancy removal processing based on the time component and coordinate component of any one of the original event data in the original data can be performed as follows:

[0188] i. Determine ΔT5;

[0189] ii. Read raw event data (x, y, t, p) from the raw data;

[0190] iii. Detect whether there are duplicate event data within a preset time range of (t-ΔT5, t+ΔT5) whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data;

[0191] iv. If duplicate event data exists within the preset time range, delete the duplicate event data.

[0192] Perform the above deduplication process on all raw event data in the original data in sequence to remove duplicate event data from the original data.

[0193] It should be noted that the above redundancy removal process can remove redundant data generated by DVS dense acquisition, thereby improving inference accuracy.

[0194] This disclosure does not impose any special limitations on how to perform event supplementation processing based on the time and coordinate components of each of the original event data in the original data.

[0195] In some embodiments, for any one of the original event data in the original data, event supplementation processing based on the time component and coordinate component of each of the original event data in the original data includes: inserting supplementary event data with a predetermined step size in a time interval centered on the time component of the original event data.

[0196] In some embodiments, inserting supplementary event data with a predetermined step size within a time interval centered on the time component of the original event data includes: detecting whether there is first original event data and second original event data; the first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment;

[0197] When the first original event data and the second original event data exist, at least one supplementary event data is inserted between the first original event data and the second original event data at a predetermined time step. The coordinate components and polarity components of the supplementary event data and the original event data are the same.

[0198] In some embodiments, event supplementation processing based on the time component and coordinate component of any one of the original event data in the original data can be performed as follows:

[0199] i. Determine the preset time increment ΔT6 and the predetermined time step dt, where dt < ΔT6;

[0200] ii. Read a raw event data (x, y, t, p) from the raw data. The time interval centered on the time component of the raw event data can be represented as (t-ΔT6, t+ΔT6).

[0201] iii. If the presence of first original event data (x, y, t-ΔT6, p) and second original event data (x, y, t+ΔT6, p) is detected in the original data, supplementary event data (x, y, t-ΔT6, t+ΔT6) are inserted in the time interval (t-ΔT6, t+ΔT6) centered on the original event data (x, y, t-ΔT6, p) between the first original event data (x, y, t-ΔT6, p) and the second original event data (x, y, t+ΔT6, p) (x, y, t-ΔT6+1×dt, p), (x, y, t-ΔT6+2×dt, p)...(x, y, t+ΔT6-1×dt, p) in the time interval (t-ΔT6, t+ΔT6, p) centered on the original event data (x, y, t-ΔT6, p) between the first original event data (x, y, t-ΔT6, p) and the second original event data (x, y, t+ΔT6-1×dt, p).

[0202] Perform the above event supplementation processing on all raw event data of the original data in sequence, and insert supplementary event data into the raw event data of the original data.

[0203] It should be noted that the above event supplementation processing can supplement missing event data in the case of sparse DVS acquisition, avoid frame breaks, and thus improve inference accuracy.

[0204] like Figure 2 As shown, the multi-channel augmented data processed by data augmentation generates frame data through the framing layer, and then inputs it into the network layer for prediction processing.

[0205] Accordingly, in some embodiments, reference is made to Figure 7 Step S200 includes:

[0206] In step S210, predictable frame data corresponding to each channel of predictable data is generated based on each channel of predictable data.

[0207] In step S220, the initial prediction results of each channel are obtained based on the data of each channel of the frame to be predicted.

[0208] The frame data to be predicted may include at least one event frame.

[0209] In some embodiments, refer to Figure 8 Step S210 includes:

[0210] In step S211, the event stream composed of each path of the data to be predicted is aggregated to generate event frame data;

[0211] In step S212, multiple consecutively adjacent event frame data are superimposed to generate the frame data to be predicted.

[0212] In some embodiments, step S210 may be performed as follows:

[0213] The input augmentation data for the framing layer consists of multiple augmentation event data in the form of (x, y, t, p) quadruples. The augmentation event data within the time window Δt forms the event stream E. t′ ={e i |e i =[x i y i ,t′,p i The framing layer aggregates the event stream and outputs event frame data X. t =q(E t′ The aggregation function q(·) can be any of the following: nonpolar function aggregation, cumulative aggregation, and logical operation aggregation; this embodiment does not impose any special limitation on this. The event frame data is in the form (c, h, w), where c represents the channel, h represents the height of the frame data, and w represents the width of the frame data. Finally, T adjacent event frame data are superimposed in chronological order to form the frame data to be predicted in the form (T, c, h, w).

[0214] In some embodiments, when the frame data to be predicted includes multiple event frame data, it can be processed for prediction by a network layer based on a spiking neural network. A network layer based on a spiking neural network can better understand the temporal information between multiple event frame data in the frame data to be predicted, formed by superimposing them in chronological order, thereby improving prediction accuracy.

[0215] like Figure 2 As shown, the fusion layer obtains the target prediction result based on the multiple initial prediction results. This embodiment of the disclosure does not impose any special limitations on how the target prediction result is obtained from the multiple initial prediction results.

[0216] In some embodiments, the step of obtaining the target prediction result based on the multiple initial prediction results includes: weighting and summing the initial prediction results from each path to obtain the target prediction result.

[0217] In some embodiments, the step of obtaining a target prediction result based on the multiple initial prediction results includes: calculating the average value of each of the initial prediction results as the target prediction result.

[0218] like Figure 2 As shown, an inverse transform layer is also included between the network layer and the fusion layer to perform the inverse transform based on the data augmentation transform. In tasks sensitive to object size and position, this avoids the impact of data augmentation transform on inference accuracy.

[0219] In some embodiments, refer to Figure 9 Before the step of obtaining the target prediction result based on the at least one initial prediction result, the data processing method further includes:

[0220] In step S400, for at least one of the at least one initial prediction results, the corresponding data augmentation transformation inverse transformation is performed.

[0221] In some embodiments, after the original data is enhanced by at least one data enhancement change in steps S112, S113, and S114, an inverse transformation is performed in step S400.

[0222] Secondly, referring to Figure 10 This disclosure provides a data processing apparatus, including:

[0223] Data augmentation module 110 is used to augment the raw data collected by the event information acquisition device according to at least one data augmentation transformation to obtain at least one channel of augmented data, wherein each channel of augmented data corresponds to at least one of the data augmentation transformations.

[0224] Prediction module 120 is used to obtain at least one initial prediction result corresponding to at least one prediction data according to at least one prediction data, wherein the at least one prediction data includes at least one enhanced data, or one original data and at least one enhanced data;

[0225] The fusion module 130 is used to obtain the target prediction result based on the at least one initial prediction result.

[0226] In some embodiments, the raw data includes multiple raw event data, each of which corresponds to an event; the raw event data includes coordinate components, time components, and polarity components, wherein the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event.

[0227] The data augmentation module 110 is used to process each of the original event data in the original data according to at least one combination of various data augmentation transformations, including at least one data augmentation transformation that transforms the coordinate components of the event data, at least one data augmentation transformation that filters the event data, and at least one data augmentation transformation that transforms the polarity components of the event data, to obtain the at least one channel of augmented data. Each combination of augmentation transformations includes at least one data augmentation transformation.

[0228] In some embodiments, in the data augmentation module 110, processing each of the original event data in the original data according to the data augmentation transformation that transforms the polarity components of the event data includes:

[0229] The polarity components of each of the original event data are transformed to obtain each of the enhanced event data constituting the enhanced data, wherein the polarity of the events corresponding to the enhanced event data is opposite to the polarity of the events corresponding to the original event data.

[0230] In some embodiments, in the data augmentation module 110, processing each of the original event data in the original data according to the data augmentation transformation that transforms the coordinate components of the event data includes:

[0231] The coordinate components of each of the original event data in the original data are scaled and transformed; or

[0232] Perform a shift transformation on the coordinate components of each of the original event data in the original data; or

[0233] The coordinate components of each of the original event data in the original data are rotated and transformed.

[0234] In some embodiments, in the data enhancement module 110, for any one of the original event data in the original data, scaling transformation of the coordinate components of each of the original event data in the original data includes:

[0235] The coordinate components of the original event data are scaled and transformed according to a preset ratio to obtain an enhanced event data that constitutes the enhanced data.

[0236] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0237] In some embodiments, in the data enhancement module 110, for any one of the original event data in the original data, performing a shift transformation on the coordinate components of each of the original event data in the original data includes:

[0238] The coordinate components of the original event data are shifted and transformed according to the shift vector to obtain an enhanced event data that constitutes the enhanced data.

[0239] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0240] In some embodiments, in the data enhancement module 110, for any one of the original event data in the original data, performing a rotation transformation on the coordinate components of each of the original event data in the original data includes:

[0241] The coordinate components of the original event data are rotated and transformed relative to the center coordinates according to a preset rotation angle to obtain an enhanced event data that constitutes the enhanced data.

[0242] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0243] In some embodiments, in the data augmentation module 110, processing each of the original event data in the original data according to the data augmentation transformation that filters the event data includes:

[0244] Denoising processing is performed based on the time and coordinate components of each original event data in the original data; or

[0245] Redundancy removal is performed based on the time and coordinate components of each original event data in the original data; or

[0246] Event supplementation processing is performed based on the time and coordinate components of each original event data in the original data.

[0247] In some embodiments, in the data enhancement module 110, for any one of the original event data in the original data, denoising processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0248] Detect whether there is event data falling within a preset spatiotemporal range centered on the original event data;

[0249] If no event data falls within a preset spatiotemporal range centered on the original event data, the original event data is deleted.

[0250] In some embodiments, in the data enhancement module 110, for any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes:

[0251] Detect whether there are duplicate event data within a preset time range whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data;

[0252] If duplicate event data exists within the preset time range, the duplicate event data is deleted.

[0253] In some embodiments, in the data enhancement module 110, for any one of the original event data in the original data, event supplementation processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0254] Supplementary event data is inserted at predetermined steps within a time interval centered on the time component of the original event data.

[0255] In some embodiments, the data enhancement module 110 inserts supplementary event data at predetermined steps within a time interval centered on the time component of the original event data, including:

[0256] Detect the existence of first and second original event data; the first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment;

[0257] When the first original event data and the second original event data exist, at least one supplementary event data is inserted between the first original event data and the second original event data at a predetermined time step. The coordinate components and polarity components of the supplementary event data and the original event data are the same.

[0258] In some embodiments, the prediction module 120 includes:

[0259] A framing unit is used to generate predictable frame data corresponding to each channel of predictable data based on each channel of predictable data.

[0260] The prediction unit is used to obtain the initial prediction result of each channel based on the data of each channel of the frame to be predicted.

[0261] In some embodiments, the framing unit is used to aggregate the event stream composed of each of the data to be predicted to generate event frame data; and to superimpose multiple sequentially adjacent event frame data to generate the data to be predicted.

[0262] In some embodiments, the fusion module 130 is used to perform a weighted summation of the initial prediction results from each path to obtain the target prediction result.

[0263] In some embodiments, the data processing apparatus further includes:

[0264] The inverse transformation module is used to perform the inverse transformation of the corresponding data augmentation transformation on at least one of the at least one initial prediction results.

[0265] Thirdly, referring to Figure 11 This disclosure provides an electronic device, including:

[0266] One or more processors 201;

[0267] The memory 202 stores one or more programs, which, when executed by one or more processors, enable the one or more processors to implement any of the data processing methods described in the first aspect of the embodiments of this disclosure.

[0268] One or more I / O interfaces 203 are connected between the processor and the memory and configured to enable information exchange between the processor and the memory.

[0269] Among them, processor 201 is a device with data processing capabilities, including but not limited to central processing unit (CPU); memory 202 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); I / O interface (read-write interface) 203 is connected between processor 201 and memory 202, and can realize information interaction between processor 201 and memory 202, including but not limited to data bus (Bus).

[0270] In some embodiments, the processor 201, memory 202, and I / O interface 203 are interconnected via bus 204, and thus connected to other components of the computing device.

[0271] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements any of the data processing methods described in the first aspect of this disclosure.

[0272] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0273] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A data processing method, comprising: The raw data collected by the event information acquisition device is enhanced by at least one data augmentation transformation to obtain at least one enhanced data stream, wherein each enhanced data stream corresponds to at least one data augmentation transformation; wherein the event information acquisition device is an event-driven sensor; the raw data includes multiple raw event data, each raw event data corresponding to one event; the raw event data includes coordinate components, time components, and polarity components, wherein the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event; At least one initial prediction result is obtained based on at least one source of data to be predicted, which corresponds one-to-one with the at least one source of data to be predicted. The at least one source of data to be predicted includes the at least one source of enhanced data, or one source of the original data and the at least one source of enhanced data. The target prediction result is obtained based on at least one initial prediction result.

2. The data processing method according to claim 1, wherein the step of enhancing the raw data collected by the event information acquisition device according to at least one data enhancement transformation to obtain at least one channel of enhanced data includes: Based on at least one combination of various data augmentation transformations, including at least one data augmentation transformation that transforms the coordinate components of the event data, at least one data augmentation transformation that filters the event data, and at least one data augmentation transformation that transforms the polarity components of the event data, each of the original event data in the original data is processed to obtain the at least one channel of augmented data, wherein each combination of augmentation transformations includes at least one data augmentation transformation.

3. The data processing method according to claim 2, wherein, The data augmentation transformation, which transforms the polarity components of the event data, processes each of the original event data in the original data, including: The polarity components of each of the original event data are transformed to obtain each of the enhanced event data constituting the enhanced data, wherein the polarity of the events corresponding to the enhanced event data is opposite to the polarity of the events corresponding to the original event data.

4. The data processing method according to claim 2, wherein, The data augmentation transformation, which transforms the coordinate components of the event data, processes each of the original event data in the original data, including: The coordinate components of each of the original event data in the original data are scaled and transformed; or Perform a shift transformation on the coordinate components of each of the original event data in the original data; or The coordinate components of each of the original event data in the original data are rotated and transformed.

5. The data processing method according to claim 4, wherein, For any one of the original event data in the original data, scaling transformation of the coordinate components of each of the original event data in the original data includes: The coordinate components of the original event data are scaled and transformed according to a preset ratio to obtain an enhanced event data that constitutes the enhanced data. If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

6. The data processing method according to claim 4, wherein, For any one of the original event data in the original data, performing a shift transformation on the coordinate components of each of the original event data in the original data includes: The coordinate components of the original event data are shifted and transformed according to the shift vector to obtain an enhanced event data that constitutes the enhanced data. If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

7. The data processing method according to claim 4, wherein, For any one of the original event data in the original data, the rotation transformation of the coordinate components of each of the original event data in the original data includes: The coordinate components of the original event data are rotated and transformed relative to the center coordinates according to a preset rotation angle to obtain an enhanced event data that constitutes the enhanced data. If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

8. The data processing method according to claim 2, wherein, The data augmentation transformation based on the filtered event data is used to process each of the original event data in the original data, including: Denoising processing is performed based on the time and coordinate components of each original event data in the original data; or Redundancy removal is performed based on the time and coordinate components of each original event data in the original data; or Event supplementation processing is performed based on the time and coordinate components of each original event data in the original data.

9. The data processing method according to claim 8, wherein, For any one of the original event data in the original data, the noise reduction process based on the time component and coordinate component of each original event data in the original data includes: Detect whether there is event data falling within a preset spatiotemporal range centered on the original event data; If no event data falls within a preset spatiotemporal range centered on the original event data, the original event data is deleted.

10. The data processing method according to claim 8, wherein, For any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes: Detect whether there are duplicate event data within a preset time range whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data; If duplicate event data exists within the preset time range, the duplicate event data is deleted.

11. The data processing method according to claim 8, wherein, For any one of the original event data in the original data, the event supplementation processing based on the time component and coordinate component of each original event data in the original data includes: Supplementary event data is inserted at predetermined time steps within a time interval centered on the time component of the original event data.

12. The data processing method according to claim 11, wherein inserting supplementary event data at predetermined time steps within a time interval centered on the time component of the original event data comprises: Detect whether there is first and second original event data; The first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment; When the first original event data and the second original event data exist, at least one supplementary event data is inserted between the first original event data and the second original event data at the predetermined time step, wherein the coordinate components and polarity components of the supplementary event data and the original event data are the same.

13. The data processing method according to any one of claims 1 to 12, wherein, The steps for obtaining at least one initial prediction result corresponding one-to-one with at least one data to be predicted include: Generate the prediction frame data corresponding to each of the prediction data streams based on each prediction data stream. The initial prediction results for each channel are obtained based on the frame data to be predicted for each channel.

14. The data processing method according to claim 13, wherein, The steps of generating the predicted frame data corresponding to each of the predicted data streams include: The event stream composed of each path of data to be predicted is aggregated to generate event frame data; The frame data to be predicted is generated by superimposing multiple sequentially adjacent event frame data.

15. The data processing method according to any one of claims 1 to 12, wherein, The steps for obtaining the target prediction result based on the at least one initial prediction result include: The initial prediction results from each path are weighted and summed to obtain the target prediction result.

16. The data processing method according to any one of claims 1 to 12, wherein, Before the step of obtaining the target prediction result based on the at least one initial prediction result, the data processing method further includes: For at least one of the initial prediction results, perform the inverse transformation of the corresponding data augmentation transformation.

17. A data processing apparatus, comprising: A data augmentation module is used to augment the raw data collected by the event information acquisition device according to at least one data augmentation transformation to obtain at least one channel of augmented data, wherein each channel of augmented data corresponds to at least one of the data augmentation transformations. The prediction module is used to obtain at least one initial prediction result corresponding to at least one prediction data according to at least one prediction data, wherein the at least one prediction data includes at least one augmented data, or one original data and at least one augmented data; The fusion module is used to obtain the target prediction result based on the at least one initial prediction result; The event information acquisition device is an event-driven sensor; the raw data includes multiple raw event data, each of which corresponds to an event; the raw event data includes coordinate components, time components, and polarity components, where the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event.

18. An electronic device comprising: One or more processors; A memory having stored one or more programs thereon, which, when executed by the one or more processors, cause the one or more processors to implement the data processing method according to any one of claims 1 to 16; One or more I / O interfaces are connected between the processor and the memory and configured to enable information interaction between the processor and the memory.

19. A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method according to any one of claims 1 to 16.

Images