Model training method and device, inference method and device, electronic equipment and medium

Abstract

This disclosure provides a model training method, comprising: augmenting raw data collected by an event information acquisition device according to at least one data augmentation transformation to obtain at least one augmented data stream, wherein the at least one augmented data stream and the raw data constitute a training sample set, wherein each augmented data stream corresponds to at least one data augmentation transformation; and training a model to be trained using the training sample set. This disclosure also provides a model training device, an inference method, an inference device, an electronic device, and a computer-readable medium.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a model training method, a model training device, an inference method, an inference device, an electronic device, and a computer-readable medium. Background Technology

[0002] An event information acquisition device is an event-driven sensor that can output a data stream reflecting changes in the scene when an event occurs.

[0003] In some related technologies, the inference accuracy based on data collected by event information acquisition devices is not yet ideal. Summary of the Invention

[0004] This disclosure provides a model training method, a model training device, an inference method, an inference device, an electronic device, and a computer-readable medium.

[0005] In a first aspect, embodiments of this disclosure provide a model training method, including:

[0006] The original data collected by the event information acquisition device is enhanced according to at least one data augmentation transformation to obtain at least one enhanced data stream. The at least one enhanced data stream and the original data form a training sample set, wherein each enhanced data stream corresponds to at least one data augmentation transformation.

[0007] The training sample set is used to train the model to be trained.

[0008] In some embodiments, the raw data includes multiple raw event data, each raw event data corresponding to an event; the raw event data includes coordinate components, time components, and polarity components, wherein the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event; the step of enhancing the raw data collected by the event information acquisition device according to at least one data enhancement transformation to obtain at least one channel of enhanced data includes:

[0009] Based on at least one combination of various data augmentation transformations, including at least one data augmentation transformation that transforms the coordinate components of the event data, at least one data augmentation transformation that filters the event data, and at least one data augmentation transformation that transforms the polarity components of the event data, each of the original event data in the original data is processed to obtain the at least one channel of augmented data, wherein each combination of augmentation transformations includes at least one data augmentation transformation.

[0010] In some embodiments, processing each of the original event data in the original data according to a data augmentation transformation that transforms the polarity components of the event data includes:

[0011] The polarity components of each of the original event data are transformed to obtain each of the enhanced event data constituting the enhanced data, wherein the polarity of the events corresponding to the enhanced event data is opposite to the polarity of the events corresponding to the original event data.

[0012] In some embodiments, processing each of the original event data in the original data according to a data augmentation transformation that transforms the coordinate components of the event data includes:

[0013] The coordinate components of each of the original event data in the original data are scaled and transformed; or

[0014] Perform a shift transformation on the coordinate components of each of the original event data in the original data; or

[0015] The coordinate components of each of the original event data in the original data are rotated and transformed.

[0016] In some embodiments, for any one of the original event data in the original data, scaling and transforming the coordinate components of each of the original event data in the original data includes:

[0017] The coordinate components of the original event data are scaled and transformed according to a preset ratio to obtain an enhanced event data that constitutes the enhanced data.

[0018] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0019] In some embodiments, for any one of the original event data in the original data, performing a shift transformation on the coordinate components of each of the original event data in the original data includes:

[0020] The coordinate components of the original event data are shifted and transformed according to the shift vector to obtain an enhanced event data that constitutes the enhanced data.

[0021] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0022] In some embodiments, for any one of the original event data in the original data, performing a rotation transformation on the coordinate components of each of the original event data in the original data includes:

[0023] The coordinate components of the original event data are rotated and transformed relative to the center coordinates according to a preset rotation angle to obtain an enhanced event data that constitutes the enhanced data.

[0024] If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

[0025] In some embodiments, processing each of the original event data in the original data according to a data augmentation transformation that filters the event data includes:

[0026] Denoising processing is performed based on the time and coordinate components of each original event data in the original data; or

[0027] Redundancy removal is performed based on the time and coordinate components of each original event data in the original data; or

[0028] Event supplementation processing is performed based on the time and coordinate components of each original event data in the original data.

[0029] In some embodiments, for any one of the original event data in the original data, denoising processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0030] Detect whether there is event data falling within a preset spatiotemporal range centered on the original event data;

[0031] If no event data falls within a preset spatiotemporal range centered on the original event data, the original event data is deleted.

[0032] In some embodiments, for any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes:

[0033] Detect whether there are duplicate event data within a preset time range whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data;

[0034] If duplicate event data exists within the preset time range, the duplicate event data is deleted.

[0035] In some embodiments, for any one of the original event data in the original data, event supplementation processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0036] Supplementary event data is inserted at predetermined time steps within a time interval centered on the time component of the original event data.

[0037] In some embodiments, inserting supplementary event data at predetermined time steps within a time interval centered on the time component of the original event data includes:

[0038] Detect the existence of first and second original event data; the first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment;

[0039] When the first original event data and the second original event data exist, at least one supplementary event data is inserted between the first original event data and the second original event data at the predetermined time step, wherein the coordinate components and polarity components of the supplementary event data and the original event data are the same.

[0040] In some embodiments, the step of training the model to be trained using the training sample set includes:

[0041] Batch data is generated based on the at least one enhanced data stream and the original data;

[0042] The model to be trained is trained based on the batch data.

[0043] In some embodiments, the step of training the model to be trained using the training sample set includes:

[0044] The training samples from the training sample set are input into the model to be trained to obtain multiple initial prediction results that correspond one-to-one with the training samples. The training samples include one source of the original data and at least one source of augmented data.

[0045] The target prediction result is obtained based on the multi-path initial prediction results;

[0046] The model parameters of the model to be trained are corrected by backpropagation based on the target prediction results.

[0047] In some embodiments, the step of inputting multiple training samples from the training sample set into the model to be trained to obtain multiple initial prediction results corresponding one-to-one with the multiple training samples includes:

[0048] Generate training sample frame data corresponding to each training sample based on each training sample.

[0049] The initial prediction results for each path are obtained based on the training sample frame data of each path.

[0050] In some embodiments, the step of generating training sample frame data corresponding to each training sample based on each training sample includes:

[0051] The event stream composed of each training sample is aggregated to generate event frame data.

[0052] The training sample frame data is generated by superimposing multiple sequentially adjacent event frame data.

[0053] In some embodiments, the step of obtaining the target prediction result based on the multiple initial prediction results includes:

[0054] The initial prediction results from each path are weighted and summed to obtain the target prediction result.

[0055] In some embodiments, prior to the step of training the model to be trained using the training sample set, the model training method further includes:

[0056] Based on the original labels already labeled in the original data and the data augmentation transformations corresponding to each path of augmented data, augmentation labels are labeled for each path of augmented data.

[0057] In some embodiments, the model to be trained is used to identify targets, and / or detect targets, and / or count the number of targets.

[0058] Secondly, embodiments of this disclosure provide a reasoning method, including:

[0059] The prediction model is used to infer the prediction result based on the data to be predicted collected by the event information collection device; wherein the prediction model is trained according to the model training method described in any one of the first aspects of the present disclosure.

[0060] In some embodiments, the step of using a prediction model to infer the prediction result based on the data to be predicted collected by the event information collection device includes:

[0061] The data to be predicted is augmented according to at least one data augmentation transformation to obtain at least one path of augmented data to be predicted, wherein each path of augmented data to be predicted corresponds to at least one of the data augmentation transformations.

[0062] Generate at least one predictable frame data corresponding one-to-one with the at least one predictable augmentation data based on the at least one predictable augmentation data;

[0063] The at least one frame data to be predicted is input into the prediction model for inference to obtain the prediction result.

[0064] Thirdly, embodiments of this disclosure provide a model training apparatus, comprising:

[0065] A data augmentation module is used to augment the raw data collected by the event information acquisition device according to at least one data augmentation transformation to obtain at least one augmented data stream. The at least one augmented data stream and the raw data stream together form a training sample set, wherein each augmented data stream corresponds to at least one data augmentation transformation.

[0066] The training module is used to train the model to be trained using the training sample set.

[0067] Fourthly, embodiments of this disclosure provide a reasoning device, including:

[0068] The inference module is used to perform inference based on the data to be predicted collected by the event information collection device using a prediction model to obtain a prediction result; wherein the prediction model is trained according to the model training method described in any one of the first aspects of the present disclosure.

[0069] Fifthly, embodiments of this disclosure provide an electronic device, including:

[0070] One or more processors;

[0071] A memory having stored one or more programs that, when executed by one or more processors, cause the one or more processors to implement the model training method according to any one of the first aspects of the present disclosure and / or the inference method according to any one of the second aspects of the present disclosure;

[0072] One or more I / O interfaces are connected between the processor and the memory and configured to enable information interaction between the processor and the memory.

[0073] In a sixth aspect, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the model training method according to any one of the first aspects of this disclosure and / or the inference method according to any one of the second aspects of this disclosure.

[0074] In the model training method provided in this embodiment, the original data is augmented to obtain augmented data with high complexity. Training the neural network with these highly complex training samples can solve the problem of overfitting during training. Furthermore, the trained neural network can handle more possible scenarios during inference, thereby achieving better processing accuracy.

[0075] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0076] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0077] Figure 1 This is a flowchart of a model training method according to an embodiment of this disclosure;

[0078] Figure 2 This is a schematic diagram of a model training architecture according to an embodiment of this disclosure;

[0079] Figure 3 This is a flowchart of some steps in another model training method in this disclosure embodiment;

[0080] Figure 4 This is a flowchart of some steps in another model training method according to an embodiment of this disclosure;

[0081] Figure 5 This is a flowchart of some steps in another model training method according to an embodiment of this disclosure;

[0082] Figure 6 This is a flowchart of some steps in another model training method according to an embodiment of this disclosure;

[0083] Figure 7 This is a flowchart of some steps in another model training method according to an embodiment of this disclosure;

[0084] Figure 8 This is a flowchart of some steps in another model training method according to an embodiment of this disclosure;

[0085] Figure 9 This is a flowchart of some steps in another model training method according to an embodiment of this disclosure;

[0086] Figure 10 This is a flowchart of some steps in another model training method according to an embodiment of this disclosure;

[0087] Figure 11 This is a flowchart of some steps in another model training method according to an embodiment of this disclosure;

[0088] Figure 12 This is a flowchart of a reasoning method according to an embodiment of this disclosure;

[0089] Figure 13 This is a flowchart of some steps in another reasoning method in this disclosure embodiment;

[0090] Figure 14 This is a block diagram of a model training device according to an embodiment of the present disclosure;

[0091] Figure 15 This is a block diagram of a reasoning device according to an embodiment of the present disclosure;

[0092] Figure 16 This is a block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0093] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0094] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0095] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0096] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0097] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0098] Firstly, referring to Figure 1 This disclosure provides a model training method, including:

[0099] In step S100, the original data collected by the event information acquisition device is enhanced according to at least one data augmentation transformation to obtain at least one enhanced data. The at least one enhanced data and the original data form a training sample set, wherein each enhanced data corresponds to at least one data augmentation transformation.

[0100] In step S200, the prediction model is trained using the training sample set.

[0101] In some embodiments, the event information acquisition device is a dynamic vision sensor (DVS). A DVS is an event-driven sensor that outputs a data stream reflecting changes in pixel brightness at a specific location within a scene when an event occurs, and has become an important component of computer vision. In some embodiments, the dynamic vision sensor is an event-driven camera.

[0102] Figure 2 This is a schematic diagram of a model training architecture according to an embodiment of this disclosure. Figure 2 The model training architecture includes multiple branches, each corresponding to at least one data augmentation transformation. For example, ... Figure 2 As shown, the model training architecture includes branches corresponding to data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n, and branches that concatenate at least two of data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n. Concatenating at least two of data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n means that at least two of data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n are sequentially performed on the original data in that branch to obtain one path of augmented data. When the original data collected by the event information acquisition device passes through each branch, the corresponding data augmentation transformation of that branch is performed to obtain each path of augmented data.

[0103] In this embodiment, data augmentation transformation enhances the original data, enabling the generation of augmented data under different conditions. The augmented data, compared to the original data, more comprehensively reflects various aspects of the events acquired by the event information acquisition device. For example, through data augmentation transformation, at least one of the following can be obtained: augmented data with different event polarities, augmented data with different event coverage sizes, augmented data with different event locations, and augmented data after event coverage rotation and / or mirror flipping. Furthermore, the obtained augmented data can eliminate the influence of redundant data in the original data, and / or eliminate the influence of noise in the original data, and / or compensate for data gaps in the original data, thereby improving the continuity in the time dimension. This embodiment does not impose any specific limitations on these aspects.

[0104] In this public implementation, Figure 2 The network layer in the training network can be any network to be trained. The network to be trained can be a target recognition network for identifying targets, a target detection network for detecting targets, or a target counting network for counting the number of targets. This disclosure does not impose any special limitations on this. The network to be trained can be a convolutional neural network or a spiking neural network. This disclosure also does not impose any special limitations on this.

[0105] In the model training method provided in this embodiment, the original data is augmented to obtain augmented data with high complexity. Training the neural network with these highly complex training samples can solve the problem of overfitting during training. Furthermore, the trained neural network can handle more possible scenarios during inference, thereby achieving better processing accuracy.

[0106] The following description uses an event information acquisition device as an example of a dynamic visual sensor to illustrate the embodiments of this disclosure.

[0107] In some embodiments, the raw data acquired by the dynamic vision sensor consists of multiple event data in the form of (x, y, t, p) quadruples, with each event data corresponding to one event. Here, the coordinate component (x, y) represents the coordinates of the event; the time component t represents the time of the event's occurrence; and the polarity component p represents the polarity of the event, i.e., whether the illumination is enhanced or weakened; for example, 0 indicates weakened illumination, and 1 indicates enhanced illumination.

[0108] Accordingly, in some embodiments, the raw data includes multiple raw event data, each raw event data corresponding to one event; the raw event data includes coordinate components, time components, and polarity components, wherein the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event; refer to Figure 3 Step S100 includes:

[0109] In step S110, each of the original event data in the original data is processed according to at least one combination of various data augmentation transformations, including at least one data augmentation transformation that transforms the coordinate components of the event data, at least one data augmentation transformation that filters the event data, and at least one data augmentation transformation that transforms the polarity components of the event data, to obtain the at least one channel of augmented data. Each combination of augmentation transformations includes at least one data augmentation transformation.

[0110] It should be noted that the data augmentation transformations that make up different combinations of augmentation transformations are not all the same.

[0111] In this embodiment of the disclosure, processing each of the original event data in the original data according to a data augmentation transformation that transforms the polarity component of the event data can obtain augmented data with different changes in illumination intensity corresponding to the target; processing each of the original event data in the original data according to at least one data augmentation transformation that transforms the coordinate component of the event data can obtain at least one of multiple augmented data such as augmented data with different sizes of the target within the visual range, augmented data with different positions of the target within the visual range, and augmented data after the target is rotated and / or mirrored within the visual range; processing each of the original event data in the original data according to at least one data augmentation transformation that filters the event data can eliminate the influence of redundant data in the original data, and / or eliminate the influence of noise in the original data, and / or compensate for data missing data in the original data, thereby improving the continuity in the time dimension.

[0112] In some embodiments, refer to Figure 4 In step S110, the data augmentation transformation that transforms the polarity components of the event data to process each of the original event data in the original data includes:

[0113] In step S111, the polarity components of each of the original event data are transformed to obtain each of the enhanced event data constituting the enhanced data, wherein the polarity of the events corresponding to the enhanced event data is opposite to the polarity of the events corresponding to the original event data.

[0114] It should be noted that, in this embodiment of the disclosure, step S111 is performed on each original event data in the original data to obtain multiple enhanced event data.

[0115] In some embodiments, step S111 can be performed as follows:

[0116] i. Read raw event data (x, y, t, p) from the raw data;

[0117] ii. If the polar component p of the original event data (x, y, t, p) is 0, it is changed to 1 to obtain the enhanced event data corresponding to the original event data; if the polar component p of the original event data (x, y, t, p) is 1, it is changed to 0 to obtain the enhanced event data corresponding to the original event data. The coordinate components of the enhanced event data are the same as those of the original event data, and the time components of the enhanced event data are the same as those of the original event data.

[0118] The above polarity component transformation is performed sequentially on all the original event data of the original data.

[0119] This disclosure does not impose any special limitations on how to process each of the original event data in the original data according to at least one data augmentation transformation that transforms the coordinate components of the event data.

[0120] In some embodiments, refer to Figure 5 In step S110, the data augmentation transformation that transforms the coordinate components of the event data to process each of the original event data in the original data includes:

[0121] In step S112, the coordinate components of each of the original event data in the original data are scaled and transformed; or

[0122] In step S113, the coordinate components of each of the original event data in the original data are shifted and transformed; or

[0123] In step S114, the coordinate components of each of the original event data in the original data are rotated and transformed.

[0124] In some embodiments, the range of amplitude of the enhancement processing of the original data through at least one data augmentation change in steps S112, S113, and S114 can be controlled. For example, when adjusting the size of the target within the visual range through data augmentation transformation, the range of target size change can be controlled; when changing the position of the target within the visual range through data augmentation transformation, the range of target displacement can be controlled; when rotating the target within the visual range through data augmentation transformation, the rotation angle of the target can be controlled. In embodiments of this disclosure, the range of amplitude of the enhancement processing is controlled using an enhancement amplitude vector.

[0125] This disclosure does not impose any special limitations on how to scale and transform the coordinate components of each of the original event data in the original data.

[0126] In some embodiments, for any one of the original event data in the original data, scaling the coordinate components of each of the original event data in the original data includes: scaling the coordinate components of the original event data according to a preset ratio to obtain an enhanced event data constituting the enhanced data; and deleting the enhanced event data if the coordinate components of the enhanced event data exceed a preset data boundary.

[0127] In some embodiments, scaling transformation of the coordinate components of any one of the original event data in the original data can be performed as follows:

[0128] i. Determine the preset scale α and the preset data boundary (X1, Y1). The preset data boundary represents the size of the raw data acquired by the dynamic vision sensor.

[0129] ii. Read raw event data (x, y, t, p) from the raw data;

[0130] iii. Multiply the coordinate components (x, y) sequentially by a preset ratio α, and round up to obtain new coordinate components ([α·x], [α·y]), thus obtaining an enhanced event data. The time component of the enhanced event data is the same as the time component of the original event data, and the polarity component of the enhanced event data is the same as the polarity component of the original event data.

[0131] iv. If [α·x] > X1 and / or [α·y] > Y1, delete the enhanced event data.

[0132] The above coordinate scaling transformation is performed sequentially on all the original event data of the original data to obtain enhanced event data with multiple coordinate components not exceeding the preset data boundary.

[0133] This disclosure does not impose any special limitations on how to perform shift transformations on the coordinate components of each of the original event data in the original data.

[0134] In some embodiments, for any one of the original event data in the original data, shifting and transforming the coordinate components of each of the original event data in the original data includes: shifting and transforming the coordinate components of the original event data according to a shift vector to obtain an enhanced event data constituting the enhanced data; and deleting the enhanced event data if the coordinate components of the enhanced event data exceed a preset data boundary.

[0135] In some embodiments, shifting and transforming the coordinate components of any one of the original event data in the original data can be performed as follows:

[0136] i. Determine the shift vector (Δx2, Δy2) and the preset data boundary (X2, Y2). The preset data boundary represents the size of the raw data acquired by the dynamic vision sensor.

[0137] ii. Read raw event data (x, y, t, p) from the raw data;

[0138] iii. Add the coordinate components (x, y) of the original event data and the shift vector (Δx2, Δy2) to obtain the coordinate components (x+Δx2, y+Δy2) of the enhanced event data;

[0139] iv. If x + Δx² > X² and / or y + Δy² > Y², delete this enhanced event data.

[0140] The above coordinate shift transformation is performed sequentially on all the original event data of the original data to obtain multiple enhanced event data whose coordinate components do not exceed the preset data boundary.

[0141] This disclosure does not impose any special limitations on how to perform rotation transformation on the coordinate components of each of the original event data in the original data.

[0142] In some embodiments, for any one of the original event data in the original data, rotating and transforming the coordinate components of each of the original event data in the original data includes: rotating and transforming the coordinate components of the original event data relative to the center coordinate by a preset rotation angle to obtain an enhanced event data constituting the enhanced data; and deleting the enhanced event data if the coordinate components of the enhanced event data exceed the preset data boundary.

[0143] In some embodiments, performing a rotation transformation on the coordinate components of any one of the original event data in the original data can be performed as follows:

[0144] i. Determine the preset rotation angle Determine the shift vector (Δx3, Δy3) to determine the preset boundary (X3, Y3) and center coordinates.

[0145] ii. Read raw event data (x, y, t, p) from the raw data;

[0146] iii. Calculate the coordinate components (x, y) of the original event data about the center coordinates using angle calculations. The angle θ and the distance R will As the rotation angle, the coordinate components (x1, y1) of the enhanced event data are determined using the rotation angle and the distance R;

[0147] iv. If x1 > X3 and / or y1 > Y3, delete this enhanced event data.

[0148] The above rotation transformation is performed sequentially on all the original event data of the original data to obtain enhanced event data with multiple coordinate components that do not exceed the preset data boundary.

[0149] This disclosure does not specifically limit how each of the original event data in the original data is processed according to at least one data augmentation transformation that filters the event data.

[0150] In some embodiments, refer to Figure 6 In step S110, processing each of the original event data in the original data according to the data augmentation transformation that filters the event data includes:

[0151] In step S115, denoising processing is performed based on the time and coordinate components of each of the original event data in the original data; or

[0152] In step S116, redundancy removal is performed based on the time and coordinate components of each of the original event data in the original data; or

[0153] In step S117, event supplementation processing is performed based on the time and coordinate components of each of the original event data in the original data.

[0154] In this embodiment of the disclosure, denoising processing enables the augmented data to eliminate the influence of noise compared to the original data, thereby improving inference accuracy. This embodiment of the disclosure does not specifically limit how denoising processing is performed based on the time and coordinate components of each of the original event data in the original data.

[0155] In some embodiments, for any one of the original event data in the original data, denoising processing based on the time component and coordinate component of each of the original event data in the original data includes: detecting whether there is event data falling within a preset spatiotemporal range centered on the original event data; and deleting the original event data if there is no event data falling within the preset spatiotemporal range centered on the original event data.

[0156] It should be noted that, in this embodiment of the disclosure, event data falling within a preset time and space range means that the time component of the event data satisfies a preset time interval and the coordinate component satisfies a preset coordinate interval.

[0157] In some embodiments, denoising based on the time and coordinate components of any one of the original event data in the original data can be performed as follows:

[0158] i. Determine (Δx4, Δy4) and ΔT4;

[0159] ii. Read a raw event data (x, y, t, p) from the raw data, and use (x-Δx4, y-Δy4, t-ΔT4) to (x+Δx4, y+Δy4, t+ΔT4) to represent the preset spatiotemporal range centered on the raw event data;

[0160] iii. Detect whether there is event data falling within the spatiotemporal range of (x-Δx4, y-Δy4, t-ΔT4) to (x+Δx4, y+Δy4, t+ΔT4);

[0161] iv. If there is no event data falling within the spatiotemporal range of (x-Δx4, y-Δy4, t-ΔT4) to (x+Δx4, y+Δy4, t+ΔT4), delete the original event data.

[0162] The above denoising process is performed sequentially on all raw event data in the original data to remove the raw event data that serves as noise.

[0163] In this embodiment of the disclosure, the redundancy removal process enables the augmented data to eliminate the influence of redundant data compared to the original data, thereby improving inference accuracy. This embodiment of the disclosure does not specifically limit how the redundancy removal process is performed based on the time and coordinate components of each of the original event data in the original data.

[0164] In some embodiments, for any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes: detecting whether there is duplicate event data within a preset time range whose coordinate component is the same as the coordinate component of the original event data and whose polarity component is the same as the polarity component of the original event; and deleting the duplicate event data if the duplicate event data exists within the preset time range.

[0165] In some embodiments, redundancy removal processing based on the time component and coordinate component of any one of the original event data in the original data can be performed as follows:

[0166] i. Determine ΔT5;

[0167] ii. Read raw event data (x, y, t, p) from the raw data;

[0168] iii. Detect whether there are duplicate event data within a preset time range of (t-ΔT5, t+ΔT5) whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data;

[0169] iv. If duplicate event data exists within the preset time range, delete the duplicate event data.

[0170] Perform the above deduplication process on all raw event data in the original data in sequence to remove duplicate event data from the original data.

[0171] It should be noted that the above redundancy removal process can remove redundant data generated by DVS dense acquisition, thereby improving inference accuracy.

[0172] This disclosure does not impose any special limitations on how to perform event supplementation processing based on the time and coordinate components of each of the original event data in the original data.

[0173] In some embodiments, for any one of the original event data in the original data, event supplementation processing based on the time component and coordinate component of each of the original event data in the original data includes: inserting supplementary event data at a predetermined time step in a time interval centered on the time component of the original event data.

[0174] In some embodiments, inserting supplementary event data at a predetermined time step within a time interval centered on the time component of the original event data includes: detecting whether there is first original event data and second original event data; the first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment; when the first original event data and the second original event data exist, inserting at least one supplementary event data between the first original event data and the second original event data at a predetermined time step, wherein the supplementary event data and the coordinate and polarity components of the original event data are all the same.

[0175] In some embodiments, event supplementation processing based on the time component and coordinate component of any one of the original event data in the original data can be performed as follows:

[0176] i. Determine the preset time increment ΔT6 and the predetermined time step dt, where dt < ΔT6;

[0177] ii. Read a raw event data (x, y, t, p) from the raw data. The time interval centered on the time component of the raw event data can be represented as (t-ΔT6, t+ΔT6).

[0178] iii. If the presence of first original event data (x, y, t-ΔT6, p) and second original event data (x, y, t+ΔT6, p) is detected in the original data, supplementary event data (x, y, t-ΔT6, t+ΔT6) are inserted in the time interval (t-ΔT6, t+ΔT6) centered on the original event data (x, y, t-ΔT6, p) between the first original event data (x, y, t-ΔT6, p) and the second original event data (x, y, t+ΔT6, p) (x, y, t-ΔT6+1×dt, p), (x, y, t-ΔT6+2×dt, p)...(x, y, t+ΔT6-1×dt, p) in the time interval (t-ΔT6, t+ΔT6, p) centered on the original event data (x, y, t-ΔT6, p) between the first original event data (x, y, t-ΔT6, p) and the second original event data (x, y, t+ΔT6-1×dt, p).

[0179] Perform the above event supplementation processing on all raw event data of the original data in sequence, and insert supplementary event data into the raw event data of the original data.

[0180] It should be noted that the above event supplementation processing can supplement missing event data in the case of sparse DVS acquisition, avoid frame breaks, and thus improve inference accuracy.

[0181] In some embodiments, refer to Figure 7 Step S200 includes:

[0182] In step S201, batch processing data is generated based on the at least one enhanced data stream and the original data.

[0183] In step S202, the model to be trained is trained based on the batch data.

[0184] It should be noted that when training the model to be trained in step S202 based on the batch data generated from at least one augmented data and the original data, the model to be trained is trained uniformly using at least one augmented data and the original data, and the parameters of the network to be trained are updated uniformly through backpropagation.

[0185] In some embodiments, refer to Figure 8 Step S200 includes:

[0186] In step S210, the multiple training samples in the training sample set are input into the model to be trained to obtain multiple initial prediction results corresponding one-to-one with the multiple training samples. The multiple training samples include one of the original data and at least one of the augmented data.

[0187] In step S220, the target prediction result is obtained based on the multi-path initial prediction result;

[0188] In step S230, the model parameters of the model to be trained are corrected by backpropagation based on the target prediction result.

[0189] This disclosure does not impose any special limitations on how to modify the model parameters of the model to be trained. For example, the weight parameters and layer structure parameters of the model to be trained can be updated through backpropagation.

[0190] like Figure 2 As shown, the multi-channel augmented data processed by data augmentation generates frame data through a framing layer, and then inputs it into the model to be trained for prediction processing.

[0191] Accordingly, in some embodiments, reference is made to Figure 9 Step S210 includes:

[0192] In step S211, training sample frame data corresponding to each training sample is generated based on each training sample.

[0193] In step S212, the initial prediction results for each path are obtained based on the training sample frame data for each path.

[0194] The training sample frame data may include at least one event frame.

[0195] In some embodiments, refer to Figure 10Step S211 includes:

[0196] In step S2111, the event stream composed of each training sample is aggregated to generate event frame data;

[0197] In step S2112, multiple sequentially adjacent event frame data are superimposed to generate the training sample frame data.

[0198] In some embodiments, step S211 may be performed as follows:

[0199] The input augmentation data for the framing layer consists of multiple augmentation event data in the form of (x, y, t, p) quadruples. The augmentation event data within the time window Δt forms the event stream E. t′ ={e i |e i =[x i y i ,t′,p i The framing layer aggregates the event stream and outputs event frame data X. t =q(E t′ The aggregation function q(·) can be any of the following: nonpolar function aggregation, cumulative aggregation, and logical operation aggregation; this embodiment does not impose any special limitation on this. The event frame data is in the form (c, h, w), where c represents the channel, h represents the height of the frame data, and w represents the width of the frame data. Finally, T adjacent event frame data are superimposed in chronological order to form training sample frame data in the form (T, c, h, w).

[0200] In some embodiments, when the training sample frame data includes multiple event frame data, the model to be trained can be constructed based on a spiking neural network. Here, "constructed based on a spiking neural network" can be understood as the prediction model including a spiking neural network. The model to be trained can also include non-spiking neural networks, such as artificial neural networks; this disclosure does not impose any limitations in this regard. The model to be trained based on a spiking neural network can better learn the temporal information between multiple event frame data in the training sample frame data formed by superimposing them in time sequence, thereby enabling the training of a prediction model with higher prediction accuracy.

[0201] like Figure 2 As shown, the fusion layer obtains the target prediction result based on the multiple initial prediction results. This embodiment of the disclosure does not impose any special limitations on how the target prediction result is obtained from the multiple initial prediction results.

[0202] In some embodiments, the step of obtaining the target prediction result based on the multiple initial prediction results includes: weighting and summing the initial prediction results from each path to obtain the target prediction result.

[0203] In some embodiments, the step of obtaining a target prediction result based on the multiple initial prediction results includes: calculating the average value of each of the initial prediction results as the target prediction result.

[0204] In some embodiments, refer to Figure 11 Before the step of obtaining the target prediction result based on the multi-path initial prediction results, the model training method further includes:

[0205] In step S300, enhancement labels are assigned to each channel of enhanced data based on the original labels already labeled in the original data and the data enhancement transformations corresponding to each channel of enhanced data.

[0206] This disclosure does not specifically limit how enhancement labels are affixed to each stream of enhanced data in step S300. For example, after performing a shift transformation on the coordinate components of each original event data in the original data, enhancement labels are added to the obtained enhanced data, and the enhancement labels represent the position information of the target object after the shift transformation.

[0207] The embodiments disclosed herein do not impose any special limitations on the application scenarios of the model to be trained.

[0208] In some embodiments, the model to be trained is used to identify targets, and / or detect targets, and / or count the number of targets.

[0209] Secondly, referring to Figure 12 This disclosure provides a reasoning method, including:

[0210] In step S400, the prediction model is used to infer the prediction result based on the data to be predicted collected by the event information collection device; wherein the prediction model is trained according to the model training method described in any one of the first aspects of the present disclosure.

[0211] In this embodiment of the present disclosure, a prediction model is obtained by training according to the model training method described in any one of the first aspects of the present disclosure. In the training process of the prediction model, the problem of overfitting during training is solved. The prediction obtained by training can correspond to more possible scenarios in the inference process, so that the prediction model can have better inference accuracy and thus obtain more accurate prediction results.

[0212] The data to be predicted can be collected in real time by the event information acquisition device. The data to be predicted can be acquired in real time or at a preset period and input into the prediction model. The data to be predicted can also be pre-stored data collected by the event information acquisition device.

[0213] In some embodiments, refer to Figure 13 Step S400 includes:

[0214] In step S410, the data to be predicted is enhanced according to at least one data augmentation transformation to obtain at least one path of enhanced data to be predicted, wherein each path of enhanced data to be predicted corresponds to at least one of the data augmentation transformations.

[0215] In step S420, at least one predictable frame data corresponding one-to-one with the at least one predictable augmentation data is generated based on the at least one predictable augmentation data.

[0216] In step S430, the at least one frame data to be predicted is input into the prediction model for inference to obtain the prediction result.

[0217] In this embodiment of the present disclosure, before the prediction model is trained using the model training method according to any one of the first aspects of the present disclosure, the data to be predicted collected by the event information collection device is first enhanced and transformed. This can obtain enhanced data to be predicted under different conditions from the data to be predicted, so that the inference has better generalization performance, and / or can eliminate the influence of redundant data in the data to be predicted, can eliminate the influence of noise in the data to be predicted, and can make up for the data missing in the data to be predicted, thereby improving the continuity in the time dimension, thereby further improving the inference accuracy of the prediction model.

[0218] Thirdly, referring to Figure 14 This disclosure provides a model training apparatus, including:

[0219] Data augmentation module 110 is used to augment the original data collected by the event information acquisition device according to at least one data augmentation transformation to obtain at least one augmented data, wherein the at least one augmented data and the original data form a training sample set, wherein each augmented data corresponds to at least one data augmentation transformation;

[0220] The training module 120 is used to train the model to be trained using the training sample set.

[0221] The model training apparatus provided in the second aspect of this disclosure is used to implement any of the model training methods described in the first aspect of this disclosure. The model training methods have already been explained above and will not be repeated here.

[0222] Fourthly, refer to Figure 15 This disclosure provides an inference device, comprising:

[0223] The inference module 200 is used to perform inference based on the data to be predicted collected by the event information collection device using a prediction model to obtain a prediction result; wherein the prediction model is trained according to the model training method described in any one of the first aspects of the present disclosure.

[0224] Fifthly, refer to Figure 16 This disclosure provides an electronic device, including:

[0225] One or more processors 301;

[0226] The memory 302 stores one or more programs, which, when executed by one or more processors, enable the one or more processors to implement any of the model training methods described in the first aspect of the present disclosure.

[0227] One or more I / O interfaces 303 are connected between the processor and the memory and configured to enable information exchange between the processor and the memory.

[0228] Among them, processor 301 is a device with data processing capabilities, including but not limited to central processing unit (CPU); memory 302 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); I / O interface (read-write interface) 303 is connected between processor 301 and memory 302, and can realize information interaction between processor 301 and memory 302, including but not limited to data bus (Bus).

[0229] In some embodiments, the processor 301, memory 302, and I / O interface 303 are interconnected via bus 304, and thus connected to other components of the computing device.

[0230] Sixthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements any of the model training methods described in the first aspect of this disclosure.

[0231] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0232] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A model training method, comprising: The original data collected by the event information acquisition device is enhanced according to at least one data augmentation transformation to obtain at least one channel of enhanced data. The at least one channel of enhanced data and the original data form a training sample set. The event information acquisition device is an event-driven camera. Each channel of enhanced data corresponds to at least one of the data augmentation transformations. The original data includes multiple original event data, which include coordinate components, time components, and polarity components. The training sample set is used to train the model to be trained; The step of training the model to be trained using the training sample set includes: aggregating the event stream and outputting event frame data for training the model to be trained; wherein the aggregation function for aggregating the event stream is any one of nonpolar function aggregation, cumulative aggregation, and logical operation aggregation.

2. The model training method according to claim 1, wherein each piece of original event data corresponds to one event; the time component represents the occurrence time of the corresponding event, and the polarity component represents the polarity of the corresponding event; the step of enhancing the original data collected by the event information acquisition device according to at least one data augmentation transformation to obtain at least one channel of enhanced data includes: Based on at least one combination of various data augmentation transformations, including at least one data augmentation transformation that transforms the coordinate components of the event data, at least one data augmentation transformation that filters the event data, and at least one data augmentation transformation that transforms the polarity components of the event data, each of the original event data in the original data is processed to obtain the at least one channel of augmented data, wherein each combination of augmentation transformations includes at least one data augmentation transformation.

3. The model training method according to claim 2, wherein, The data augmentation transformation, which transforms the polarity components of the event data, processes each of the original event data in the original data, including: The polarity components of each of the original event data are transformed to obtain each of the enhanced event data constituting the enhanced data, wherein the polarity of the events corresponding to the enhanced event data is opposite to the polarity of the events corresponding to the original event data.

4. The model training method according to claim 2, wherein, The data augmentation transformation, which transforms the coordinate components of the event data, processes each of the original event data in the original data, including: The coordinate components of each of the original event data in the original data are scaled and transformed; or Perform a shift transformation on the coordinate components of each of the original event data in the original data; or The coordinate components of each of the original event data in the original data are rotated and transformed.

5. The model training method according to claim 4, wherein, For any one of the original event data in the original data, scaling transformation of the coordinate components of each of the original event data in the original data includes: The coordinate components of the original event data are scaled and transformed according to a preset ratio to obtain an enhanced event data that constitutes the enhanced data. If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

6. The model training method according to claim 4, wherein, For any one of the original event data in the original data, performing a shift transformation on the coordinate components of each of the original event data in the original data includes: The coordinate components of the original event data are shifted and transformed according to the shift vector to obtain an enhanced event data that constitutes the enhanced data. If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

7. The model training method according to claim 4, wherein, For any one of the original event data in the original data, the rotation transformation of the coordinate components of each of the original event data in the original data includes: The coordinate components of the original event data are rotated and transformed relative to the center coordinates according to a preset rotation angle to obtain an enhanced event data that constitutes the enhanced data. If the coordinate components of the enhanced event data exceed the preset data boundary, the enhanced event data is deleted.

8. The model training method according to claim 2, wherein, The data augmentation transformation based on the filtered event data is used to process each of the original event data in the original data, including: Denoising processing is performed based on the time and coordinate components of each original event data in the original data; or Redundancy removal is performed based on the time and coordinate components of each original event data in the original data; or Event supplementation processing is performed based on the time and coordinate components of each original event data in the original data.

9. The model training method according to claim 8, wherein, For any one of the original event data in the original data, the noise reduction process based on the time component and coordinate component of each original event data in the original data includes: Detect whether there is event data falling within a preset spatiotemporal range centered on the original event data; If no event data falls within a preset spatiotemporal range centered on the original event data, the original event data is deleted.

10. The model training method according to claim 8, wherein, For any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes: Detect whether there are duplicate event data within a preset time range whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data; If duplicate event data exists within the preset time range, the duplicate event data is deleted.

11. The model training method according to claim 8, wherein, For any one of the original event data in the original data, the event supplementation processing based on the time component and coordinate component of each original event data in the original data includes: Supplementary event data is inserted at predetermined time steps within a time interval centered on the time component of the original event data.

12. The model training method according to claim 11, wherein, Inserting supplementary event data at predetermined steps within a time interval centered on the time component of the original event data includes: Detect the existence of first and second original event data; the first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment; When the first original event data and the second original event data exist, at least one supplementary event data is inserted between the first original event data and the second original event data at the predetermined time step, wherein the coordinate components and polarity components of the supplementary event data and the original event data are the same.

13. The model training method according to any one of claims 1 to 12, wherein, The steps for training the model to be trained using the training sample set include: Batch data is generated based on the at least one enhanced data stream and the original data; The model to be trained is trained based on the batch data.

14. The model training method according to any one of claims 1 to 12, wherein, The steps for training the model to be trained using the training sample set include: The training samples from the training sample set are input into the model to be trained to obtain multiple initial prediction results that correspond one-to-one with the training samples. The training samples include one source of the original data and at least one source of augmented data. The target prediction result is obtained based on the multi-path initial prediction results; The model parameters of the model to be trained are corrected by backpropagation based on the target prediction results.

15. The model training method according to claim 14, wherein, The step of inputting multiple training samples from the training sample set into the model to be trained to obtain multiple initial prediction results corresponding one-to-one with the multiple training samples includes: Generate training sample frame data corresponding to each training sample based on each training sample. The initial prediction results for each path are obtained based on the training sample frame data of each path.

16. The model training method according to claim 15, wherein, The steps of generating training sample frame data corresponding to each training sample based on each training sample include: The event stream composed of each training sample is aggregated to generate event frame data. The training sample frame data is generated by superimposing multiple sequentially adjacent event frame data.

17. The model training method according to claim 14, wherein, The steps for obtaining the target prediction result based on the multi-path initial prediction results include: The initial prediction results from each path are weighted and summed to obtain the target prediction result.

18. The model training method according to any one of claims 1 to 12, wherein, Before the step of training the model to be trained using the training sample set, the model training method further includes: Based on the original labels already labeled in the original data and the data augmentation transformations corresponding to each path of augmented data, augmentation labels are labeled for each path of augmented data.

19. The model training method according to any one of claims 1 to 12, wherein, The model to be trained is used to identify targets, and / or detect targets, and / or count the number of targets.

20. A reasoning method, comprising: The prediction model is used to infer the prediction result based on the data to be predicted collected by the event information collection device; wherein the prediction model is trained by the model training method according to any one of claims 1 to 19.

21. The reasoning method according to claim 20, wherein, The steps for obtaining prediction results by using a prediction model to infer from the data to be predicted collected by the event information collection device include: The data to be predicted is augmented according to at least one data augmentation transformation to obtain at least one path of augmented data to be predicted, wherein each path of augmented data to be predicted corresponds to at least one of the data augmentation transformations. Generate at least one predictable frame data corresponding one-to-one with the at least one predictable augmentation data based on the at least one predictable augmentation data; The at least one frame data to be predicted is input into the prediction model for inference to obtain the prediction result.

22. A model training apparatus for performing the model training method according to any one of claims 1 to 19, wherein the model training apparatus comprises: A data augmentation module is used to augment the raw data collected by the event information acquisition device according to at least one data augmentation transformation to obtain at least one augmented data stream. The at least one augmented data stream and the raw data stream together form a training sample set, wherein each augmented data stream corresponds to at least one data augmentation transformation. The training module is used to train the model to be trained using the training sample set.

23. A reasoning device, comprising: The inference module is used to perform inference based on the data to be predicted collected by the event information collection device using a prediction model to obtain a prediction result; wherein the prediction model is trained by the model training method according to any one of claims 1 to 19.

24. An electronic device, comprising: One or more processors; A memory having stored one or more programs that, when executed by one or more processors, cause the one or more processors to implement the model training method according to any one of claims 1 to 19 and / or the inference method according to claim 20 or 21; One or more I / O interfaces are connected between the processor and the memory and configured to enable information interaction between the processor and the memory.

25. A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the model training method according to any one of claims 1 to 19 and / or the inference method according to claim 20 or 21.

Images