Target detection method and device, electronic device, computer readable medium

Abstract

The present disclosure provides a target detection method, comprising: performing enhancement processing on original data collected by an event information collection device according to at least one data enhancement transformation, to obtain at least one piece of enhanced data, wherein each piece of enhanced data corresponds to at least one data enhancement transformation; inputting the at least one piece of enhanced data into a target detection network to obtain at least one piece of initial detection result of detecting a target object in the original data, wherein the at least one piece of initial detection result corresponds to the at least one piece of enhanced data in a one-to-one manner; and obtaining a target detection result of detecting a target object in the original data according to the at least one piece of initial detection result. The present disclosure also provides a target detection device, an electronic device and a computer readable medium.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a target detection method, a target detection device, an electronic device, and a computer-readable medium. Background Technology

[0002] An event information acquisition device is an event-driven sensor that can output a data stream reflecting changes in the scene when an event occurs.

[0003] In some related technologies, the accuracy of target detection based on data collected by event information acquisition devices is not yet ideal. Summary of the Invention

[0004] This disclosure provides a target detection method, a target detection device, an electronic device, and a computer-readable medium.

[0005] In a first aspect, embodiments of this disclosure provide a target detection method, including:

[0006] The raw data collected by the event information acquisition device is enhanced according to at least one data enhancement transformation to obtain at least one enhanced data channel, wherein each enhanced data channel corresponds to at least one data enhancement transformation.

[0007] The at least one enhanced data stream is input into the target detection network to obtain at least one initial detection result for detecting the target object in the original data, and the at least one initial detection result corresponds one-to-one with the at least one enhanced data stream.

[0008] The target detection result is obtained by detecting the target object in the original data based on the at least one initial detection result.

[0009] In some embodiments, the raw data includes multiple raw event data, each raw event data corresponding to an event; the raw event data includes coordinate components, time components, and polarity components, wherein the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event; the step of enhancing the raw data collected by the event information acquisition device according to at least one data enhancement transformation to obtain at least one channel of enhanced data includes:

[0010] The original event data is processed using various data augmentation transformations that filter event data to obtain one stream of augmented data; or

[0011] An augmented data stream is obtained by using multiple data augmentation transformations to filter event data, as well as at least one data augmentation transformation to transform the coordinate components of the event data and / or the polarity components of the event data.

[0012] In some embodiments, processing each of the original event data in the original data according to a variety of data augmentation transformations that filter event data includes:

[0013] The original event data in the original data is processed by denoising, redundancy removal, and event supplementation based on the time and coordinate components of each event data.

[0014] In some embodiments, for any one of the original event data in the original data, denoising processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0015] Detect whether there is event data falling within a preset spatiotemporal range centered on the original event data;

[0016] If no event data falls within a preset spatiotemporal range centered on the original event data, the original event data is deleted.

[0017] In some embodiments, for any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes:

[0018] Detect whether there are duplicate event data within a preset time range whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data;

[0019] If duplicate event data exists within the preset time range, the duplicate event data is deleted.

[0020] In some embodiments, for any one of the original event data in the original data, event supplementation processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0021] Supplementary event data is inserted at predetermined steps within a time interval centered on the time component of the original event data.

[0022] In some embodiments, inserting supplementary event data with a predetermined step size within a time interval centered on the time component of the original event data includes: detecting whether there is first original event data and second original event data; the first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment;

[0023] When the first original event data and the second original event data exist, at least one supplementary event data is inserted between the first original event data and the second original event data at a predetermined time step. The coordinate components and polarity components of the supplementary event data and the original event data are the same.

[0024] In some embodiments, the step of inputting the at least one enhanced data stream into a target detection network to obtain at least one initial detection result for detecting the target object in the original data includes:

[0025] Generate the detection frame data corresponding to each of the enhanced data streams based on each stream of enhanced data;

[0026] The target detection network is input into each of the target detection frames to obtain the initial detection results for each of the paths.

[0027] In some embodiments, the step of generating detection frame data corresponding to each of the enhanced data streams includes:

[0028] The event stream composed of each enhanced data stream is aggregated to generate event frame data;

[0029] The detection frame data is generated by superimposing multiple sequentially adjacent event frame data.

[0030] In some embodiments, when multiple initial detection results are included, the step of obtaining a target detection result for detecting a target object in the original data based on the at least one initial detection result includes:

[0031] Multiple target detection boxes are obtained based on the multi-path initial detection results, and the target detection boxes represent the position information of the target object;

[0032] The multiple target detection boxes are deduplicated to obtain the target detection results.

[0033] In some embodiments, prior to the step of obtaining a target detection result for detecting a target object in the original data based on the at least one initial detection result, the target detection method further includes:

[0034] For at least one initial detection result in the multiple initial detection results that corresponds to at least one data augmentation transformation that transforms the coordinate components of the event data, perform an inverse transformation corresponding to the data augmentation transformation that transforms the coordinate components of the event data;

[0035] The steps for obtaining target detection results for detecting target objects in the original data based on at least one initial detection result include:

[0036] The target detection result is obtained based on the at least one initial detection result after inverse transformation.

[0037] Secondly, embodiments of this disclosure provide a target detection device, comprising:

[0038] A data augmentation module is used to augment the raw data collected by the event information acquisition device according to at least one data augmentation transformation to obtain at least one channel of augmented data, wherein each channel of augmented data corresponds to at least one of the data augmentation transformations.

[0039] The detection module is used to input the at least one enhanced data into the target detection network to obtain at least one initial detection result for detecting the target object in the original data, wherein the at least one initial detection result corresponds one-to-one with the at least one enhanced data;

[0040] The fusion module is used to obtain the target detection result for detecting the target object in the original data based on the at least one initial detection result.

[0041] In some embodiments, the raw data includes multiple raw event data, each of which corresponds to an event; the raw event data includes coordinate components, time components, and polarity components, wherein the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event.

[0042] The data augmentation module is used to process each of the original event data in the original data according to various data augmentation transformations that filter event data, to obtain one stream of augmented data; and / or

[0043] The data augmentation module is used to obtain one channel of augmented data based on various data augmentation transformations that filter event data, at least one data augmentation transformation that transforms the coordinate components of the event data, and / or a data augmentation transformation that transforms the polarity components of the event data.

[0044] In some embodiments, in the data augmentation module, processing each of the original event data in the original data according to various data augmentation transformations that filter event data includes:

[0045] The original event data in the original data is processed by denoising, redundancy removal, and event supplementation based on the time and coordinate components of each event data.

[0046] In some embodiments, in the data enhancement module, for any one of the original event data in the original data, denoising processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0047] Detect whether there is event data falling within a preset spatiotemporal range centered on the original event data;

[0048] If no event data falls within a preset spatiotemporal range centered on the original event data, the original event data is deleted.

[0049] In some embodiments, in the data enhancement module, for any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes:

[0050] Detect whether there are duplicate event data within a preset time range whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data;

[0051] If duplicate event data exists within the preset time range, the duplicate event data is deleted.

[0052] In some embodiments, in the data enhancement module, for any one of the original event data in the original data, event supplementation processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0053] Supplementary event data is inserted at predetermined steps within a time interval centered on the time component of the original event data.

[0054] In some embodiments, in the data augmentation module, inserting supplementary event data at predetermined steps within a time interval centered on the time component of the original event data includes:

[0055] Detect the existence of first and second original event data; the first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment;

[0056] When the first original event data and the second original event data exist, at least one supplementary event data is inserted between the first original event data and the second original event data at a predetermined time step. The coordinate components and polarity components of the supplementary event data and the original event data are the same.

[0057] In some embodiments, the detection module includes:

[0058] A framing unit is used to generate the frame data to be detected corresponding to each of the enhanced data streams based on each of the enhanced data streams.

[0059] The detection unit is used to input the data of each of the frames to be detected into the target detection network to obtain the initial detection results of each of the paths.

[0060] In some embodiments, the framing unit is used to aggregate the event stream composed of each of the enhanced data streams to generate event frame data; and to superimpose multiple sequentially adjacent event frame data to generate the frame data to be detected.

[0061] In some embodiments, the fusion module is configured to, when including multiple initial detection results, obtain multiple target detection boxes based on the multiple initial detection results, wherein the target detection boxes represent the position information of the target object; and deduplicate the multiple target detection boxes to obtain the target detection result.

[0062] In some embodiments, the target detection device further includes:

[0063] The inverse transformation module is used to perform an inverse transformation corresponding to the data augmentation transformation that transforms the coordinate components of the event data in at least one initial detection result that corresponds to at least one data augmentation transformation that transforms the coordinate components of the event data.

[0064] The fusion module is used to obtain the target detection result based on the at least one initial detection result after inverse transformation.

[0065] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0066] One or more processors;

[0067] A memory having stored one or more programs, which, when executed by one or more processors, enable the one or more processors to implement any of the target detection methods described in the first aspect of the present disclosure.

[0068] One or more I / O interfaces are connected between the processor and the memory and configured to enable information interaction between the processor and the memory.

[0069] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements any of the target detection methods described in the first aspect of this disclosure.

[0070] In the target detection method provided in this embodiment, the original data acquired by the event information acquisition device is subjected to data augmentation transformation to obtain at least one augmented data, and target detection is performed based on the at least one augmented data to obtain at least one initial detection result. Finally, the target detection result is obtained based on the at least one initial detection result. Through data augmentation transformation, the influence of redundant data in the original data can be eliminated, and / or the influence of noise in the original data can be eliminated, and / or the data missing in the original data can be compensated for, thereby improving the continuity of target detection in the time dimension. This makes the target detection result that integrates the initial detection results more accurate, that is, it improves the accuracy of target detection based on the data of the event information acquisition device.

[0071] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0072] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0073] Figure 1 This is a flowchart of a target detection method according to an embodiment of this disclosure;

[0074] Figure 2 This is a schematic diagram of a target detection architecture according to an embodiment of this disclosure;

[0075] Figure 3 This is a flowchart of some steps in another target detection method according to an embodiment of this disclosure;

[0076] Figure 4 This is a flowchart of some steps in another target detection method according to an embodiment of this disclosure;

[0077] Figure 5 This is a flowchart of some steps in another target detection method according to an embodiment of this disclosure;

[0078] Figure 6 This is a flowchart of some steps in another target detection method according to an embodiment of this disclosure;

[0079] Figure 7 This is a flowchart of some steps in another target detection method according to an embodiment of this disclosure;

[0080] Figure 8 This is a flowchart of some steps in another target detection method according to an embodiment of this disclosure;

[0081] Figure 9 This is a block diagram of a target detection device according to an embodiment of the present disclosure;

[0082] Figure 10 This is a block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0083] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0084] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0085] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0086] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0087] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0088] Firstly, referring to Figure 1 This disclosure provides a target detection method, including:

[0089] In step S100, the raw data collected by the event information acquisition device is enhanced according to at least one data enhancement transformation to obtain at least one enhanced data channel, wherein each enhanced data channel corresponds to at least one data enhancement transformation.

[0090] In step S200, the at least one enhanced data is input into the target detection network to obtain at least one initial detection result for detecting the target object in the original data, and the at least one initial detection result corresponds one-to-one with the at least one enhanced data.

[0091] In step S300, a target detection result is obtained based on the at least one initial detection result to detect the target object in the original data.

[0092] In some embodiments, the event information acquisition device includes a dynamic vision sensor (DVS). A DVS is an event-driven sensor that outputs a data stream reflecting changes in pixel brightness at a specific location within a scene when an event occurs, and has become an important component of computer vision. In some embodiments, the dynamic vision sensor is an event-driven camera.

[0093] Figure 2 This is a schematic diagram of a target detection architecture according to an embodiment of this disclosure. Figure 2The object detection architecture includes multiple branches, each corresponding to at least one data augmentation transformation. For example, such as... Figure 2 As shown, the target detection architecture includes branches corresponding to data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n, and also includes branches that concatenate at least two of data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n. Concatenating at least two of data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n means that at least two of data augmentation transformation 1, data augmentation transformation 2, ..., data augmentation transformation n are sequentially performed on the original data in that branch to obtain one path of augmented data. When the original data collected by the event information acquisition device passes through each branch, the corresponding data augmentation transformation of that branch is performed to obtain each path of augmented data. In this embodiment, the network layer is a target detection network. It should be noted that in this embodiment, the target detection architecture may also include only one branch, which corresponds to one type of data augmentation transformation or multiple concatenated data augmentation transformations.

[0094] It should be noted that when only one branch is included, the initial detection result of that branch can be determined as the target detection result. When multiple branches are included, multiple initial detection results can be obtained, such as multiple target detection boxes representing the position information of the target object. The multiple target detection boxes can be deduplicated to obtain the target detection result.

[0095] In this embodiment of the disclosure, the original data is enhanced according to the data augmentation transformation. The resulting enhanced data can at least eliminate the influence of redundant data in the original data, and / or at least eliminate the influence of noise in the original data, and / or at least compensate for data deficiencies in the original data, thereby improving the continuity of target detection in the time dimension. This embodiment of the disclosure does not impose any special limitations on this.

[0096] It should be noted that, in this embodiment of the disclosure, there is no particular limitation on the detection of target objects in the original data. In some embodiments, the detection of target objects in the original data includes identifying the target objects and marking the identified target objects with a target detection bounding box.

[0097] like Figure 2 As shown, in step S200, target detection is performed on each path of enhanced data through a target detection network at the network layer to obtain initial detection results for each path. This embodiment of the disclosure does not impose any special limitations on the target detection network. For example, the target detection network can be a convolutional neural network or a spiking neural network.

[0098] In this embodiment, the object detection network can be trained using any training method, and this embodiment does not impose any special limitations on it. For example, it can be trained using augmented data obtained by augmenting the original training data collected by the event information acquisition device. The augmentation process can include any augmentation transformation method in this embodiment, and this disclosure does not impose any limitations on it. Since the original training data has been augmented, the resulting training data has high complexity. Training the neural network with high-complexity training data can solve the problem of overfitting during training. Furthermore, the trained neural network can correspond to more possible scenarios during inference, thereby enabling the trained neural network to have better processing accuracy.

[0099] In the target detection method provided in this embodiment, the original data acquired by the event information acquisition device is subjected to data augmentation transformation to obtain at least one augmented data, and target detection is performed based on the at least one augmented data to obtain at least one initial detection result. Finally, the target detection result is obtained based on the at least one initial detection result. Through data augmentation transformation, the influence of redundant data in the original data can be eliminated, and / or the influence of noise in the original data can be eliminated, and / or the data missing in the original data can be compensated for, thereby improving the continuity of target detection in the time dimension. This makes the target detection result that integrates the initial detection results more accurate, that is, it improves the accuracy of target detection based on the data of the event information acquisition device.

[0100] The following description uses an event information acquisition device as an example of a dynamic visual sensor to illustrate the embodiments of this disclosure.

[0101] In some embodiments, the raw data acquired by the dynamic vision sensor consists of multiple event data in the form of (x, y, t, p) quadruples, with each event data corresponding to an event. Here, the coordinate component (x, y) represents the coordinates of the event; the time component t represents the time of the event's occurrence; and the polarity component p represents the polarity of the event, i.e., whether the light intensity increases or decreases; for example, 0 indicates decreased light intensity, and 1 indicates increased light intensity.

[0102] Accordingly, in some embodiments, the raw data includes multiple raw event data, each raw event data corresponding to one event; the raw event data includes coordinate components, time components, and polarity components, wherein the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event; refer to Figure 3 Step S100 includes:

[0103] In step S110, each of the original event data in the original data is processed according to various data augmentation transformations that filter the event data, to obtain at least one augmented data stream; and / or

[0104] In step S120, an enhanced data path is obtained by performing various data augmentation transformations that filter event data, at least one data augmentation transformation that transforms the coordinate components of the event data, and / or a data augmentation transformation that transforms the polarity components of the event data.

[0105] It should be noted that step S110 can eliminate redundant data and noise in the original data, and can make up for missing data in the original data, thereby improving the continuity of target detection in the time dimension.

[0106] This disclosure does not impose any special limitations on how to process each of the original event data in the original data according to various data augmentation transformations that filter event data.

[0107] In some embodiments, refer to Figure 4 In steps S110 and S120, processing each of the original event data in the original data according to various data augmentation transformations that filter event data includes:

[0108] In step S111, noise reduction, redundancy removal, and event supplementation are performed on the time and coordinate components of each original event data in the original data to obtain one channel of enhanced data.

[0109] This disclosure does not impose any special limitations on how to perform denoising processing based on the time and coordinate components of each of the original event data in the original data.

[0110] In some embodiments, for any one of the original event data in the original data, denoising processing based on the time component and coordinate component of each of the original event data in the original data includes: detecting whether there is event data falling within a preset spatiotemporal range centered on the original event data; and deleting the original event data if there is no event data falling within the preset spatiotemporal range centered on the original event data.

[0111] It should be noted that, in this embodiment of the disclosure, event data falling within a preset time and space range means that the time component of the event data satisfies a preset time interval and the coordinate component satisfies a preset coordinate interval.

[0112] In some embodiments, denoising based on the time and coordinate components of any one of the original event data in the original data can be performed as follows:

[0113] i. Determine (Δx1, Δy1) and ΔT1;

[0114] ii. Read a raw event data (x, y, t, p) from the raw data, and use (x-Δx1, y-Δy1, t-ΔT1) to (x+Δx1, y+Δy1, t+ΔT1) to represent the preset spatiotemporal range centered on the raw event data;

[0115] iii. Detect whether there is event data falling within the spatiotemporal range of (x-Δx1, y-Δy1, t-ΔT1) to (x+Δx1, y+Δy1, t+ΔT1);

[0116] iv. If there is no event data falling within the spatiotemporal range of (x-Δx1, y-Δy1, t-ΔT1) to (x+Δx1, y+Δy1, t+ΔT1), delete the original event data.

[0117] The above denoising process is performed sequentially on all raw event data in the original data to remove the raw event data that serves as noise.

[0118] This disclosure does not impose any special limitations on how to perform redundancy removal processing based on the time and coordinate components of each of the original event data in the original data.

[0119] In some embodiments, for any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes: detecting whether there is duplicate event data within a preset time range whose coordinate component is the same as the coordinate component of the original event data and whose polarity component is the same as the polarity component of the original event; and deleting the duplicate event data if the duplicate event data exists within the preset time range.

[0120] In some embodiments, redundancy removal processing based on the time component and coordinate component of any one of the original event data in the original data can be performed as follows:

[0121] i. Determine ΔT2;

[0122] ii. Read raw event data (x, y, t, p) from the raw data;

[0123] iii. Detect whether there are duplicate event data within a preset time range of (t-ΔT2, t+ΔT2) whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data;

[0124] iv. If duplicate event data exists within the preset time range, delete the duplicate event data.

[0125] Perform the above deduplication process on all raw event data in the original data in sequence to remove duplicate event data from the original data.

[0126] It should be noted that the above redundancy removal process can remove redundant data generated by DVS dense acquisition, thereby improving the target detection accuracy.

[0127] This disclosure does not impose any special limitations on how to perform event supplementation processing based on the time and coordinate components of each of the original event data in the original data.

[0128] In some embodiments, for any one of the original event data in the original data, event supplementation processing based on the time component and coordinate component of each of the original event data in the original data includes: inserting supplementary event data with a predetermined step size in a time interval centered on the time component of the original event data.

[0129] In some embodiments, inserting supplementary event data at predetermined steps within a time interval centered on the time component of the original event data includes:

[0130] The system detects whether first and second original event data exist; the first and second original event data, as well as the coordinate and polarity components of the original event data, are identical; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment; when the first and second original event data exist, at least one supplementary event data is inserted between the first and second original event data with a predetermined time step, wherein the supplementary event data, as well as the coordinate and polarity components of the original event data, are identical.

[0131] In some embodiments, event supplementation processing based on the time component and coordinate component of any one of the original event data in the original data can be performed as follows:

[0132] i. Determine the preset time increment ΔT3 and the predetermined time step dt, where dt < ΔT3;

[0133] ii. Read a raw event data (x, y, t, p) from the raw data. The time interval centered on the time component of the raw event data can be represented as (t-ΔT3, t+ΔT3).

[0134] iii. If the presence of first original event data (x, y, t-ΔT3, p) and second original event data (x, y, t+ΔT3, p) is detected in the original data, supplementary event data (x, y, t-ΔT3, t+ΔT3) are inserted in the time interval (t-ΔT3, t+ΔT3) centered on the original event data (x, y, t-ΔT3, p) between the first original event data (x, y, t-ΔT3, p) and the second original event data (x, y, t+ΔT3, p).

[0135] Perform the above event supplementation processing on all raw event data of the original data in sequence, and insert supplementary event data into the raw event data of the original data.

[0136] It should be noted that the above event supplementation processing can supplement missing event data in the case of sparse DVS acquisition, avoid frame breaks, and thus improve the target detection accuracy.

[0137] This embodiment of the disclosure does not impose any special limitations on how the data augmentation transformation that transforms the polarity components of the event data is applied to process each of the original event data in the original data in step S120.

[0138] In some embodiments, processing each of the original event data in the original data according to the data augmentation transformation that transforms the polarity components of the event data includes: transforming the polarity components of each of the original event data to obtain each of the augmented event data constituting the augmented data, wherein the polarity of the events corresponding to the augmented event data is opposite to the polarity of the events corresponding to the original event data.

[0139] This disclosure does not specifically limit how each of the original event data in the original data is processed according to a data augmentation transformation that transforms at least one coordinate component of the event data.

[0140] In some embodiments, processing each of the original event data in the original data according to the data augmentation transformation that transforms the coordinate components of the event data includes: scaling the coordinate components of each of the original event data in the original data; or shifting the coordinate components of each of the original event data in the original data; or rotating the coordinate components of each of the original event data in the original data.

[0141] like Figure 2As shown, the multi-channel augmented data, after data augmentation processing, is used to generate frame data through a framing layer, and then input into the target detection network in the network layer for target detection. In this embodiment of the disclosure, generating frame data through the framing layer means combining one channel of data to be detected within a specific time window into a frame to obtain frame data.

[0142] Accordingly, in some embodiments, reference is made to Figure 5 Step S200 includes:

[0143] In step S210, the frame data to be detected corresponding to each channel of enhanced data is generated based on each channel of enhanced data;

[0144] In step S220, the initial detection results of each channel are obtained based on the data of each channel of the frame to be detected.

[0145] The frame data to be predicted may include at least one event frame.

[0146] In some embodiments, refer to Figure 6 Step S210 includes:

[0147] In step S211, the event stream composed of each enhanced data stream is aggregated to generate event frame data;

[0148] In step S212, multiple sequentially adjacent event frame data are superimposed to generate the frame data to be detected.

[0149] It should be noted that in step S211, the event stream consisting of one stream of data to be detected within a specific time window is aggregated to generate event frame data.

[0150] In some embodiments, step S210 may be performed as follows:

[0151] The input augmentation data for the framing layer consists of multiple augmentation event data in the form of (x, y, t, p) quadruples. The augmentation event data within the time window Δt forms the event stream E. t′ ={e i |e i =[x i y i ,t′,p i The framing layer aggregates the event stream and outputs event frame data X. t =q(E t′The aggregation function q(·) can be any of the following: nonpolar function aggregation, cumulative aggregation, and logical operation aggregation; this embodiment does not impose any special limitation on this. The event frame data is in the form (c, h, w), where c represents the channel, h represents the height of the frame data, and w represents the width of the frame data. Finally, T adjacent event frame data are superimposed in chronological order to form the detection frame data in the form (T, c, h, w).

[0152] In some embodiments, when the frame data to be detected includes multiple event frame data, it can be processed for prediction by a network layer based on a spiking neural network. A network layer based on a spiking neural network can better understand the temporal information between multiple event frame data in the frame data to be detected, which is formed by superimposing them in chronological order, thereby improving the accuracy of target detection.

[0153] like Figure 2 As shown, the fusion layer obtains the target detection result based on the multiple initial detection results. This embodiment of the disclosure does not impose any special limitations on how the target detection result is obtained from the multiple initial detection results.

[0154] In some embodiments, refer to Figure 7 The step of obtaining the target detection result for the target object in the original data based on the at least one initial detection result includes:

[0155] In step S310, a target detection box is obtained based on the at least one initial detection result, and the target detection box represents the position information of the target object;

[0156] In step S320, duplicates are removed from the target detection boxes to obtain the target detection results.

[0157] This disclosure does not specifically limit how to deduplicate the target detection boxes. In some embodiments, non-maximum suppression (NMS) is applied to the target detection boxes to deduplicate them, resulting in the final result.

[0158] like Figure 2 As shown, an inverse transform layer is also included between the network layer and the fusion layer to perform the inverse transform based on the data augmentation transform. In target detection tasks that are sensitive to object size and position, this avoids the impact of the data augmentation transform on target detection accuracy.

[0159] In some embodiments, refer to Figure 8 Before the step of obtaining the target detection result for detecting the target object in the original data based on the at least one initial detection result, the target detection method further includes:

[0160] In step S400, for at least one initial detection result that corresponds to at least one data augmentation transformation that transforms the coordinate components of the event data, an inverse transformation corresponding to the data augmentation transformation that transforms the coordinate components of the event data is performed.

[0161] Reference Figure 8 The step of obtaining the target detection result for the target object in the original data based on the at least one initial detection result is performed as follows:

[0162] In step S330, the target detection result is obtained based on the at least one initial detection result after inverse transformation.

[0163] It should be noted that the at least one initial detection result after the inverse transformation can be understood as all initial detection results including the at least one initial detection result after the inverse transformation process. Specifically, some or all of the initial detection results may be the initial detection results obtained after the inverse transformation process.

[0164] Secondly, referring to Figure 9 This disclosure provides a target detection device, comprising:

[0165] The data augmentation module 110 is used to augment the raw data acquired by the dynamic vision sensor according to at least one data augmentation transformation to obtain at least one channel of augmented data, wherein each channel of augmented data corresponds to at least one data augmentation transformation.

[0166] Detection module 120 is used to input the at least one enhanced data into the target detection network to obtain at least one initial detection result for detecting the target object in the original data, wherein the at least one initial detection result corresponds one-to-one with the at least one enhanced data;

[0167] The fusion module 130 is used to obtain a target detection result for detecting the target object in the original data based on the at least one initial detection result.

[0168] In some embodiments, the raw data includes multiple raw event data, each of which corresponds to an event; the raw event data includes coordinate components, time components, and polarity components, wherein the coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event.

[0169] The data augmentation module 110 is used to process each of the original event data in the original data according to various data augmentation transformations that filter event data, to obtain one path of augmented data; and / or

[0170] The data augmentation module 110 is used to obtain one channel of augmented data based on multiple data augmentation transformations that filter event data, at least one data augmentation transformation that transforms the coordinate components of the event data, and / or a data augmentation transformation that transforms the polarity components of the event data.

[0171] In some embodiments, in the data augmentation module 110, processing each of the original event data in the original data according to various data augmentation transformations that filter event data includes:

[0172] The original event data in the original data is processed by denoising, redundancy removal, and event supplementation based on the time and coordinate components of each event data.

[0173] In some embodiments, in the data enhancement module 110, for any one of the original event data in the original data, denoising processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0174] Detect whether there is event data falling within a preset spatiotemporal range centered on the original event data;

[0175] If no event data falls within a preset spatiotemporal range centered on the original event data, the original event data is deleted.

[0176] In some embodiments, in the data enhancement module 110, for any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes:

[0177] Detect whether there are duplicate event data within a preset time range whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data;

[0178] If duplicate event data exists within the preset time range, the duplicate event data is deleted.

[0179] In some embodiments, in the data enhancement module 110, for any one of the original event data in the original data, event supplementation processing based on the time component and coordinate component of each of the original event data in the original data includes:

[0180] Supplementary event data is inserted at predetermined steps within a time interval centered on the time component of the original event data.

[0181] In some embodiments, in the data augmentation module 110, inserting supplementary event data at predetermined steps within a time interval centered on the time component of the original event data includes:

[0182] Detect the existence of first and second original event data; the first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment;

[0183] When the first original event data and the second original event data exist, at least one supplementary event data is inserted between the first original event data and the second original event data at a predetermined time step. The coordinate components and polarity components of the supplementary event data and the original event data are the same.

[0184] In some embodiments, the detection module 120 includes:

[0185] A framing unit is used to generate the frame data to be detected corresponding to each of the enhanced data streams based on each of the enhanced data streams.

[0186] The detection unit is used to input the data of each of the frames to be detected into the target detection network to obtain the initial detection results of each of the paths.

[0187] In some embodiments, the framing unit is used to aggregate the event stream composed of each of the enhanced data streams to generate event frame data; and to superimpose multiple sequentially adjacent event frame data to generate the frame data to be detected.

[0188] In some embodiments, the fusion module 130 is configured to, when including multiple initial detection results, obtain multiple target detection boxes based on the multiple initial detection results, wherein the target detection boxes represent the position information of the target object; and deduplicate the multiple target detection boxes to obtain the target detection result.

[0189] In some embodiments, the target detection device further includes:

[0190] The inverse transformation module is used to perform an inverse transformation corresponding to the data augmentation transformation that transforms the coordinate components of the event data in at least one initial detection result that corresponds to at least one data augmentation transformation that transforms the coordinate components of the event data.

[0191] The fusion module 130 is used to obtain the target detection result based on the at least one initial detection result after inverse transformation.

[0192] Thirdly, referring to Figure 10 This disclosure provides an electronic device, including:

[0193] One or more processors 201;

[0194] The memory 202 stores one or more programs, which, when executed by one or more processors, enable the one or more processors to implement any of the target detection methods described in the first aspect of the present disclosure.

[0195] One or more I / O interfaces 203 are connected between the processor and the memory and configured to enable information exchange between the processor and the memory.

[0196] Among them, processor 201 is a device with data processing capabilities, including but not limited to central processing unit (CPU); memory 202 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); I / O interface (read-write interface) 203 is connected between processor 201 and memory 202, and can realize information interaction between processor 201 and memory 202, including but not limited to data bus (Bus).

[0197] In some embodiments, the processor 201, memory 202, and I / O interface 203 are interconnected via bus 204, and thus connected to other components of the computing device.

[0198] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements any of the target detection methods described in the first aspect of this disclosure.

[0199] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0200] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A target detection method, comprising: The raw data collected by the event information acquisition device is enhanced by at least one data augmentation transformation to obtain at least one enhanced data stream. The event information acquisition device is an event-driven sensor. Each enhanced data stream corresponds to at least one data augmentation transformation. The raw data includes multiple raw event data, each of which corresponds to one event. The raw event data includes coordinate components, time components, and polarity components. The coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event. The at least one enhanced data stream is input into the target detection network to obtain at least one initial detection result for detecting the target object in the original data, and the at least one initial detection result corresponds one-to-one with the at least one enhanced data stream. The target detection result is obtained by detecting the target object in the original data based on the at least one initial detection result.

2. The target detection method according to claim 1, wherein the step of enhancing the raw data collected by the event information acquisition device according to at least one data augmentation transformation to obtain at least one channel of enhanced data includes: The original event data in the original data is processed by various data augmentation transformations that filter event data to obtain one path of augmented data; and / or An augmented data stream is obtained by using multiple data augmentation transformations to filter event data, as well as at least one data augmentation transformation to transform the coordinate components of the event data and / or the polarity components of the event data.

3. The target detection method according to claim 2, wherein, The processing of each of the original event data in the original data includes various data augmentation transformations that filter event data, including: The original event data in the original data is processed by denoising, redundancy removal, and event supplementation based on the time and coordinate components of each event data.

4. The target detection method according to claim 3, wherein, For any one of the original event data in the original data, the noise reduction process based on the time component and coordinate component of each original event data in the original data includes: Detect whether there is event data falling within a preset spatiotemporal range centered on the original event data; If no event data falls within a preset spatiotemporal range centered on the original event data, the original event data is deleted.

5. The target detection method according to claim 3, wherein, For any one of the original event data in the original data, the redundancy removal process based on the time component and coordinate component of each of the original event data in the original data includes: Detect whether there are duplicate event data within a preset time range whose coordinate components are the same as those of the original event data and whose polarity components are the same as those of the original event data; If duplicate event data exists within the preset time range, the duplicate event data is deleted.

6. The target detection method according to claim 3, wherein, For any one of the original event data in the original data, the event supplementation processing based on the time component and coordinate component of each original event data in the original data includes: Supplementary event data is inserted at predetermined steps within a time interval centered on the time component of the original event data.

7. The target detection method according to claim 6, wherein inserting supplementary event data with a predetermined step size within a time interval centered on the time component of the original event data comprises: Detect whether there is first and second original event data; The first original event data, the second original event data, and the coordinate and polarity components of the original event data are all the same; the difference between the time component of the original event data and the time component of the first original event data is equal to a preset time increment, and the difference between the time component of the second original event data and the time component of the original event data is equal to the preset time increment; When the first original event data and the second original event data exist, at least one supplementary event data is inserted between the first original event data and the second original event data at a predetermined time step. The coordinate components and polarity components of the supplementary event data and the original event data are the same.

8. The target detection method according to any one of claims 1 to 7, wherein, The step of inputting the at least one enhanced data stream into the target detection network to obtain at least one initial detection result for detecting the target object in the original data includes: Generate the frame data to be detected corresponding to each channel of enhanced data based on each channel of enhanced data; The target detection network is input into each of the target detection frames to obtain the initial detection results for each of the paths.

9. The target detection method according to claim 8, wherein, The steps of generating the frame data to be detected corresponding to each channel of enhanced data based on each channel of enhanced data include: The event stream composed of each enhanced data stream is aggregated to generate event frame data; The detection frame data is generated by superimposing multiple sequentially adjacent event frame data.

10. The target detection method according to any one of claims 1 to 7, wherein, When multiple initial detection results are included, the step of obtaining a target detection result for detecting the target object in the original data based on at least one initial detection result includes: Multiple target detection boxes are obtained based on the multi-path initial detection results, and the target detection boxes represent the position information of the target object; The multiple target detection boxes are deduplicated to obtain the target detection results.

11. The target detection method according to any one of claims 1 to 7, wherein, Before the step of obtaining the target detection result for detecting the target object in the original data based on the at least one initial detection result, the target detection method further includes: For at least one initial detection result that corresponds to at least one data augmentation transformation that transforms the coordinate components of the event data, perform an inverse transformation corresponding to the data augmentation transformation that transforms the coordinate components of the event data. The steps for obtaining target detection results for detecting target objects in the original data based on at least one initial detection result include: The target detection result is obtained based on the at least one initial detection result after inverse transformation.

12. A target detection device, comprising: A data augmentation module is used to enhance the raw data collected by the event information acquisition device according to at least one data augmentation transformation to obtain at least one channel of enhanced data. Each channel of enhanced data corresponds to at least one data augmentation transformation. The event information acquisition device is an event-driven sensor. The raw data includes multiple raw event data, each of which corresponds to an event. The raw event data includes coordinate components, time components, and polarity components. The coordinate components represent the coordinates of the corresponding event, the time components represent the occurrence time of the corresponding event, and the polarity components represent the polarity of the corresponding event. The detection module is used to input the at least one enhanced data into the target detection network to obtain at least one initial detection result for detecting the target object in the original data, wherein the at least one initial detection result corresponds one-to-one with the at least one enhanced data; The fusion module is used to obtain the target detection result for detecting the target object in the original data based on the at least one initial detection result.

13. An electronic device, comprising: One or more processors; A memory having stored one or more programs thereon, which, when executed by one or more processors, cause the one or more processors to implement the target detection method according to any one of claims 1 to 11; One or more I / O interfaces are connected between the processor and the memory and configured to enable information interaction between the processor and the memory.

14. A computer-readable medium having a computer program stored thereon, the program being executed by a processor to implement the target detection method according to any one of claims 1 to 11.

Images