A test tube detection method based on event camera

Abstract

The present application relates to the technical field of target detection, and particularly relates to a test tube detection method based on an event camera, comprising: acquiring an event data set output by the event camera in a preset time period; performing filtering processing on the event data set by a preprocessing module to obtain an event set; inputting the event set into a detection module to predict a center position, an offset position and a classification result of a test tube to obtain test tube position information; and calibrating the test tube position information according to a time point of a captured image, a motion trail and a time point of a manipulator to calculate current position information of the test tube. The event camera judges whether an event is formed according to brightness change of each pixel sensor, and if a threshold is reached, the corresponding pixel sensor generates an event stream output. Therefore, the event has the advantages of fast response and no motion blur, and the accuracy of the detection result is improved, and the problem of low accuracy of the detection result of the existing test tube detection method is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, in particular to a test tube detection method based on an event camera. BACKGROUND

[0002] With the development of science and technology, medical instruments are gradually developing towards automation and rapid detection. The detection of objects such as samples or sample carriers is one of the commonly used automatic detection technologies in medical instruments. Taking test tubes as an example, there are different types of test tubes such as venous blood, peripheral blood, quality control materials, and cleaning liquids in blood cell analyzers, and different types of test tubes such as samples, quality controls, and cleaning liquids in urine analyzers. Therefore, in automatic instruments, it is often necessary to identify test tubes to distinguish the types of test tubes to complete different functional operations. These methods usually include scanning barcodes on test tubes, identifying test tube shapes, and identifying test tube cap colors to distinguish different types of test tubes.

[0003] In some large sample analysis instruments, samples are stored in sample trays or test tube racks, and test tubes are usually classified into different types to match different functions. In automatic operation, it is usually necessary to detect whether a test tube exists in the corresponding test tube hole of a sample tray or a test tube rack and to distinguish the type of the test tube. After detecting the presence of the test tube, a mechanical hand is used to grab or sample the corresponding test tube to complete the corresponding automatic action.

[0004] The existing technology usually uses a CMOS or CCD camera to obtain images by taking pictures, and then obtains corresponding data through image processing. Image processing methods usually include traditional machine vision methods such as binarization and circle detection, and deep neural network methods. However, the frame rate of a general camera determines that motion blur is easy to occur when the instrument runs at high speed, which greatly affects the detection result. At the same time, over-bright / over-dark conditions caused by environmental light interference in different hospitals also affect the detection effect. Thus, the accuracy of the final result is reduced. SUMMARY

[0005] The present application aims to provide a test tube detection method based on an event camera, which aims to solve the problem of low accuracy of the detection result of the existing test tube detection method.

[0006] To achieve the above-mentioned purpose, the present application provides a test tube detection method based on an event camera, comprising the following steps:

[0007] Obtain an event data set output by an event camera in a preset time period;

[0008] A preprocessing module filters the event data set to obtain an event set;

[0009] The event set is input into a detection module to predict a center position, an offset position and a classification result of the test tube, and to obtain test tube position information;

[0010] The test tube position information is calibrated according to a time point of capturing an image and a motion trajectory and a time point of the manipulator, and current position information of the test tube is calculated.

[0011] The event data includes generated data of a pixel sensor with a luminance change exceeding a certain threshold at a current time point;

[0012] The generated data includes a pixel coordinate, a timestamp and a polarity of the pixel sensor.

[0013] The preprocessing module filters the event data set to obtain an event set, including:

[0014] Scattered points in a dense part of each event data in the event data set are fitted into a circle, and scattered points that cannot be fitted and have a density less than a preset value are removed to obtain filtered data of each event data;

[0015] The filtered data in a preset time is merged to obtain a merged data set;

[0016] Scattered points in a dense part of the merged data set are fitted into a circle, and scattered points that cannot be fitted and have a density less than a preset value are removed to obtain an event set.

[0017] The detection module includes any one or a combination of both of a spiking neural network and an artificial neural network trained based on a spike data set.

[0018] The artificial neural network is YOLOv3-tiny.

[0019] The event set is input into the combination of the spiking neural network and the artificial neural network trained based on the spike data set to predict a center position, an offset position and a classification result of the test tube, and to obtain test tube position information, including:

[0020] In the spiking neural network, a spikeConv layer and a spikePooling layer replace a Conv layer and a Pooling layer of the artificial neural network, an IF neuron replaces an activation function Relu after each Pooling layer, a leakyRelu replaces a Relu function in the artificial neural network, and the combination of the spiking neural network and the artificial neural network trained based on the spike data set is obtained.

[0021] The event set is input into the combined network, the pulse neural network of the combined network extracts features, and the extracted features are input into an artificial neural network to predict the center position, offset position and classification result of the test tube, so as to obtain test tube position information.

[0022] The test tube position information is calibrated according to the time point of the captured image and the motion trajectory and time point of the manipulator, and the current position information of the test tube is calculated.

[0023] The NMS operation is performed on the test tube position information to obtain a plurality of detection boxes.

[0024] The plurality of detection boxes are screened, and the detection boxes meeting the preset value of coincidence degree are merged to obtain a final detection box.

[0025] The test tube position information is calibrated according to the time point of the captured image and the motion trajectory and time point of the manipulator based on the final detection box, and the current position information of the test tube is calculated.

[0026] The test tube detection method based on the event camera provided by the application comprises the following steps: acquiring an event data set output by an event camera in a preset time period; performing filtering processing on the event data set by a preprocessing module to obtain an event set; inputting the event set into a detection module to predict a center position, an offset position and a classification result of a test tube, so as to obtain test tube position information; and calibrating the test tube position information according to a time point of a captured image and a motion trajectory and time point of a manipulator, and calculating current position information of the test tube. The event camera judges whether an event is formed according to the brightness change of each pixel sensor, and if the threshold value is reached, the corresponding pixel sensor generates an event stream output. Therefore, the event has the advantages of fast response and no motion blur, the accuracy of the detection result is improved, and the problem of low accuracy of the detection result of the existing test tube detection method is solved. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0028] Figure 1 It is a schematic diagram of the principle of the test tube detection method based on the event camera provided by the application.

[0029] Figure 2 It is a schematic diagram of the principle of the test tube detection method based on the event camera provided by the application.

[0030] Figure 3is the data after preprocessing converted into an image.

[0031] Figure 4 is a network structure diagram of YOLOv3-tiny.

[0032] Figure 5 is a structure diagram of a combined network.

[0033] Figure 6 is a test tube of different types.

[0034] Figure 7 is an output image.

[0035] Figure 8 is a flowchart of a test tube detection method based on an event camera provided by the present application. DETAILED DESCRIPTION

[0036] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numbers represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0037] Referring to Figures 1 to 8 , the present application provides a test tube detection method based on an event camera, comprising the following steps:

[0038] S1 acquiring an event data set output by an event camera in a preset time period;

[0039] Specifically, a time period, i.e. a preset time period, is specified, and the time period should be at the level of milliseconds or microseconds, and all data collected in the time period is captured. The event data includes generated data of a pixel sensor whose brightness changes by more than a certain threshold at the current time; the generated data includes the pixel coordinates of the pixel sensor, a timestamp, and a polarity (the direction of darkening or brightening), but the data generated at one time may be too sparse, so the data generated in one millisecond is taken as the input of the preprocessing module (about 100 data).

[0040] The detection method using an event camera can solve the problem that the detection result is greatly affected by motion blur caused by the transmission frame rate of an ordinary camera when the instrument runs at high speed; and the problem that the over-bright / over-dark situation caused by the environmental light interference of different hospitals affects the detection effect, the event camera judges whether an event is formed according to the brightness change of each pixel sensor, and if the threshold is reached, the corresponding pixel sensor generates an event stream output, so the event has the advantages of fast response and no motion blur, and its sampling rate can be as high as one million hertz.

[0041] S2 filtering the event data set by a preprocessing module to obtain an event set;

[0042] Specifically, filtering is performed on each group of data in the event data set. The filtering includes: 1. using a clustering algorithm to classify the scatter points according to a specific area; 2. filtering the scatter points that are less than a certain value or are not classified. After filtering, all images in a time period are combined, and then filtering is performed again.

[0043] Specifically, the filtering includes the following steps:

[0044] S21, fitting the scatter points in the dense part of each event data in the event data set into a circle, removing the scatter points that cannot be fitted and have a density less than a preset value, to obtain filtered data of each event data;

[0045] S22, combining the filtered data in a preset time to obtain a combined data set;

[0046] S23, fitting the scatter points in the dense part of the combined data set into a circle, removing the scatter points that cannot be fitted and have a density less than a preset value, to obtain an event set.

[0047] Specifically, in some cases, the preprocessing module can be removed. If a single event data contains sufficient information, multiple event data need not be combined, or the event camera output need not be filtered and directly input into the detection module.

[0048] S3, inputting the event set into a detection module to predict the center position, offset position and classification result of the test tube, to obtain test tube position information;

[0049] Specifically, the detection module includes any one or a combination of both of a spiking neural network and an artificial neural network trained based on a spiking data set. The artificial neural network is YOLOv3-tiny. When the YOLOv3-tiny is used for prediction, the output is the predicted test tube center position (x, y), offset position (offset x, offset y), and test tube classification result n.

[0050] The inputting of the event set into the combination of the spiking neural network and the artificial neural network trained based on the spiking data set to predict the center position, offset position and classification result of the test tube, to obtain test tube position information, includes: converting the processed event set into an event data set, passing through a spiking neural network composed of multiple spiking convolution-spiking pooling-IF neurons, and finally connecting to an artificial neural network for classification prediction, taking the two layers in the network as the output of the feature map, inputting into the artificial neural network for bounding box prediction, and finally performing NMS processing on the data of the prediction box to obtain the final position and classification information. Specifically, the following steps are included:

[0051] S31 replaces the Conv layer, the Pooling layer of the artificial neural network with the spikeConv layer, the spikePooling layer in the spiking neural network, and replaces the activation function Relu with the IF neuron after each Pooling layer, replaces the Relu function with the leakyRelu in the artificial neural network, to obtain a combined network of the spiking neural network and the artificial neural network trained based on the spike dataset;

[0052] Specifically, the spiking neural network uses a model that best fits the mechanism of biological neurons for calculation, which is closer to the mechanism of biological neurons. The neurons in the spiking neural network only transmit information after the membrane potential reaches a certain value, sending signals to adjacent neurons, and the posterior neurons can increase or decrease this potential. Such characteristics can well cooperate with the event camera to process data.

[0053] In the model of the spiking neural network, the IF neuron is the basic component structure, and the IF neuron accumulates the input z to the membrane voltage The following represents:

[0054]

[0055] The membrane voltage calculation of the jth neuron of the first layer is as formula 1, is a pulse, V th is a critical voltage, is the input of the neuron, and the formula is:

[0056]

[0057] The multiple inputs are accumulated, and w and b are weights and biases:

[0058]

[0059] When the membrane voltage is greater than the critical voltage V th , a pulse is generated. U is a unit step function, which satisfies an output of 1 and an output of 0 when not satisfied.

[0060] The spiking neural network uses a pulse sequence for information transmission between neurons, Figure 5The structure diagram of the example pulse neural network combined with an artificial neural network. In the pulse neural network, spikeConv and spikePooling replace the Conv and Pooling layers of the artificial neural network, and an IF neuron replaces the activation function Relu after each Pooling layer. The pulse neural network extracts features and inputs them into the artificial neural network. In the artificial neural network, a leakyRelu function replaces the Relu function. The artificial neural network outputs the final test tube prediction result. Finally, the center position, offset distance and classification result of the test tube are output.

[0061] S32 inputs the event set into the combined network. The pulse neural network of the combined network extracts features, and inputs the extracted features into the artificial neural network to predict the center position, offset position and classification result of the test tube, thereby obtaining the test tube position information.

[0062] Specifically, neither the pulse neural network nor the artificial neural network needs to be limited to the above method. Similar detection networks can have similar effects.

[0063] S4 calibrates the test tube position information according to the time point of the captured image and the motion trajectory and time point of the manipulator, and calculates the current position information of the test tube.

[0064] The specific manner is as follows:

[0065] S41 performs an NMS operation on the test tube position information, thereby obtaining a plurality of detection boxes.

[0066] S42 screens the plurality of detection boxes, merges the detection boxes with a coincidence degree meeting a preset value, and obtains a final detection box.

[0067] S43 calibrates the test tube position information according to the time point of the captured image and the motion trajectory and time point of the manipulator based on the final detection box, and calculates the current position information of the test tube.

[0068] Specifically, after the calibration module obtains the output result (the position information of the test tube), performs an NMS operation, screens the generated detection boxes, and merges two detection boxes if the coincidence degree meets a certain value. After obtaining the final detection box (the positioning box), the output result is calibrated according to the time point of the captured image and the motion trajectory and time point of the manipulator, and the current position information of the test tube is calculated. Assuming that the output result of step 4 is (X, Y), the time point of the captured image is T0, the current time point is T1, and the speed of the lens movement is v, the output result X t , Y t is formula (4):

[0069] (X t , Y t) = (X, Y) + v * (T1 - T0) (4)

[0070] If the detection speed is fast enough, the calibration module can be removed. In the present application, test tubes are used as an example, and the detection objects include but are not limited to test tubes.

[0071] The above only discloses a preferred embodiment of a test tube detection method based on an event camera of the present application, and of course cannot limit the scope of the present application. Those skilled in the art can understand that all or part of the above-mentioned embodiments can be implemented, and equivalent changes made according to the claims of the present application still fall within the scope of the present application.

Claims

1. A test tube detection method based on an event camera, characterized in that, Includes the following steps: Obtain the event dataset output by the event camera within a preset time period; The preprocessing module filters the event dataset to obtain an event set; The event set is input into the detection module to predict the center position, offset position and classification result of the test tube, and the test tube position information is obtained. The test tube position information is calibrated based on the time point of the captured image and the movement trajectory and time point of the robotic arm, and the current position information of the test tube is calculated.

2. The test tube detection method based on an event camera as described in claim 1, characterized in that, The event data includes data generated by pixel sensors when the brightness change exceeds a certain threshold at the current moment; The generated data includes the pixel coordinates, timestamps, and polarity of the pixel sensor.

3. The test tube detection method based on an event camera as described in claim 2, characterized in that, The preprocessing module filters the event dataset to obtain an event set, including: The scattered points of the dense part of each event data in the event dataset are fitted into a circle, and scattered points that cannot be fitted or whose density is less than a preset value are removed to obtain the filtered data of each event data. The filtered data within a preset time period are merged to obtain a merged dataset; The scattered points of the dense part of the merged dataset are fitted into circles, and scattered points that cannot be fitted or whose density is less than a preset value are removed to obtain the event set.

4. The test tube detection method based on an event camera as described in claim 1, characterized in that, The detection module includes any one or a combination of two of the following: a spiking neural network and an artificial neural network trained on a spiking dataset.

5. The test tube detection method based on an event camera as described in claim 4, characterized in that, The artificial neural network is YOLOv3-tiny.

6. The test tube detection method based on an event camera as described in claim 4, characterized in that, The process of inputting the event set into a combined network of the spiking neural network and the artificial neural network trained on the spiking dataset to predict the center position, offset position, and classification result of the test tube, thereby obtaining test tube position information, includes: In the spiking neural network, the spikeConv layer and spikePooling layer are used to replace the Conv layer and Pooling layer of the artificial neural network. After each Pooling layer, the IF neuron is used to replace the ReLU activation function. In the artificial neural network, the leakyReLU function is used to replace the ReLU function, thus obtaining the combined network of the spiking neural network and the artificial neural network trained based on the spiking dataset. The event set is input into the combined network, the spiking neural network of the combined network extracts features, and the extracted features are input into the artificial neural network to predict the center position, offset position and classification result of the test tube, thereby obtaining the test tube position information.

7. The test tube detection method based on an event camera as described in claim 1, characterized in that, The step of calibrating the test tube position information based on the time point of the captured image and the movement trajectory and time point of the robotic arm, and calculating the current position information of the test tube, includes: The test tube position information is subjected to NMS operation to obtain multiple detection frames; The multiple detection frames are screened, and the detection frames whose overlap meets the preset value are merged to obtain the final detection frame; Based on the final detection frame, the test tube position information is calibrated according to the time point of the captured image and the movement trajectory and time point of the robot arm, and the current position information of the test tube is calculated.

Images