Empty box detection method based on event camera

Through the empty box detection method combined with event camera and deep neural network, the problem of degradation of detection effect and difficulty in identification in the existing technology is solved, and high accuracy judgment of objects with small size, hollow structure and heavier weight is achieved. Low power consumption and on-site environmental requirements.

CN118550000BActive Publication Date: 2024-11-01SHENZHEN MAXVISION TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410897634.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-05
Publication Date
2024-11-01
Estimated Expiration
2044-07-05

AI Technical Summary

Technical Problem

The existing empty box detection methods have problems such as degradation of detection effect, impact on equipment service life, deviation of detection results and difficulty in identifying objects with smaller volume, hollow structure and heavier weight, especially the contactless gas-mediated excitation method is affected by environmental noise. And the excitation signal is quite different.

Method used

Using the event camera-based detection method, the event camera collects the vertical vibration signals evenly distributed on the side of the box, and combines infrared light dot matrix and deep neural network to perform signal encoding, downsampling and classification judgment, allowing longer-distance detection and reducing objects Position influences and improves accuracy through multiple judgments of voting rules.

Benefits of technology

It improves the accuracy of judging objects with small size, hollow structure and heavier weight, adapts to low power consumption needs, and does not affect the service life and detection yield of the equipment, and is suitable for on-site working environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118550000B_ABST
    Figure CN118550000B_ABST
Patent Text Reader

Abstract

The present invention discloses an empty box detection method based on an event camera, comprising: (A) collecting vertical vibration signals uniformly distributed on the side of a box through an event camera; (B) using an infrared light dot matrix with a preset frequency coding band to emit to the side of the box; (C) detecting light spots encoded with a preset frequency through an event camera to generate side dot matrices of the box and the vehicle, and calculating the vibration frequency and vibration amplitude of the light spots along the vertical direction through the displacement of the light spots; (D) encoding the collected vibration signal into a (x, y, t, p) format; (E) performing downsampling and compression processing on the vibration signal collected by the event camera to generate a processed time series signal; (F) sending the processed time series signal to a deep neural network for classification; (G) judging whether there is an object inside the box, and obtaining a probability value through model calculation. If the probability value is greater than or equal to the preset value, it is judged to be an empty box; if the probability value is less than the preset value, it is judged to be a non-empty box.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of empty box detection, and in particular to an empty box detection method based on an event camera. Background Art

[0002] The existing empty container detection methods are mainly divided into contact hammer excitation method and non-contact air medium excitation method. The contact hammer excitation method excites the bottom beam of the container body outside the box through an exciter, so that the container body vibrates to generate sound waves, and the sound waves generated inside the box are collected by a sound wave collection sensor. After processing and analyzing the sound waves, the internal sound field mode of the container is obtained, and compared with the preset empty box sound field. If they are the same, it is predicted to be an empty box, otherwise it is predicted to be a non-empty box; the non-contact air medium excitation method uses an air medium excitation signal to perform non-contact excitation on a specified position on the container, which also makes the container body vibrate to generate sound waves, and uses a laser sensor to collect the composite vibration response signal of the box at the excitation position on the container, and finally intelligently identifies whether the container is empty.

[0003] The contact hammer excitation method of the prior art has the following problems: it is necessary to contact the side of the container and find the crossbeam at the bottom of the side of the container so that the exciter can effectively excite the container. Inaccurate positioning will lead to a decrease in the detection effect. At the same time, the movement of the vehicle during the detection process will damage the detection equipment, affecting the service life and detection yield of the equipment; the non-contact air medium excitation method has the following problems: it is necessary for the excitation device to generate a signal of the same scale each time the detection is performed, but the excitation equipment produced by different manufacturers and batches will be more or less different, resulting in the excitation signal being affected by the excitation device. In addition, the excitation signal is transmitted through the air and will be affected by the surrounding environmental noise, resulting in deviations in the detection results. In addition, the prior art uses laser sensors to collect vibration signals at a certain point on the side of the container, and has high requirements on the position of the objects placed inside the container. If the position of the object is far away from the specific signal collection point, the recognition effect will be greatly reduced, and the deviation of the detection result will be large. In addition, when collecting vibration signals in the horizontal direction of the side to analyze whether there are objects inside the container, such signals are not sensitive to objects with small volume, hollow structure, and heavy weight, resulting in detection difficulties. Summary of the invention

[0004] One object of the present invention is to provide an empty box detection method based on an event camera, which can cover the signal acquisition to the entire side of the box, reduce the influence of the placement of objects inside the box on the detection yield, and improve the accuracy of judging objects with small volume, hollow structure, and heavy weight inside the box. At the same time, the event camera collects signals and allows detection at a longer distance, which is friendly to the on-site working environment and does not affect the service life and detection yield of the equipment. In addition, the SNN algorithm is used, which has low requirements on computing power and can meet the needs of low power consumption.

[0005] Other advantages and features of the present invention will become more apparent from the following detailed description and may be realized by means of the instrumentalities and combinations particularly pointed out in the appended claims.

[0006] According to one aspect of the present invention, an empty box detection method based on an event camera of the present invention, which can achieve the aforementioned objects and other objects and advantages, comprises the following steps:

[0007] (A) The event camera collects the vertical vibration signal evenly distributed on the side of the box;

[0008] (B) Using infrared light dot arrays with preset frequency coding bands to emit to the side of the box;

[0009] (C) The event camera detects the light spots encoded with the preset frequency, generates the side dot matrix of the box and the vehicle, and calculates the vibration frequency and vibration amplitude of the light spots in the vertical direction through the displacement of the light spots;

[0010] (D) Encode the collected vibration signal into (x, y, t, p) format;

[0011] (E) Downsampling and compressing the vibration signal acquired by the event camera to generate a processed time series signal;

[0012] (F) The processed time series signal is sent to the deep neural network for classification;

[0013] (G) Determine whether there is an object inside the box. Use the deep learning model obtained through training to calculate the probability value. If the probability value is greater than or equal to the preset value, it is judged to be an empty box. If the probability value is less than the preset value, it is judged to be a non-empty box.

[0014] According to one embodiment of the present invention, the step (H) further includes voting based on the results of multiple judgments, and the voting rule is to repeat steps (A) to (G) multiple times, and the number of repetitions is an odd number of times, so as to obtain an odd number of probability values, and calculate the average value of the probability values. If the average value is greater than or equal to a preset value, it is judged to be an empty box, and if the average value is less than the preset value, it is judged to be a non-empty box.

[0015] According to one embodiment of the present invention, the step (A) includes a sub-step (A1) that the event camera uses a filter with a wavelength band of 940 nm.

[0016] According to an embodiment of the present invention, the step (B) includes a sub-step (B1) that a fixed flashing frequency can be set for each light spot of each row or column of the light source.

[0017] According to one embodiment of the present invention, step (C) includes a sub-step (C1) of counting the number of times the light spot flashes N within 1 second, the vibration frequency H = 1 / N, and the vibration amplitude is the y-axis coordinate of the light spot in the image of the event camera.

[0018] According to one embodiment of the present invention, step (D) includes sub-step (D1): x and y are the row and column numbers of the light spot, t is the specific time, and p is the y-axis coordinate value of the light spot in the picture of the event camera.

[0019] According to an embodiment of the present invention, the step (E) includes a sub-step (E1) of calculating the mean value e and the standard deviation a according to the vertical movement amplitude p value of the collected vibration signal, wherein ,in, , i is any of (1,2,3...n), and n is the interval size of accumulated data along the time dimension t.

[0020] According to one embodiment of the present invention, the step (E) includes a sub-step (E2) of vertically moving the amplitude average value b = a / n, accumulating n P values ​​into {0, q1, q2, q3, q4...}, using the difference m between two adjacent q values, the m value is compared with the b value, and the m value greater than or equal to b is recorded as 1, and the m value less than b is recorded as 0, thereby generating a processed timing signal V.

[0021] According to one embodiment of the present invention, the step (G) includes a sub-step (G1) in which the probability value Z obtained is between the interval [0,1], the threshold is set to 0.5, and when the probability value Z is greater than or equal to 0.5, it is judged to be an empty box, and when the probability value Z is less than 0.5, it is judged to be a non-empty box.

[0022] According to one embodiment of the present invention, step (H) includes sub-step (H1) of repeating steps (A) to (G) 5 times to obtain 5 probability values ​​[z1, z2, z3, z4, z5], and then calculating the average value of the probability , where i is any of (1,2,3,4,5), according to the threshold 0.5, If it is greater than or equal to 0.5, it is an empty box. If it is less than 0.5, it is not an empty box.

[0023] The beneficial effects of the present invention are as follows: the present invention can cover the signal acquisition to the entire side of the box, reduce the influence of the placement of objects inside the box, and improve the accuracy of judging objects with small volume, hollow structure and heavy weight inside the box. At the same time, the event camera allows a longer distance for signal acquisition, which is friendly to the on-site working environment. In addition, the SNN algorithm is used, which has low requirements on computing power and can meet the needs of low power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The figure is a flowchart of the steps of an empty box detection method based on an event camera according to an embodiment of the present invention.

[0025] Figure 2 It is a schematic diagram of the difference between horizontal vibration and vertical vibration of the container side in the empty container detection method based on event camera in the above embodiment of the present invention.

[0026] Figure 3 Schematic diagram of dot matrix light source distribution of the empty box detection method based on event camera in the above embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0028] Those skilled in the art should understand that, in the disclosure of the present invention, the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.

[0029] It is understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number. In addition, in the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more than two.

[0030] In the description of the present invention, it should be understood that "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through a medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0031] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0032] like Figures 1 to 3 As shown, an empty box detection method based on an event camera of an embodiment of the present invention is disclosed. The empty box detection method based on the event camera can cover the signal acquisition to the entire side of the box, reducing the influence of the placement of objects inside the box, and improving the accuracy of judging objects with small volume, hollow structure and heavy weight inside the box. At the same time, the event camera allows a longer distance to collect vibration signals, which is friendly to the on-site working environment. In addition, the SNN algorithm is adopted, which has low requirements on computing power and can meet the needs of low power consumption.

[0033] like Figure 1 FIG. 1 is a flowchart of the method for detecting an empty box based on an event camera according to the above embodiment of the present invention, comprising the following steps:

[0034] (1) The vertical vibration signal evenly distributed on the side of the box is collected by the event camera. In detail, the uniform distribution of the vertical vibration signal is achieved by dot matrix infrared light. After the event camera is equipped with a 940nm band filter, only dot matrix infrared light is allowed to pass, thereby avoiding interference from natural light sources, such as Figure 2 , which is a schematic diagram of the difference between horizontal vibration and vertical vibration of the container side in the empty container detection method based on event camera in the above embodiment of the present invention;

[0035] (2) A 940nm infrared light dot matrix encoded with a preset frequency is used to emit to the entire truck, including the side of the container, to avoid interference from external light. The light spots of the preset frequency-encoded light source can reduce the interference of the later point tracking algorithm. Each light spot of each row or column of the light source can be set with a fixed flashing frequency, such as 50HZ, so that the collected light spots can be distinguished by frequency to a more accurate position. In detail, the event camera can capture pixel changes as low as 1 microsecond, reducing frequency information loss;

[0036] (3) The event camera detects the light spots encoded with the preset frequency and generates the side dot matrix of the box and the vehicle. The vibration frequency and vibration amplitude of the light spots along the vertical direction are calculated through the displacement of the light spots. Specifically, the number of times the light spots flash in 1 second is counted, the vibration frequency is H = 1 / N, and the vibration amplitude is the y-axis coordinate of the light spot in the image of the event camera, as follows: Figure 3 FIG. 1 is a schematic diagram of dot matrix light source distribution of an empty box detection method based on an event camera according to the above embodiment of the present invention;

[0037] (4) Encode the collected vibration signal into the (x, y, t, p) format, where x and y are the plane codes of the light spots in the infrared light dot matrix, where each point of the dot matrix light source is a light spot, t is the time, and p is the vertical movement amplitude of the light spot. Specifically, x and y are the row and column numbers of the light spot, respectively. For example, there are 8 rows and 8 columns of light spots in total, and the light spot in the first row and first column is coded as 0, 0, and the light spot in the first row and second column is coded as 0, 1..., where t is the specific time, and p is the y-axis coordinate value of the light spot in the image of the event camera, such as, ,

[0038] (5) Downsampling and compressing the vibration signal collected by the event camera. Specifically, the mean value e and standard deviation a are calculated based on the vertical movement amplitude p of the collected vibration signal, where ,in, , i is any one of (1,2,3...n), in detail, the vertical moving amplitude average value b = a / n, n is the interval size of the accumulated data along the time dimension t, that is, n is the compression ratio of the downsampled data of the dimension t, and the preferred interval of n is in the interval [2,10]. The number of P values ​​is accumulated into {0,q1,q2,q3,q4...}, and the difference m between two adjacent q values ​​is used. The m value is compared with the b value. The m value is greater than or equal to b and is recorded as 1, and the m value is less than b and is recorded as 0 to generate the processed time series signal V. Furthermore, when the collected signal is compressed, the extracted time-frequency feature is used for replacement. In the training stage, the feature is normalized, and then the CNN is used for classification training model to obtain the classification model of empty box and non-empty box; in the prediction stage, the feature is normalized, and then the classification model of empty box and non-empty box is used to predict whether the input category is an empty box;

[0039] (6) Sending the processed time series signal V to a deep neural network for classification, such as a spike neural network (SNN) classification, a convolutional neural network (CNN) or a self-attention neural network (TNSFER);

[0040] (7) Determine whether there is an object inside the box. The deep learning model of SNN, CNN or tansformer obtained through training is used to calculate the probability value, such as probability value Z = SNN (time series signal V). At this time, the obtained probability value Z is between [0, 1], and the threshold is preferably set to 0.5. When the probability value Z is greater than or equal to 0.5, it is judged to be an empty box, and when the probability value Z is less than 0.5, it is judged to be a non-empty box;

[0041] (8) Further, voting is performed based on the results of multiple judgments. The voting rule is to repeat steps (1) to (7) multiple times, preferably 5 times, to obtain 5 probability values ​​[z1, z2, z3, z4, z5], and then calculate the average probability value , where i is any of (1,2,3,4,5), according to the threshold 0.5, If it is greater than or equal to 0.5, it is an empty box. If it is less than 0.5, it is a non-empty box, which improves the recall rate of the empty box detection method based on event camera.

[0042] The event camera used in the empty box detection method based on event camera of the present invention can achieve low latency, high dynamic range, low power consumption and high time resolution. The event camera is used to collect the vibration signal of the box, and the signal in the frequency range of 1HZ to a maximum of 5000HZ can be effectively collected. In the specific implementation process, the vibration signals for judging whether there are objects inside the box are mostly concentrated in the frequency range below 1000HZ. The above data represents the data collected by the event camera, which can effectively judge whether there are objects inside the box.

[0043] The empty box detection method based on event camera of the present method adopts the vibration signal in the vertical direction, which is different from the vibration signal in the horizontal direction of the side. According to the principle of resonance of rigid objects, whether there is an object inside the box body will affect the frequency of resonance between the body and the container. Among them, the factors with greater influencing factors include the mass of the internal object. The weight of the object will affect the frequency of vibration of the container. The heavier the object, the easier it is to detect.

Claims

1. An empty box detection method based on event camera, characterized in that: The following steps are involved: (A) The vertical vibration signal evenly distributed on the side of the box is collected by the event camera. The uniform distribution of the vertical vibration signal is achieved by dot matrix infrared light; (B) Using infrared light dot arrays with preset frequency coding bands to emit to the side of the box; (C) The event camera detects the light spots encoded with the preset frequency, generates the side dot matrix of the box and the vehicle, and calculates the vibration frequency and vibration amplitude of the light spots along the vertical direction through the displacement of the light spots. The number of times the light spots flash in 1 second is counted, the vibration frequency is H = 1 / N, and the vibration amplitude is the y-axis coordinate of the light spot in the image of the event camera; (D) Encode the collected vibration signal into the (x, y, t, p) format, where x and y are the row and column numbers of the light spot, t is the specific time, and p is the y-axis coordinate value of the light spot in the image of the event camera; (E) Downsample and compress the vibration signal collected by the event camera to generate a processed time series signal. According to the p value, the mean e and standard deviation a are calculated, where ,in , i is any one of (1,2,3...n), n is the interval size of the accumulated data along the time dimension t, the vertical moving amplitude average value b = a / l, where l is any number in the interval [2.10], the number of p values ​​is accumulated into {0,q1,q2,q3,q4......}, the difference m between two adjacent q values ​​is used, the m value is compared with the b value, the m value greater than or equal to b is recorded as 1, and the m value less than b is recorded as 0, and the processed time series signal V is generated; (F) The processed time series signal is sent to a deep neural network for classification. The deep neural network is a pulse neural network (SNN). (G) The deep learning model obtained through training calculates the probability value Z=snn(time series signal V). If the probability value is greater than or equal to the preset value, it is judged as an empty box. If the probability value is less than the preset value, it is judged as a non-empty box.

2. The empty box detection method according to claim 1, characterized in that: The method further includes step (H) of voting based on the results of multiple judgments, wherein the voting rule is to repeat steps (A) to (G) multiple times, and the number of repetitions is an odd number, so as to obtain an odd number of probability values, and calculate the average value of the probability values. If the average value is greater than or equal to a preset value, it is judged to be an empty box, and if the average value is less than the preset value, it is judged to be a non-empty box.

3. The empty box detection method according to claim 2, characterized in that: The step (A) includes the sub-step (A1) that the event camera adopts a filter of the 940nm band.

4. The empty box detection method according to claim 3, characterized in that: The step (B) includes a sub-step (B1) that a fixed flashing frequency can be set for each light spot of each row or column of the light source.

5. The empty box detection method according to claim 1, characterized in that: The step (G) includes a sub-step (G1) in which the probability value Z obtained is between the interval [0, 1], the threshold is set to 0.5, when the probability value Z is greater than or equal to 0.5, it is judged to be an empty box, and when the probability value Z is less than 0.5, it is judged to be a non-empty box.

6. The empty box detection method according to claim 2, characterized in that: The step (H) includes a sub-step (H1) of repeating steps (A) to (G) 5 times to obtain 5 probability values ​​[z1, z2, z3, z4, z5], and then calculating the average value of the probability , where i is any of (1,2,3,4,5), according to the threshold 0.5, If it is greater than or equal to 0.5, it is an empty box. If it is less than 0.5, it is not an empty box.