Key point detection method and device based on light spot

Abstract

The present invention discloses a key point detection method and device based on light spots. In order to quickly detect the fusion or splitting of light spots, the key point detection method based on light spots of the present invention includes: the light spot generated by laser irradiation is at least partially set in the field of view of the event camera; the event stream is obtained from the event camera, and a clustering operation is performed on all events in the event window to obtain a number of clusters, and the cluster coordinates and cluster timestamps are obtained; at least according to the cluster coordinates, the movement trend of a pair of clusters is judged; if the movement trend meets the preset movement trend, and the distance between the cluster coordinates of the pair of clusters meets the preset range, the pair of clusters is judged to enter the cluster fusion state or the cluster split state, and the cluster coordinates and / or cluster timestamps are recorded as the association information of a key point; wherein the cluster timestamps of the pair of clusters are the same or in the same time period. The present invention has the advantages of low power consumption and low latency.

Description

Technical Field

[0001] The invention relates to the field of traditional Chinese medicine manufacturing, and in particular to a key point detection method and device based on light spots. Background Art

[0002] Light spot usually refers to the optical phenomenon produced by reflection and diffuse reflection after laser irradiates the surface of an object.

[0003] In some fields, including but not limited to the field of drug alcohol precipitation, light spots can be used as marking key points. For example, by identifying the splitting and fusion of light spots, the corresponding supernatant surface or the interface between the supernatant and the precipitate can be respectively identified, thereby completing tasks such as supernatant extraction.

[0004] If the splitting and fusion of light spots are detected based on traditional computer vision solutions, and frame images containing a large amount of redundant information are processed, it often means higher power consumption, higher computing costs and delays, which does not meet the actual needs of industrial production.

[0005] To this end, the present invention proposes a light spot detection solution based on an event camera (also called a dynamic vision sensor) such as a vision sensor to overcome the shortcomings of traditional solutions in terms of power consumption, cost or delay. Summary of the invention

[0006] In order to alleviate or partially alleviate the above technical problems, the solution of the present invention is as follows:

[0007] A key point detection method based on light spots comprises: a light spot generated by laser irradiation is at least partially set in the field of view of an event camera; an event stream is obtained from the event camera, and a clustering operation is performed on all events in the event window to obtain a plurality of clusters, and cluster coordinates and cluster timestamps are obtained; a movement trend of a pair of clusters is determined at least according to the cluster coordinates; if the movement trend conforms to a preset movement trend, and the distance between the cluster coordinates of the pair of clusters conforms to a preset range, the pair of clusters is determined to enter a cluster fusion state or a cluster split state, and the cluster coordinates and / or cluster timestamps are recorded as association information of a key point; wherein the cluster timestamps of the pair of clusters are the same or within the same time period.

[0008] Furthermore, before performing a clustering operation on all events in the event window, a noise reduction operation is first performed on the events in the event window to filter out noise events.

[0009] Furthermore, for any event within the event window, if the number of pixels in a preset area around the pixel generating the event reaches a threshold within a preset time interval where the timestamp of the event is located, the event is determined to be a valid event, otherwise it is a noise event.

[0010] Furthermore, the preset movement trend refers to that the pair of clusters are approaching or moving away.

[0011] Furthermore, determining whether the pair of clusters enter the cluster fusion state or the cluster split state is also based on whether the cluster coordinates of the pair of clusters are within a preset boundary area.

[0012] Furthermore, within a number of consecutive event windows, the radius of the neighborhood in the clustering operation is reduced with an exponential trend, and the movement trend of the pair of clusters is detected; if within the number of consecutive event windows, it is determined that the pair of clusters conform to a preset movement trend of approaching, and the distance between the cluster coordinates of the pair of clusters is within a preset range, then it is determined that the pair of clusters enter a cluster fusion state.

[0013] Further, for the pair of clusters that conform to the preset motion trend, a stack is set, and association information about the pair of clusters is stored in the stack.

[0014] Further, for the pair of clusters in the stack that no longer conform to the preset motion trend, the association information about the pair of clusters is removed from the stack.

[0015] Furthermore, in the alcohol precipitation tank, the height information related to the supernatant surface and / or the height information related to the interface between the supernatant and the sediment is determined according to the correlation information of the key points, thereby controlling the supernatant extraction action.

[0016] A key point detection device based on light spots, comprising a laser and an event camera; wherein the laser is used to generate laser light; the light spot generated by the laser irradiation is at least partially arranged in the field of view of the event camera; the key point detection device based on light spots obtains the associated information of the key points according to the key point detection method based on light spots described in any of the above items.

[0017] The technical solution of the present invention has one or more of the following beneficial technical effects:

[0018] (1) Using only minimal power consumption and computing resources, some key points in the system can be tracked or captured with low latency, which helps the system to make timely feedback or adjustments.

[0019] (2) For the field of alcohol precipitation of traditional Chinese medicine, a new supernatant extraction action control solution is provided to prevent drug residues from being extracted in time.

[0020] In addition, other beneficial effects of the present invention will be mentioned in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a compressed frame graph of the event stream of the event camera;

[0022] Figure 2 It is a compressed frame image after preprocessing and noise reduction;

[0023] Figure 3 It is a schematic diagram after event clustering;

[0024] Figure 4 It is a schematic diagram of the locations of clusters generated after event clustering;

[0025] Figure 5 It is a schematic diagram of the fusion between clusters;

[0026] Figure 6 It is a schematic diagram of different interfaces when the present invention is applied to an alcohol precipitation tank. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

[0029] The event camera generates a sequence of pulse events, also called an event stream, based on the light changes in the field of view. An event stream consists of events one after another. The event is represented by a quaternion (x, y, p, t), where x and y are the pixel coordinates of the event, p represents the polarity, which can usually be 1 or -1 to indicate an increase or decrease in brightness, and t is the timestamp of the event.

[0030] Figure 1 It is a compressed frame graph of the event stream of the event camera. By accumulating the event stream over a period of time, that is, the frame compression operation, we can obtain Figure 1 The diagram shows the event camera data in the form of a traditional frame image. In addition to the event stream generated by the change of light in the field of view, event cameras usually also generate random noise events due to various leakages in the circuit.

[0031] For example, by scanning the laser up and down in the alcohol precipitation tank, the generated light spot will present different states at key positions, such as the light spot splitting or merging. Based on this, the supernatant surface and / or the interface between the supernatant surface and the precipitate are determined by identifying the key points. The precipitate is usually precipitated by drug residues.

[0032] First, an event window is set, which can include an integer number of events. For example, the events generated by the event camera within 1 millisecond or 10 milliseconds can be defined as an event window. Here, it is assumed that an event window includes N events, and is recorded as an event set E={ev1, ev2, …, ev i , ……, ev N}, where N is a positive integer and i is a positive integer not greater than N.

[0033] Preferably, the events within the event window are denoised. i =(x i ,y i , p i , t i ), take the pixel or pixel coordinates (x i , y i )The coordinate index set O in the surrounding m×m area i . For example, m=2.

[0034] In the event set E, count the number of pixels that meet the following conditions: the pixel coordinates belong to the coordinate index set O i , and the timestamp of the event is in any event ev i The timestamp of the preset time interval (t i , t i +δ), where δ is a time parameter and can be positive or negative.

[0035] Alternatively, it is also possible not to count the number of pixels satisfying the above conditions in the event set E, but to globally count the number of pixels satisfying the conditions.

[0036] If the number of pixels that meet the above conditions exceeds the threshold N b , then the event ev i In other words, for any event in the event window, if the number of pixels in a preset area around the pixel generating the event reaches a threshold value within the preset time interval where the timestamp of the event is located, the event is determined to be a valid event, otherwise it is a noise event.

[0037] After filtering out all noise events in the event set E, a noise-reduced event set E' is obtained. Figure 2 It is a compressed frame image after preprocessing and noise reduction. Figure 1 , the number of noise events will be significantly reduced.

[0038] Then, a clustering operation is performed on the events in the event set E'. Figure 3 This is a schematic diagram after event clustering. Take any event ev in the event set E' i , determine the event ev i Whether the number of all events in the ε neighborhood exceeds the threshold N c If so, then the event ev i is the core event and will be the event i The corresponding cluster is marked as k, that is, event ev i Classify into cluster c k Otherwise, the event i Marked as an outlier event. The first cluster has a label k=1, the second cluster has a label k=2, and so on; event ev i The event in the ε neighborhood is the distance from the pixel coordinate (x i , y i ), where ε is a positive number and k is the label of the cluster.

[0039] If the event ev i It is judged as a core event, and the event ev i Events ev in the ε neighborhood of j The mark indicates that it is an outlier event or an event that has not been traversed, then the event ev j Also marked as cluster k, and if the event ev j It is also a core event, then the event ev i The same operation applies to the event ev j .

[0040] In this way, we can find all events in cluster k, and then start searching for all events in cluster k+1 until all events in the event set E' are traversed. Thus, we get the cluster set C={c1, c2, …, c k , ……,c n}, where c k The cluster is composed of all events marked with cluster k, n is a positive integer, and k is a positive integer not greater than n.

[0041] For any cluster c in the cluster set C k , according to the timestamp of the event, a corresponding cluster timestamp can be generated to record the generation time of the cluster. And, for any cluster c k, according to the pixel coordinates of the events therein, a cluster coordinate corresponding to the cluster center can be generated, such as the center coordinates of the events in the cluster.

[0042] Figure 4 It is a schematic diagram of the location of the clusters generated after event clustering. By recording the historical information about the cluster set C and tracking or assigning cluster identifiers (IDs) based on at least the currently identified cluster coordinate information, the movement trend of the cluster is identified.

[0043] Furthermore, after sliding to a new event window, the cluster corresponding to the specific cluster identifier updates the cluster coordinates and / or the cluster timestamp.

[0044] If the cluster corresponding to the specific cluster identifier still does not appear after a preset time, the information related to the cluster corresponding to the specific cluster identifier is removed.

[0045] Figure 5 For clusters that meet the preset motion trend, for example, the laser scanning direction determines whether the spot moves up or down. If there are two different clusters c at the same time or in the same time period, p and cluster c q , the movement trend indicates that the two are approaching, and cluster c p The cluster coordinates and cluster c q The cluster coordinates of x are within the preset first boundary area. For example, x p <x b , x q <x b ,y p <y b ,y q <y b , then cluster c p and cluster c q Enter the pre-fusion state. Where p and q are both positive integers not greater than n, (x p , y p ) is cluster c p The cluster coordinates, (x q , y q ) is cluster c q The cluster coordinates, x b and b It is the preset first boundary area coordinate.

[0046] Furthermore, the preset movement trend mentioned here refers to a pair of clusters approaching each other.

[0047] For a pair of clusters entering the pre-fusion state, cluster c p and cluster c q , in the continuous w event windows, the radius of the ε neighborhood in the clustering operation is reduced exponentially to detect cluster cp and cluster c q , where w is a positive integer greater than 1. For example, the neighborhood radius ε applied to the i-th event window in the w consecutive event windows is i =ε×exp(W i / W), where the length of the event window W<1.

[0048] If within a continuous w event window, cluster c p and cluster c q The movement trends of all clusters are consistent with the preset movement trends, such as approaching and detecting cluster c p The cluster coordinates and cluster c q The distance between the cluster coordinates of is less than the preset threshold, or cluster c p and cluster c q One of them disappears (in the process of tracking and identifying cluster identifiers, cluster c p and cluster c q merge), then cluster c is determined p and cluster c q Entering the cluster fusion state, at this time it corresponds to a key point in the entire system, and records the cluster c when entering the cluster fusion state p or cluster c q The cluster coordinates (x m ,y m ) or / and cluster timestamp t m , as the associated information of the key points.

[0049] Furthermore, it is possible to provide a cluster c that meets the preset motion trend. p and cluster c q A stack is set, and associated information about the pair of clusters, such as cluster coordinates and cluster timestamps, is stored in the stack for subsequent cluster fusion status checking.

[0050] For cluster c p and cluster c q When any of the clusters does not conform to the preset movement trend (such as cluster identifier change, movement direction change, cluster disappearance, etc.), cluster c p and cluster c q The associated information is removed from the stack, and the cluster pairs entering the pre-fusion state are re-detected.

[0051] On the other hand, if two clusters c are detected based on the cluster coordinate information p , cluster c q The distances between the cluster coordinates are within the preset range, and cluster c is detected based on multiple consecutive event windows. p , cluster c q If it is moving in the opposite direction and meets the preset movement trend, it is determined to enter the cluster split state, corresponding to another key point, and the cluster c when the split is first detected is recorded.p or cluster c q The cluster coordinates are (x d , y d ) or / and cluster timestamp t d , as the associated information of the key points.

[0052] Furthermore, the preset movement trend mentioned here refers to that a pair of clusters are moving away from each other.

[0053] Furthermore, the cluster splitting state is determined based on the cluster coordinates (x d , y d ) is within the preset second boundary area.

[0054] Figure 6 It is a schematic diagram of different interfaces when the present invention is applied to an alcohol precipitation tank. For example, taking the detection of the liquid level in a traditional Chinese medicine alcohol precipitation tank as an example, through the means of detecting key points disclosed in the present invention, in the alcohol precipitation tank, the height information related to the supernatant liquid surface, or / and the height information related to the interface between the supernatant and the precipitate are judged according to the associated information of the key points, thereby controlling the supernatant extraction action.

[0055] It is worth mentioning that the spot-based key point detection method of the present invention can be applied to alcohol precipitation tanks, and can also be applied to any other application field where key points are generated when laser spots are fused or split. The present invention is not limited to the alcohol precipitation tank scenario.

[0056] On the other hand, the present invention also discloses a key point detection device based on light spots, which includes a laser and an event camera; wherein the laser is used to generate laser light; the light spot generated by the laser irradiation is at least partially set within the field of view of the event camera; the key point detection device based on light spots obtains the associated information of the key points according to the key point detection method based on light spots described above.

[0057] In order to better illustrate the present invention, numerous specific details are provided in the above specific embodiments. It should be understood by those skilled in the art that the present invention can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present invention.

[0058] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A key point detection method based on light spots, characterized in that: include: The light spot generated by the laser illumination is at least partially disposed within the field of view of the event camera; Obtain an event stream from an event camera, perform clustering operations on all events in the event window to obtain several clusters, and obtain cluster coordinates and cluster timestamps; At least determine the movement trend of a pair of clusters based on the cluster coordinates; If the motion trend conforms to the preset motion trend, and the distance between the cluster coordinates of the pair of clusters conforms to the preset range, the pair of clusters is determined to enter a cluster fusion state or a cluster split state, and the cluster coordinates and / or cluster timestamps are recorded as association information of a key point; wherein the cluster timestamps of the pair of clusters are the same or within the same time period.

2. The key point detection method based on light spot according to claim 1, characterized in that: Before performing a clustering operation on all events in the event window, a noise reduction operation is first performed on the events in the event window to filter out noise events.

3. The key point detection method based on light spot according to claim 2 is characterized in that: For any event in the event window, if the number of pixels in a preset area around the pixel generating the event reaches a threshold within the preset time interval where the timestamp of the event is located, the event is determined to be a valid event, otherwise it is a noise event.

4. The key point detection method based on light spot according to claim 1 or 3, characterized in that: The preset movement trend refers to that the pair of clusters are approaching or moving away.

5. The key point detection method based on light spots according to claim 4 is characterized in that: Whether the pair of clusters enter into a cluster fusion state or a cluster split state is determined based on whether the cluster coordinates of the pair of clusters are within a preset boundary area.

6. The key point detection method based on light spots according to claim 5 is characterized in that: In a number of consecutive event windows, the radius of the neighborhood in the clustering operation is reduced in an exponential trend, and the movement trend of the pair of clusters is detected; If it is determined that the pair of clusters conform to the approaching preset movement trend in the continuous event windows, and the distance between the cluster coordinates of the pair of clusters is within a preset range, it is determined that the pair of clusters enter a cluster fusion state.

7. The method for detecting key points based on light spots according to claim 6, characterized in that: For the pair of clusters that conform to the preset motion trend, a stack is set, and association information about the pair of clusters is stored in the stack.

8. The key point detection method based on light spots according to claim 7 is characterized in that: For the pair of clusters in the stack that no longer conform to the preset motion trend, the associated information about the pair of clusters is removed from the stack.

9. The method for detecting key points based on light spots according to claim 1, 2 or 6, characterized in that: In the alcohol precipitation tank, the height information related to the supernatant liquid surface and / or the height information related to the interface between the supernatant liquid and the sediment is determined according to the correlation information of the key points, thereby controlling the supernatant liquid extraction action.

10. A key point detection device based on light spots, characterized in that: The key point detection device based on light spots includes a laser and an event camera; wherein, The laser is used to generate laser light; The light spot produced by the laser illumination is at least partially disposed within the field of view of the event camera; The light spot-based key point detection device obtains the associated information of the key points according to the light spot-based key point detection method according to any one of claims 1 to 8.

Images