Information processing apparatus and information processing system

Abstract

The present technology relates to an information processing apparatus and an information processing system capable of accurately detecting information of a liquid droplet from a dispenser. The information processing apparatus includes: an event sensor including a pixel configured to photoelectrically convert a light signal and output a pixel signal, the event sensor outputting a temporal luminance change of the light signal as an event signal based on the pixel signal; and a processor detecting a liquid droplet ejected from the dispenser based on the event signal. For example, the present technology can be applied to a dispenser control system that controls a dispenser, and the like.

Description

TECHNICAL FIELD

[0001] The present technology relates to an information processing apparatus and an information processing system, and more particularly, to an information processing apparatus and an information processing system capable of accurately detecting a liquid droplet from a dispenser. BACKGROUND

[0002] In a semiconductor manufacturing process, there is a process of applying a liquid such as an adhesive to a substrate, a lead frame, or the like by using a dispenser. A system for measuring a drop amount has been proposed in which a volume of a liquid droplet ejected from a dispenser is measured by a camera, and feedback control of parameters is performed on the dispenser to adjust the amount of the liquid droplet (see, for example, Patent Literature 1).

[0003] LIST OF CITATIONS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2006-195402 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] In a case where a high-speed camera with a frame rate of about 1000 fps is used as a camera for capturing an image of a liquid droplet ejected from a dispenser, it is difficult to accurately capture the liquid droplet since only about three images can be captured for one liquid droplet.

[0008] The present technology is made in view of such a situation, and an object thereof is to be able to accurately detect a liquid droplet from a dispenser.

[0009] SOLUTION TO PROBLEM

[0010] An information processing apparatus according to a first aspect of the present technology includes an event sensor including a pixel configured to photoelectrically convert a light signal and output a pixel signal, the event sensor being configured to output, as an event signal, a temporal luminance change of the light signal based on the pixel signal; and a processor configured to detect a liquid droplet injected from a dispenser based on the event signal.

[0011] An information processing system according to a second aspect of the present technology includes a dispenser configured to inject a predetermined liquid; an event sensor including a pixel configured to photoelectrically convert a light signal and output a pixel signal, the event sensor being configured to output, as an event signal, a temporal luminance change of the light signal based on the pixel signal; and a processor configured to detect a liquid droplet injected from the dispenser based on the event signal.

[0012] In the first and second aspects of the present technology, a pixel configured to photoelectrically convert a light signal and output a pixel signal is provided to an event sensor, a time luminance variation of the light signal is output as an event signal based on the pixel signal, and a droplet injected from a dispenser is detected based on the event signal.

[0013] The information processing apparatus, the imaging apparatus, and the control system can be independent apparatuses or modules incorporated in other apparatuses. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a diagram illustrating a configuration example of a dispenser control system to which the present technology is applied.

[0015] Figure 2 is a diagram for explaining droplet control of the dispenser control system. Figure 1

[0016] Figure 3 is a diagram illustrating an example of event data output by an EVS camera.

[0017] Figure 4 is a diagram for explaining a method of generating frame data of event data.

[0018] Figure 5 is a diagram illustrating an example of an event image generated based on event data.

[0019] Figure 6 is a diagram illustrating a first configuration example of an illumination device.

[0020] Figure 7 is a diagram illustrating a second configuration example of an illumination device.

[0021] Figure 8 is a diagram illustrating an image capturing direction of an EVS camera with respect to a moving direction of a droplet.

[0022] Figure 9 is a diagram for explaining an image capturing method in a case where a volume of a droplet is measured.

[0023] Figure 10 is a block diagram illustrating a functional configuration example of a control device.

[0024] Figure 11 is a diagram for explaining generation of an event image and a display image by a first frame processing unit.

[0025] Figure 12 is a diagram for explaining generation of an event image and a display image by a first frame processing unit.

[0026] Figure 13 ​is a diagram for explaining generation of an event image and a display image by a first frame processing unit.

[0027] Figure 14 is a flowchart showing a first framing process of the first frame processing unit.

[0028] Figure 15 is a diagram for explaining generation of a reconstruction image by a second frame processing unit.

[0029] Figure 16 is a diagram for explaining generation of a reconstruction image by a second frame processing unit.

[0030] Figure 17 is a flowchart showing a second framing process of the second frame processing unit.

[0031] Figure 18 is a diagram showing an example of a reconstruction image after a certain period of time has passed.

[0032] Figure 19 is a diagram for explaining an example of a noise removal process in the second framing process.

[0033] Figure 20 is a diagram for explaining a noise removal process of a noise removal processing unit.

[0034] Figure 21 is a diagram for explaining a droplet detection process of a droplet detection unit.

[0035] Figure 22 is a diagram for explaining a droplet detection process of a droplet detection unit.

[0036] Figure 23 is a diagram for explaining a droplet detection process of a droplet detection unit.

[0037] Figure 24 is a diagram for explaining a method of calculating a size of a droplet.

[0038] Figure 25 is a flowchart for explaining a droplet detection process of a droplet detection unit.

[0039] Figure 26 is a diagram for explaining a search process of a droplet tracking unit.

[0040] Figure 27 is a diagram for explaining a search process of a droplet tracking unit.

[0041] Figure 28 is a flowchart for explaining a droplet tracking process of a droplet tracking unit.

[0042] Figure 29is a flowchart for explaining a droplet control process of the dispenser control system.

[0043] Figure 30 is a diagram for explaining a DNN.

[0044] Figure 31 is a flowchart for explaining a droplet control process using a DNN.

[0045] Figure 32 is a flowchart for explaining a droplet control process using a DNN.

[0046] Figure 33 A configuration example of a second embodiment of the dispenser control system to which the present technology is applied is shown.

[0047] Figure 34 is a diagram showing another arrangement example of the EVS camera and the RGB camera in the second embodiment.

[0048] Figure 35 A configuration example of a third embodiment of the dispenser control system to which the present technology is applied is shown.

[0049] Figure 36 is a block diagram showing a configuration example of the EVS camera.

[0050] Figure 37 is a perspective view showing a schematic configuration example of an imaging element.

[0051] Figure 38 is a plan view showing a configuration example of a light-receiving chip.

[0052] Figure 39 is a plan view showing a configuration example of a detection chip.

[0053] Figure 40 is a plan view showing details of an address event detection unit.

[0054] Figure 41 is a block diagram showing a configuration example of an address event detection circuit.

[0055] Figure 42 is a diagram showing a detailed configuration of a current-voltage conversion circuit.

[0056] Figure 43 is a diagram showing a detailed configuration of a subtracter and a quantizer.

[0057] Figure 44 is a diagram showing another arrangement example of the light-receiving chip and the detection chip.

[0058] Figure 45 is a block diagram showing a configuration example of hardware of a computer. DETAILED DESCRIPTION

[0059] Hereinafter, an embodiment for implementing the present technology (hereinafter, referred to as an embodiment) will be described with reference to the drawings. Note that, in the present specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant explanation is omitted. The description will be given in the following order.

[0060] 1. First embodiment of dispenser control system

[0061] 2. Examples of event data

[0062] 3. Method of capturing image of liquid droplet

[0063] 4. Configuration example of control device

[0064] 5. Processing of first frame processing unit

[0065] 6. Processing of second frame processing unit

[0066] 7. Processing of noise removal processing unit

[0067] 8. Processing of liquid droplet detection unit

[0068] 9. Processing of liquid droplet tracking unit

[0069] 10. Liquid droplet control processing of dispenser control system

[0070] 11. Application of DNN

[0071] 12. Second embodiment of dispenser control system

[0072] 13. Third embodiment of dispenser control system

[0073] 14. Conclusion

[0074] 15. Configuration example of computer

[0075] <1. First embodiment of dispenser control system>

[0076] Figure 1 A configuration example of the first embodiment of the dispenser control system to which the present technology is applied is shown.

[0077] Figure 1 The dispenser control system 1 of the present embodiment includes a dispenser 11, an EVS camera 12, a control device 13, and a display 14, and controls ejection of liquid droplets by the dispenser 11.

[0078] The dispenser 11 ejects a predetermined liquid as a target onto the substrate 21 placed and conveyed on the conveyer 22. The liquid ejected from the dispenser 11 becomes a droplet 10 and drops toward the substrate 21.

[0079] The EVS camera 12 is a camera including a pixel that photoelectrically converts a light signal and outputs a pixel signal and an event sensor that outputs a time luminance change of the light signal as an event signal (event data) based on the pixel signal. Such an event sensor is also called an event-based vision sensor (EVS). When a camera including a normal image sensor captures an image in synchronization with a vertical synchronization signal and outputs frame data of the image data as one frame (picture) in a period of the vertical synchronization signal, the EVS camera 12 outputs the event data only at a time of an event occurrence. Therefore, it can be said that the EVS camera 12 is an asynchronous (or address control) camera.

[0080] The EVS camera 12 detects a time luminance change as an event based on the droplet 10 ejected from the dispenser 11 and outputs the event data to the control device 13.

[0081] The control device 13 detects the droplet 10 from the dispenser 11 based on the event data output from the EVS camera 12, generates control information for controlling ejection of the droplet 10, and outputs the control information to the dispenser 11. Further, the control device 13 generates a display image to be monitored by a worker based on the event data output from the EVS camera 12 and causes the display 14 to display the display image.

[0082] Figure 2 is a diagram for explaining droplet control by the dispenser control system 1.

[0083] In a case where the timing at which the dispenser 11 ejects the droplet 10, the speed of the droplet 10, and the like are not appropriate, the droplet 10 cannot be accurately applied to the substrate 21 flowing on the conveyer 22, as shown on the left side of Figure 2 The control device 13 calculates trajectory information of the droplet 10 including at least one of a position, a speed, a moving direction, and the like of the droplet 10 based on the event data output from the EVS camera 12. Then, based on the calculated trajectory information of the droplet 10, the control device 13 feeds back parameters for controlling ejection timing, ejection strength, ejection direction, and the like of the droplet 10 to the dispenser 11 as feedback control information. Further, the control device 13 can also calculate a size, a volume, and the like of the droplet 10 based on the event data and can also control the amount and the viscosity of the droplet 10. Therefore, as Figure 2As shown on the right, a droplet 10 of appropriate timing and amount can be applied to the substrate 21. It is possible to determine which parameters are provided to the dispenser 11 as feedback control information. Furthermore, scattering by conductive objects can cause short circuits and lead to defective products, making it possible to detect droplets, such as accompanying droplets, in addition to the main body of the droplet 10.

[0084] <2. Examples of Event Data>

[0085] Figure 3 An example of event data output by EVS camera 12 is shown.

[0086] For example, such as Figure 3 As shown, the EVS camera 12 output includes the time t when the event occurs. i The coordinates (x, y) of the pixel where the event occurred. i y i The polarity of the brightness change p) i The event data is used as an event.

[0087] The time t of the event i It is a timestamp indicating the time when an event occurred, and is represented by, for example, the count value of a counter based on a predetermined clock signal in a sensor. It can be considered that the timestamp corresponding to the time when an event has occurred is time information indicating the (relative) time that the event has occurred, as long as the interval between events is maintained when the event occurs.

[0088] polarity p i This indicates the direction of the brightness change when a brightness change (light intensity change) exceeding a predetermined threshold occurs as an event, and indicates whether the brightness change is in a positive direction (hereinafter also referred to as positive) or a negative direction (hereinafter also referred to as negative). The polarity of the event p i For example, it is represented as "1" in the positive case and as "0" in the negative case.

[0089] exist Figure 3 In the event data, at time t of a certain event i The time t of the event adjacent to this event i+1 The interval between them does not need to be constant. That is, the time t of the event. i and time t i+1 These can be the same time or different times. However, assuming the event's time t... i and t i+1 There exists a given expression t i ≤t i+1 The relationship indicated.

[0090] Unlike image data (frame data) of a frame format that is output in a frame period synchronized with a vertical synchronization signal, event data is output each time an event occurs. Therefore, like the event data, it cannot be displayed as an image by a display that displays an image corresponding to the frame data, such as a projector, and cannot be input to an identifier (classifier) for image processing. The event data needs to be converted into frame data.

[0091] Figure 4 is a diagram for explaining an example of a method for generating frame data from event data.

[0092] In Figure 4 , in a three-dimensional (temporal) space including an x-axis, a y-axis, and a time axis t, a time t of an event included in the event data is plotted as a point of the event data, and a coordinate (x, y) of a pixel of the event.

[0093] That is, assuming that a position (x, y, t) in a three-dimensional space represented by a time t of an event included in the event data and a pixel (x, y) of the event is referred to as a spatiotemporal position of the event, in Figure 4 , the event data is plotted as a point at the spatiotemporal position of the event (x, y, t).

[0094] By using the event data output from the EVS camera 12 as pixel values, an event image can be generated for each predetermined frame interval using the event data within a predetermined frame width from the start of the predetermined frame interval.

[0095] The frame width and the frame interval can be specified by time or by the number of pieces of event data. One of the frame width and the frame interval can be specified by time, and the other can be specified by the number of pieces of event data.

[0096] Here, in a case where the frame width and the frame interval are specified by time and are the same, the frame volumes are in a state of being in contact with each other without a gap. Further, in a case where the frame interval is greater than the frame width, the frame volumes are in a state of being arranged with a gap. In a case where the frame width is greater than the frame interval, the frame volumes are in a state of being arranged in a partially overlapping manner.

[0097] For example, by setting a pixel at a position (x, y) of an event in a frame to white and setting pixels at other positions in the frame to a predetermined color such as gray, generation of the event image can be performed.

[0098] Further, in a case where a polarity of an amount of light is distinguished as an event for the event data, generation of the frame data can be performed by, for example, setting a pixel to white in a case where the polarity is positive, setting a pixel to black in a case where the polarity is negative, and setting pixels at other positions of the frame to a predetermined color such as gray.

[0099] Figure 5 An example of an event image generated based on event data is shown.

[0100] Figure 5 Image 31 is an image obtained by capturing the detected target by the EVS camera 12 using an image sensor that outputs an RGB image. In image 31, a scene of a person walking in front of a bookshelf serves as the background. Event image 32 is obtained when the captured scene is detected by the EVS camera 12 and frame data is generated based on the output event data. Event image 32 is generated by setting white to pixels with positive polarity, setting black to pixels with negative polarity, and setting gray to pixels in other positions within the frame.

[0101] <3. Droplet Image Acquisition Methods>

[0102] Figure 6 and Figure 7 An example of the arrangement relationship between the EVS camera 12 and the lighting device is shown.

[0103] The lighting device 61 is positioned at a location where the background of the droplet 10 becomes uniform and produces a contrast with the droplet 10, and illuminates the droplet 10 with light.

[0104] Figure 6 This is a diagram showing a first configuration example of the lighting device 61.

[0105] In the first configuration example, such as Figure 6 As shown in Figure A, the EVS camera 12 and the illumination device 61 are arranged facing each other to sandwich the droplet 10 in between. In other words, the illumination device 61 illuminates the droplet 10 from behind, and the EVS camera 12 detects changes in the brightness of the droplet 10 illuminated from behind. A diffuser 62 is disposed in front of the illumination device 61. When the image of the droplet 10 is captured in this configuration, the droplet 10 is photographed as shown... Figure 6 The black outline shown in B.

[0106] Figure 7 This is a diagram showing a second configuration example of the lighting device 61.

[0107] In the second configuration example, such as Figure 7 As shown in Figure A, the EVS camera 12 and the illumination device 61 are arranged in the same direction relative to the droplet 10. A black anti-reflective plate 63 is arranged on the background side of the droplet 10 to prevent light reflection. When capturing images of the droplet 10 in this configuration, as shown in Figure A... Figure 7 As shown in B, droplet 10 is photographed as a white outline.

[0108] In this embodiment, the following is adopted: Figure 7The second arrangement example shown is an arrangement of the illumination device 61, and the EVS camera 12 captures an image of the droplet 10 from the same direction as the illumination device 61.

[0109] Figure 8 is a diagram showing an image capturing direction of the EVS camera 12 with respect to a moving direction of the droplet 10.

[0110] The EVS camera 12 includes a light receiving portion 51 in which pixels for detecting a luminance change are arranged in a matrix two-dimensionally. It is assumed that Figure 8 the vertical direction of the light receiving portion 51 in is the x-axis direction and Figure 8 the horizontal direction in is the y-axis direction, then when an event has occurred, the EVS camera 12 performs a reading operation of reading signals in column units in the x-axis direction of the light receiving portion 51. In this case, as Figure 8 shown, the direction of the EVS camera 12 is arranged so that the reading direction of the pixel signals coincides with the moving direction of the droplet 10. As a result, for example, in a case where a plurality of droplets 10 pass through the light receiving portion 51 of the EVS camera 12, the plurality of droplets 10 are arranged along the reading direction of the pixel signals. Therefore, it is possible to acquire signals of the plurality of droplets 10 simultaneously by reading the pixel signals of one column.

[0111] Further, the EVS camera 12 can set a predetermined region of interest 52 with respect to the entire region of the light receiving portion 51, and read only the signals of the set region of interest 52. Due to the arrangement in which the reading direction of the pixels coincides with the moving direction of the droplet 10, it is possible to reduce the number of columns (the number of pixels in the y-axis direction) in which signals are read, and to improve the reading speed (detection speed).

[0112] Figure 9 is a diagram for explaining an image capturing method in a case where the volume of the droplet 10 is measured.

[0113] In a case where the volume of the droplet 10 is measured, it is necessary to measure the area of the droplet 10 on a plane perpendicular to the falling direction of the droplet 10. Therefore, the EVS camera 12 detects the droplet 10 from two orthogonal directions by any of the methods of Figure 9 A or B. Figure 9 In , the direction perpendicular to the page indicates the falling direction of the droplet 10.

[0114] Figure 9 A of

[0115] The first image capturing method is a method in which the prism 41 is disposed in front of the EVS camera 12, and one EVS camera 12 detects the droplet 10 in two directions, a first direction in which the droplet 10 is directly observed and a second direction perpendicular to the first direction, simultaneously via the prism 41.

[0116] Figure 9 The B is an example of the second image capturing method of the droplet 10 in the case where the volume is measured.

[0117] The second image capturing method is a method in which two EVS cameras 12A and 12B are arranged in orthogonal directions, and each of the two EVS cameras 12A and 12B captures an image from one direction to detect the droplet 10 in two orthogonal directions. The image capturing is performed in a state where the time stamps of the EVS cameras 12A and 12B are synchronized with each other.

[0118] In the droplet control process described later, the calculation of the volume is omitted while the lateral width of the droplet 10 is calculated. However, in the case where the volume of the droplet 10 is calculated, the volume of the droplet 10 is calculated from the area of the droplet 10 calculated from the image capturing result obtained by the above-described first or second image capturing method and the length of the droplet 10 in the moving direction.

[0119] <4. Configuration example of control device>

[0120] Figure 10 is a block diagram showing a functional configuration example of the control device 13.

[0121] The control device 13 includes a preprocessing unit 101, an image output unit 102, a droplet detection unit 103, a droplet tracking unit 104, and a parameter determination unit 105.

[0122] As a preprocessing for detecting the droplet 10, the preprocessing unit 101 generates three types of images, i.e., an event image, a reconstructed image, and a display image, based on event data output from the EVS camera 12.

[0123] The preprocessing unit 101 includes a framing processing unit 111 including a first frame processing unit 121 and a second frame processing unit 122, and a noise removal unit 112.

[0124] The first frame processing unit 121 generates an event image based on event data from the EVS camera 12. The first frame processing unit 121 generates a positive event image based on positive events and a negative event image based on negative events as the event image. Further, the first frame processing unit 121 also generates a display image based on the positive event image and the negative event image.

[0125] The second frame processing unit 122 estimates luminance values based on the event data from the EVS camera 12 to generate a reconstructed image.

[0126] The framing processing unit 111 supplies the generated event image and the reconstructed image to the de-noising processing unit 112, and supplies the display image to the image output unit 102. As will be described later with reference to Figure 13 and Figure 18 The event image and the reconstructed image are formed of binary images of white and black, and the display image is formed of ternary images of white, black, and gray.

[0127] The de-noising processing unit 112 performs de-noising processing that removes noise on each of the event image and the reconstructed image that are binary images. Although specific processing will be described later with reference to Figure 20 , the de-noising processing unit 112 performs filtering processing that uses expansion processing and contraction processing as de-noising processing on white pixels. The event image and the reconstructed image after the noise removal processing are supplied to the droplet detection unit 103 and the droplet tracking unit 104.

[0128] The image output unit 102 supplies the display image supplied from the framing processing unit 111 to the display 14.

[0129] The droplet detection unit 103 detects the droplet 10 from each of the event image and the reconstructed image supplied from the pre-processing unit 101. The droplet detection unit 103 supplies information on the droplet 10 detected from the event image as information on a tracking target to the droplet tracking unit 104. Further, the droplet detection unit 103 calculates the size of the droplet 10 from the droplet 10 detected from the reconstructed image, and supplies the size to the parameter determination unit 105.

[0130] The droplet tracking unit 104 tracks the droplet 10 detected from the event image in the droplet detection unit 103, calculates trajectory information of the droplet 10 (including at least one of the position, the speed, the moving direction, and the like of the droplet 10), and supplies the trajectory information to the parameter determination unit 105.

[0131] The parameter determination unit 105 determines whether or not the parameters of the dispenser 11 for controlling the ejection timing, the ejection direction, and the like are within a normal range based on the trajectory information of the droplet 10 supplied from the droplet tracking unit 104. Further, the parameter determination unit 105 determines whether or not the ejection amount each time is within a normal range based on the size (width) and the volume of the droplet 10 supplied from the droplet detection unit 103. In a case where it is determined that the parameters are not within the normal range, the parameter determination unit 105 generates control information for correcting the parameters as feedback control information, and outputs the control information to the dispenser 11.

[0132] The processing performed by each unit of the control device 13 will be described in more detail below.

[0133] <5. Processing by the first frame processing unit>

[0134] First, referring to Figures 11 to 14 , the generation of event images and display images by the first frame processing unit 121 will be described.

[0135] The first frame processing unit 121 generates a positive event image based on positive events and a negative event image based on negative events based on event data from the EVS camera 12.

[0136] Figure 11 is a diagram showing units that generate each event image in a case where the first frame processing unit 121 generates event images.

[0137] The first frame processing unit 121 starts detection of events (imaging of an image of the liquid droplet 10) at time TO, and generates an event image at time T i The i-th frame is framed, that is, an event image is generated, at time T i-1 + i * Δt. The time Δt from time T i corresponds to the frame interval described in Figure 4 , and corresponds to the inverse of the frame rate FPS, which is the number of event images to be taken per second. The time Δt is a period of one frame corresponding to the frame rate, and is also referred to as a one-frame period Δt.

[0138] The first frame processing unit 121 generates, as the i-th frame event image at time T i where framing of the i-th frame is performed, an image in which event data from the EVS camera 12 is collected for each predetermined integration time h. More specifically, the first frame processing unit 121 generates the i-th frame event image based on event data for which the time t is from time (T i - h) to time T i .

[0139] Here, the predetermined integration time h is shorter than the one-frame period Δt. When the predetermined integration time h is changed, the shape of the liquid droplet 10 in the event image changes, as shown in Figure 12 . By setting the integration time h separately from the one-frame period Δt (h = Δt is not fixed) without fixing the integration time h to the one-frame period Δt, it is possible to separately set the frequency of framing (frame rate) and the shape of the liquid droplet 10 in the event image.

[0140] from time (T i - h) to time T iThe first frame processing unit 121 generates a binary image in which the pixel value of the pixel (x, y) in which the positive event is detected is set to 1 (white) and the pixel value of the other pixels is set to 0 (black) in the predetermined integration time h from the time (T

[0141] Further, in the predetermined integration time h from the time (T i -h) to the time T i , the first frame processing unit 121 generates a binary image in which the pixel value of the pixel (x, y) in which the negative event is detected is set to 1 (white) and the pixel value of the other pixels is set to 0 (black) and sets it as the negative event image of the i-th frame.

[0142] Further, the first frame processing unit 121 generates a ternary image in which the pixel value of 1 (white) in the positive event image of the i-th frame is set to 255 (white), the pixel value of 1 (white) in the negative event image of the i-th frame is set to 0 (black), and the pixel value of the other pixels is set to 128 (gray), as the display image of the i-th frame.

[0143] Figure 13 Examples of the positive event image, the negative event image, and the display image generated by the first frame processing unit 121 are shown.

[0144] With reference to the flowchart in Figure 14 , the first framing processing performed by the first frame processing unit 121 will be described. This processing starts, for example, at the same time as the start of image capturing by the EVS camera 12.

[0145] The first frame processing unit 121 performs the first framing processing of Figure 14 at the same time as acquiring the event data supplied from the EVS camera 12 at the timing of detecting the event. The time information (time t) is synchronized between the EVS camera 12 and the control device 13.

[0146] First, in step S11, the first frame processing unit 121 sets a variable i for identifying the number of frames to 1.

[0147] In step S12, the first frame processing unit 121 determines whether the time t is greater than the time (T i -h) and equal to or less than the time T i , and repeats the determination processing in step S12 until it is determined that the time t is greater than the time (T i -h) and equal to or less than the time T i .

[0148] Then, when it is determined in step S12 that the time t is greater than the time (T i -h) and equal to or less than the time T iWhen it is determined in step S13 that the polarity p of the event data supplied from the EVS camera 12 is positive, the processing proceeds to step S14, and the first frame processing unit 121 sets the pixel value of the pixel of the positive event image corresponding to the event occurrence position of the event data supplied from the EVS camera 12 to "1".

[0149] When it is determined in step S13 that the polarity p of the event data supplied from the EVS camera 12 is positive, the processing proceeds to step S14, and the first frame processing unit 121 sets the pixel value of the pixel of the positive event image corresponding to the event occurrence position of the event data supplied from the EVS camera 12 to "1".

[0150] When it is determined in step S13 that the polarity p of the event data supplied from the EVS camera 12 is positive, the processing proceeds to step S14, and the first frame processing unit 121 sets the pixel value of the pixel of the positive event image corresponding to the event occurrence position of the event data supplied from the EVS camera 12 to "1".

[0151] After step S14 or S15, the processing proceeds to step S16, and the first frame processing unit 121 determines whether the time t has exceeded the time T i When it is determined in step S16 that the time t has not exceeded the time T i , the processing returns to step S13, and the processing of steps S13 to S16 described above is repeated.

[0152] When it is determined in step S16 that the time t has not exceeded the time T i , the processing proceeds to step S17, and the first frame processing unit 121 generates the positive event image, the negative event image, and the display image of the i-th frame. More specifically, the first frame processing unit 121 generates a binary image in which the pixels whose pixel values are set to "1" in the above step S14 are set to white and the other pixels are set to black as the positive event image. Further, the first frame processing unit 121 generates a binary image in which the pixels whose pixel values are set to "1" in the above step S15 are set to white and the other pixels are set to black as the negative event image. Further, the first frame processing unit 121 generates a ternary image in which the pixel values of the pixels whose pixel values are set to "1" in the positive event image are set to 255 (white), the pixel values of the pixels whose pixel values are set to "1" in the negative event image are 0 (black), and the pixel values of the other pixels are set to 128 (gray) as the display image.

[0153] Next, in step S18, the first frame processing unit 121 increases the variable i for identifying the number of frames by 1, and then returns the processing to step S12. Thereafter, the processing in steps S12 to S18 described above is repeatedly executed, and when an instruction is given to end the operation of the entire distributor control system 1 or the control device 13, Figure 14 , the first framing processing ends.

[0154] As described above, according to the first framing process, a positive event image and a negative event image are generated by setting predetermined pixel values ​​based on the location where the event occurs.

[0155] <6. Processing of the Second Frame Processing Unit>

[0156] Next, refer to Figures 15 to 18 The image will be described as being reconstructed by the second frame processing unit 122.

[0157] The second frame processing unit 122 estimates the brightness value based on event data from the EVS camera 12 to generate a reconstructed image.

[0158] like Figure 13 As shown, in the positive and negative event images, due to the brightness variation relative to the background, the front and rear portions corresponding to the movement direction of the droplet 10 are extracted, making the shape and size of the entire droplet 10 unknown. Therefore, the second frame processing unit 122 generates a reconstructed image by simply recovering the brightness values ​​from the provided event data, so that the outline of the entire droplet 10 can be identified.

[0159] For example, such as Figure 15 As shown in the graph on the left, suppose that in a predetermined pixel (x, y), a positive event ("1") occurs from time t1 to time t2, a positive event ("1") occurs from time t2 to time t3, a negative event ("0") occurs from time t4 to time t5, and a negative event ("0") occurs from time t5 to time t6.

[0160] Given that the brightness threshold for a positive event is d1 and the brightness threshold for a negative event is d2, such as Figure 15 As shown in the graph on the right, the second frame processing unit 122 estimates the brightness value of pixel (x, y) by adding d1 based on the positive event from time t1 to time t2, adding d1 based on the positive event from time t2 to time t3, subtracting d2 based on the negative event from time t4 to time t5, and subtracting d2 based on the negative event from time t5 to time t6. In this way, by accumulating brightness based on past events, the brightness value can be easily recovered and a reconstructed image can be generated. Furthermore, the thresholds d1 and d2 can be the same value or different values.

[0161] In this embodiment, due to the use of Figure 7The illustrated illumination arrangement takes an image of the liquid droplet 10, so the background is darkest. When the luminance value of the background is set to 0, the luminance value does not become a negative value less than 0. Therefore, in a case where the luminance value becomes a negative value due to the accumulation of the luminance value described above, the second frame processing unit 122 performs processing of resetting the luminance value to 0.

[0162] Figure 16 is a diagram for explaining a specific process in a case where the second frame processing unit 122 generates a reconstructed image.

[0163] The second frame processing unit 122 generates a reconstructed image for each T i (i.e., for each frame rate period corresponding to a frame rate). For example, in a case where a reconstructed image of the i-th frame is generated, the second frame processing unit 122 generates a reconstructed image in which the luminance value is estimated by accumulating all past events from a time T0 at which the event detection is started to a time T i at which the i-th frame is framed.

[0164] More specifically, for example, as shown in the upper part of Figure 16 , a reconstructed image FR1 of the first frame is generated by accumulating events from the time T0 to the time T1. A reconstructed image FR2 of the second frame is generated by accumulating events from the time T0 to the time T2. A reconstructed image FR3 of the third frame is generated by accumulating events from the time T0 to the time T3.

[0165] However, in actual calculation, as shown in the lower part of Figure 16 , in a case where a reconstructed image FRi of the i-th frame is generated, the reconstructed image FRi is generated by accumulating events in the next frame rate period (from the time T i-1 to the time T i ) to the luminance value of the reconstructed image FR i-1 of the previous frame. i .

[0166] With reference to the flowchart in Figure 17 , the second framing process performed by the second frame processing unit 122 will be described. For example, the process starts at the same time as the start of image taking by the EVS camera 12.

[0167] The second frame processing unit 122 performs the second framing process of Figure 17 at the same time as the detection of the timing of the event while acquiring the event data supplied from the EVS camera 12. The time information (time t) is synchronized between the EVS camera 12 and the control device 13.

[0168] First, in step S41, the second frame processing unit 122 sets a variable i for identifying the number of frames to 1.

[0169] In step S42, the second frame processing unit 122 determines whether the variable i is 2 or more, that is, whether the current frame is the second frame or a subsequent frame.

[0170] When it is determined in step S42 that the current frame is not the second frame or the subsequent frame (that is, the current frame is the first frame), the processing proceeds to step S43, and the second frame processing unit 122 sets the reconstructed image of the first frame in which the pixel values of all the pixels are set to 0.

[0171] However, when it is determined in step S42 that the current frame is the second or subsequent frame, the processing proceeds to step S44, and the second frame processing unit 122 sets the reconstructed image of the i-th frame to the initial value in which the reconstructed image of the previous frame is set to the initial value.

[0172] In step S45, the second frame processing unit 122 determines whether the polarity p of the event data provided from the EVS camera 12 is positive.

[0173] When it is determined in step S45 that the polarity p of the event data provided from the EVS camera 12 is positive, the processing proceeds to step S46, and the second frame processing unit 122 adds di to the pixel value of the pixel of the reconstructed image corresponding to the event occurrence position of the event data provided from the EVS camera 12.

[0174] However, when it is determined in step S45 that the polarity p of the event data provided from the EVS camera 12 is negative, the processing proceeds to step S47, and the second frame processing unit 122 subtracts d2 from the pixel value of the pixel of the reconstructed image corresponding to the event occurrence position of the event data provided from the EVS camera 12.

[0175] After step S46 or S47, the processing proceeds to step S48, and the second frame processing unit 122 determines whether there is a pixel having a negative pixel value, and when it is determined that there is no pixel having a negative pixel value, the processing of the next step S49 is skipped.

[0176] However, when it is determined in step S48 that there is a pixel having a negative pixel value, the processing proceeds to step S49, and the second frame processing unit 122 resets the negative pixel value to 0.

[0177] Next, in step S50, it is determined whether At time corresponding to one frame period has elapsed. When it is determined that At time has not elapsed, the processing returns to step S45, and the processing of steps S45 to S50 described above is repeated.

[0178] On the other hand, in the case where it is determined in step S50 that At time has elapsed, the processing proceeds to step S51, and the second frame processing unit 122 outputs the reconstructed image of the i-th frame.

[0179] Next, in step S52, the second frame processing unit 122 increments the variable i used to identify the number of frames by 1, and then returns the processing to step S42. Thereafter, the processing in steps S42 to S52 is repeated, and the process ends when an instruction is given to terminate the operation of the entire distributor control system 1 or control device 13. Figure 17 The second frame-forming process in the process.

[0180] As described above, according to the second framing process, a reconstructed image with estimated brightness values ​​is generated by accumulating pixel values ​​corresponding to brightness thresholds d1 and d2.

[0181] Note that, as mentioned above, in the generation of the reconstructed image, the time from the start of image capture T0 to the time when the i-th frame is captured is accumulated. i All past events. Therefore, objects that were not present at the start of image capture and are reflected from the middle continue to be retained in the reconstructed image. Noise is also accumulated and continues to be retained.

[0182] Figure 18 Examples are shown of the initial reconstructed image at time T0 close to the start of the image capture and the reconstructed image after a certain period of time.

[0183] In the reconstructed image after a specific time period, droplets 10 and noise are generated and retained in specific locations.

[0184] Since the droplet 10 to be detected is a moving body, it is desirable to remove objects that are stationary for a certain period of time.

[0185] Therefore, as part of the second framing process, the second frame processing unit 122 may perform the following noise removal process.

[0186] For example, such as Figure 19 As shown, in the second frame processing unit 122 at time T n In the case of generating the reconstructed image of the nth frame, noise removal processing is performed (where time T is used to remove noise). n For reference, the previously scheduled period T NR Pixels for which no event occurred are set to 0, which removes pixels that did not experience an event during a predetermined period T. NR An object that remains stationary for an extended period of time, or even longer. Here, the predetermined period T... NR The period is set to be longer than the object's transit time and shorter than the ejection interval of droplet 10 (object transit time < T). NR <The ejection interval of droplet 10). The transit time of the object is the time it takes for the object to travel the distance in the direction of movement of droplet 10.

[0187] The previously scheduled period T NRThe noise removal processing that sets the pixel value of a pixel in which no event has occurred to 0 can be performed by any algorithm. For example, the following processing can be employed: the time information of the most recent event occurring in each pixel is held, and the pixel value of a pixel in which no event has occurred since the time T n a long predetermined period T NR is set to 0.

[0188] <7. Processing of the noise removal processing unit>

[0189] Next, the noise removal processing of the noise removal processing unit 112 will be described.

[0190] As the noise removal processing, the noise removal processing unit 112 performs filter processing using the dilation processing and the erosion processing of white pixels on each of the event image and the reconstructed image that are binary images.

[0191] First, the noise removal processing unit 112 performs opening processing in which the erosion processing and the dilation processing of white pixels are sequentially performed on each of the event image and the reconstructed image, and then performs closing processing in which the dilation processing and the erosion processing of white pixels are sequentially performed. The filter size is, for example, five pixels.

[0192] Figure 20 Examples of the reconstructed image before and after the noise removal processing performed by the noise removal processing unit 112 are shown. Small white pixels estimated as noise are erased by the noise removal processing of the noise removal processing unit 112.

[0193] <8. Processing of the droplet detection unit>

[0194] Next, the processing of the droplet detection unit 103 will be described.

[0195] The droplet detection unit 103 detects the droplet 10 from each of the event image and the reconstructed image provided from the preprocessing unit 101. For example, the droplet detection unit 103 detects the droplet 10 from the event image and provides the result as information on a tracking target to the droplet tracking unit 104. Further, the droplet detection unit 103 detects the droplet 10 from the reconstructed image, calculates the size of the detected droplet 10, and provides the result to the parameter determination unit 105. Note that the result of the droplet 10 detected from the reconstructed image can be provided as information on a tracking target to the droplet tracking unit 104, or the size of the droplet 10 can be calculated from the droplet 10 detected from the event image.

[0196] Figure 21 is a diagram for explaining the droplet detection processing of the droplet detection unit 103.

[0197] The droplet detection unit 103 performs a labeling process on the binary image 151 of the event image or the reconstructed image supplied from the preprocessing unit 101 to attribute labels 161 to the droplets 10 in the binary image 151. In Figure 21 In the example, the labels 161A to 161C are attributed to the three droplets 10 in the binary image 151.

[0198] The droplet detection unit 103 determines whether the detected labels 161A to 161C extend across two boundary lines 171 that are set in advance for the binary image 151, and selects a label that extends across the two boundary lines 171 as a detection candidate. In Figure 21 In the example, the label 161A extends across the two boundary lines 171, and the droplet 10 of the label 161A is selected as a detection candidate.

[0199] Next, as shown in Figure 22 , of the two boundary lines 171, assuming that the upper side opposite to the moving direction of the droplet 10 is the boundary line 171A and the lower side is the boundary line 171B, the droplet detection unit 103 registers (stores) an image of a region 182 that is obtained by enlarging the periphery from the lower side of the boundary line 171A by a certain width with respect to a rectangular region 181 that encloses the region from the lower side of the droplet 10 of the upper side boundary line 171A as a template image. The droplet detection unit 103 supplies the registered template image to the droplet tracking unit 104 as information on a tracking target. As described above, a part of the distal end of the droplet 10 is registered as the template image and set as a tracking target, so that tracking can be performed even if the droplet 10 has a shape with a drawn tail.

[0200] Note that, depending on the setting of the frame rate, for example, as shown in Figure 23 , the same droplet 10 as the template image that has been registered and tracked by the droplet tracking unit 104 can be detected in a frame adjacent to the frame in which the template image is registered. In this case, the droplet detection unit 103 does not register (store) the template image.

[0201] More specifically, referring to Figure 23 , in a predetermined frame, a template image 183 shown on the left side of Figure 23 is registered based on a rectangular region 181 that surrounds the region of the droplet 10 from the lower side of the boundary line 171A. Then, in the next frame, the same droplet 10 is selected as a detection candidate, and a rectangular region 181' that surrounds the region of the droplet 10 located on the lower side of the boundary line 171A is set. However, because the rectangular region 181' overlaps the template image 183 that is being tracked, the rectangular region is not registered as a template image.

[0202] Figure 24is a diagram that is an example of a method for explaining the size of the liquid droplet 10.

[0203] The liquid droplet detection unit 103 calculates the width 184 of the liquid droplet 10 in the first row of the registered template image 183 as the size of the liquid droplet 10. More specifically, the liquid droplet detection unit 103 sets the number of pixels of the liquid droplet 10 in the first row of the registered template image 183 as the width 184 of the liquid droplet 10.

[0204] Note that, as described above, the width 184 of the liquid droplet 10 in the first row of the registered template image 183 can be calculated as the size of the liquid droplet 10, or the size of the liquid droplet 10 can be obtained by other calculation methods. For example, the number of pixels of the liquid droplet 10 in the registered template image 183, the number of pixels in the vertical direction and the horizontal direction of the registered template image 183, and the like can be calculated as the size. Further, with reference to Figure 9 The described volume can be calculated as the size.

[0205] Next, with reference to the flowchart of Figure 25 the liquid droplet detection processing performed by the liquid droplet detection unit 103 is described. This processing is started, for example, when the binary image 151 of the event image or the reconstructed image is first provided from the preprocessing unit 101. Figure 25 The liquid droplet detection processing in is processing of one binary image 151 provided from the preprocessing unit 101.

[0206] First, in step S71, the liquid droplet detection unit 103 performs a labeling process on one binary image 151 provided from the preprocessing unit 101. As a result, a label 161 is given to all liquid droplets 10 included in the one binary image 151.

[0207] In step S72, the liquid droplet detection unit 103 selects one predetermined label 161 from one or more labels 161 (liquid droplets 10) included in the binary image 151.

[0208] In step S73, the liquid droplet detection unit 103 determines whether the selected label 161 extends across two boundary lines 171.

[0209] When it is determined that the selected label 161 in step S73 does not extend beyond two boundary lines 171, the processing proceeds to step S77.

[0210] However, when it is determined that the selected label 161 in step S73 extends across two boundary lines 171, the processing proceeds to step S74, and the liquid droplet detection unit 103 determines whether the selected label 161 overlaps with the template image being tracked. More specifically, as with reference to Figure 23It is described whether the rectangular region 181 that encloses a region of the selected mark 161 in which the droplet 10 is below the upper side boundary line 171A overlaps the template image 183 being tracked.

[0211] When it is determined that the selected mark 161 in step S74 overlaps the template image being tracked, the processing proceeds to step S77.

[0212] However, when it is determined in step S74 that the selected mark 161 does not overlap the template image being tracked, the processing proceeds to step S75, and the droplet detection unit 103 registers a portion of the distal end of the selected mark 161 as the template image 183. More specifically, among the selected mark 161, the droplet detection unit 103 registers, as the template image 183, a region 182 obtained by enlarging the outer periphery from the lower side of the boundary line 171A by a constant width with respect to the rectangular region 181 that encloses a region of the droplet 10 below the upper side boundary line 171A. The registered template image 183 is supplied to the droplet tracking unit 104.

[0213] In step S76, the droplet detection unit 103 calculates the width 184 of the droplet 10 in the first row of the registered template image 183 as the size of the droplet 10. The calculated width 184 of the droplet 10 is supplied to the parameter determination unit 105.

[0214] In step S77, the droplet detection unit 103 determines whether all of the marks 161 have been selected.

[0215] When it is determined in step S77 that all of the marks 161 have not been selected, the processing returns to step S72, and the above-described steps S72 to S77 are executed again. That is, a mark 161 that has not been selected is selected, and it is determined whether the mark 161 extends across two boundary lines 171 or overlaps the template image being tracked.

[0216] However, when it is determined in step S77 that all of the marks 161 have been selected, Figure 25 the droplet detection processing ends.

[0217] Figure 25 The droplet detection processing in the above-described steps S71 to S77 is processing on one binary image 151 supplied from the preprocessing unit 101, and the above-described droplet detection processing is executed on the binary image 151 supplied continuously from the preprocessing unit 101.

[0218] <9. Processing of droplet tracking unit>

[0219] Next, the processing of the droplet tracking unit 104 will be described.

[0220] The droplet tracking unit 104 tracks the droplet 10 detected by the droplet detection unit 103, calculates trajectory information of the droplet 10, and provides the trajectory information to the parameter determination unit 105. More specifically, the droplet tracking unit 104 searches for the droplet 10 by using template matching of the template image 183 provided from the droplet detection unit 103, to find a frame after the frame in which the template image 183 is registered.

[0221] Figure 26 is a diagram for explaining the search of the droplet 10 in the first frame from the frame in which the template image is registered.

[0222] In Figure 26 , the droplet 10 indicated by the dotted line indicates the position of the droplet 10 in a frame before the current frame (i.e., the frame in which the template image 183 is registered).

[0223] The droplet tracking unit 104 searches for the droplet 10 in the current frame by template matching using a range of a predetermined radius r1 from the center 191 of the template image 183 in the previous frame as a search range.

[0224] Figure 27 is a diagram for explaining the search process of the droplet 10 in the second frame and subsequent frames from the frame in which the template image is registered.

[0225] In Figure 27 , the droplet 10 indicated by the dotted line indicates the position of the droplet 10 searched in a frame before the current frame. The position 192 corresponds to the center of the template image 183 with respect to which the droplet 10 is searched. In the case where the current frame is the second frame from the frame in which the template image is registered, Figure 27 the position 192 in Figure 26 corresponds to

[0226] In the second frame and subsequent frames, the droplet tracking unit 104 calculates the movement amount 193 of the droplet 10 within one frame period At. The movement amount 193 of the droplet 10 within one frame period At is calculated using the distance between the center position of the template image 183 detected in a frame one frame before and the center position of the template image 183 detected in a frame two frames before.

[0227] Then, the droplet tracking unit 104 calculates the position 194 of the droplet 10 moved from the position 192 of the template image 183 in the frame one frame before by the movement amount 193 within one frame period At as a predicted position of the droplet 10, and searches for the droplet 10 by using a range of a predetermined radius r2 centered on the predicted position as a search range. Here, the radius r2 used to set the search range of the second frame and subsequent frames is set to be smaller than the radius r1 used to set the search range of the first frame (r2 < r1).

[0228] As described above, the moving amount of the droplet 10 cannot be predicted in the next frame (first frame) of the frame in which the template image 183 is registered, so that the droplet 10 is searched by template matching with a range of the radius r1 from the center 191 at the time when the template image 183 is registered as a search range.

[0229] However, in the second frame and subsequent frames of the frame in which the template image 183 is registered, the moving amount of the droplet 10 can be calculated from the previous search result. Therefore, the droplet 10 is searched by template matching with a radius r2 smaller than the radius r1 centered on the predicted position as a search range, based on the calculated moving amount.

[0230] Reference Figure 28 The droplet tracking processing performed by the droplet tracking unit 104 will be described with reference to the flowchart of FIG. 10. This processing is started, for example, when the binary image 151 of the event image or the reconstructed image is first supplied from the preprocessing unit 101. Figure 28 The droplet tracking processing in FIG. 10 is processing for one binary image 151 supplied from the preprocessing unit 101.

[0231] First, in step S101, the droplet tracking unit 104 determines whether the droplet 10 being searched is a search of the second frame or subsequent frames from the frame in which the template image 183 is registered.

[0232] When it is determined in step S101 that the droplet 10 being searched is not a search of the second frame or subsequent frames (i.e., a search of the next frame of the frame in which the template image 183 is registered), the processing proceeds to step S102, and the droplet tracking unit 104 sets a range of a predetermined radius r1 from the center 191 of the template image 183 in the frame one frame before as a search range.

[0233] On the other hand, when it is determined in step S101 that the droplet 10 being searched is a search of the second frame or subsequent frames, the processing proceeds to step S103, and the droplet tracking unit 104 calculates the moving amount 193 of the droplet 10 within the one frame period At based on the center position of the template image 183 detected in the frame one frame before and the center position of the template image 183 detected in the frame one frame before.

[0234] Subsequently, in step S104, the droplet tracking unit 104 calculates the position 194 of the droplet 10 moved by the moving amount 193 from the position 192 of the template image 183 of the frame one frame before within the one frame period At as a predicted position of the droplet 10, and sets a range of a predetermined radius r2 centered on the predicted position as a search range.

[0235] In step S105, the droplet tracking unit 104 searches for the droplet 10 within the search range set in step S102 or step S104 by template matching. In template matching, for example, the droplet tracking unit 104 calculates a correlation value by normalized cross-correlation, and obtains the coordinates of the image having the highest correlation value.

[0236] In step S106, the droplet tracking unit 104 determines whether or not an image having a correlation value equal to or greater than a predetermined threshold is detected. In step S106, in a case where the correlation value of the image having the highest correlation value detected in step S105 is less than the predetermined threshold, it is determined that an image having a correlation value equal to or greater than the predetermined threshold is not detected. However, in a case where the correlation value of the image having the highest correlation value detected in step S105 is equal to or greater than the predetermined threshold, it is determined that an image having a correlation value equal to or greater than the predetermined threshold is detected.

[0237] When it is determined in step S106 that an image having a correlation value equal to or greater than the predetermined threshold has been detected, the process proceeds to step S107, and the droplet tracking unit 104 updates the trajectory information of the droplet 10 being tracked. Specifically, the droplet tracking unit 104 adds the position information of the search result of the droplet 10 in the current frame to the position information of the droplet 10 stored as the trajectory information of the droplet 10 being tracked, per frame until the previous frame.

[0238] In step S108, the droplet tracking unit 104 updates the template image used in template matching to the image detected in the current frame. Note that the process in step S108 can be omitted, and the template image provided from the droplet detection unit 103 can be used continuously.

[0239] However, when it is determined in step S106 that an image having a correlation value equal to or greater than the predetermined threshold is not detected, the process proceeds to step S109, and the droplet tracking unit 104 considers that the droplet 10 has been lost, and provides the trajectory information of the droplet 10 stored so far in the detection to the parameter determination unit 105.

[0240] Figure 28 The droplet tracking process in the parameter determination unit 105 is a process on one binary image 151 provided from the preprocessing unit 101, and the above-described droplet detection process is performed on the binary image 151 provided continuously from the preprocessing unit 101. Until the droplet 10 being tracked is considered to be lost, the droplet 10 is searched for by template matching on the subsequent binary image 151, and the trajectory information is updated.

[0241] Note that the position information of the droplet 10 of each frame is stored and provided to the parameter determination unit 105 as the trajectory information of the droplet 10, but the trajectory information of the droplet 10 can be other information. For example, the velocity, moving direction, and the like of the droplet 10 calculated from the position of the droplet 10 of each frame can be used as the trajectory information of the droplet 10.

[0242] <10. Droplet control processing of dispenser control system>

[0243] Next, the droplet control processing performed by the entire dispenser control system 1 will be described with reference to the flowchart of Figure 29 . This processing is started, for example, when a predetermined control start operation is performed on the dispenser control system 1.

[0244] First, in step S141, the EVS camera 12 detects a luminance change as an event based on the droplet 10 ejected from the dispenser 11 and outputs event data to the control device 13.

[0245] In step S142, the first frame processing unit 121 of the control device 13 performs first framing processing of generating an event image and a display image based on the event data from the EVS camera 12. Specifically, the first frame processing unit 121 performs the first framing processing described with reference to Figure 14 .

[0246] In step S143, the second frame processing unit 122 of the control device 13 performs second framing processing of generating a reconstruction image in which a luminance value is estimated based on the event data from the EVS camera 12. Specifically, the second frame processing unit 122 performs the second framing processing described with reference to Figure 17 .

[0247] In step S144, the noise removal processing unit 112 performs noise removal processing by filter processing of the expansion processing and the contraction processing with respect to each of the event image or the reconstruction image generated by the framing processing unit 111.

[0248] In step S145, the display 14 acquires the display image generated by the framing processing unit 111 from the control device 13 and displays the display image.

[0249] In step S146, the droplet detection unit 103 of the control device 13 performs droplet detection processing of detecting the droplet 10 as a tracking target in each of the event image and the reconstruction image provided from the preprocessing unit 101. Specifically, the droplet detection unit 103 performs the droplet detection processing described with reference to Figure 25 . In the droplet detection processing, the template image of the droplet 10 is provided to the droplet tracking unit 104, and the width 184 of the droplet 10 is calculated and provided to the parameter determination unit 105.

[0250] In step S147, the droplet tracking unit 104 of the control device 13 performs the following droplet tracking process: tracking the droplet 10 detected by the droplet detection unit 103, calculating the trajectory information of the droplet 10, and providing the trajectory information to the parameter determination unit 105. Specifically, the droplet tracking unit 104 performs a reference... Figure 28 The droplet detection process is described, and the trajectory information of the droplet 10 is provided to the parameter determination unit 105.

[0251] In step S148, the parameter determination unit 105 of the control device 13 performs anomaly determination processing to determine whether the control parameters of the dispenser 11 are within the normal range. For example, as anomaly determination processing, the parameter determination unit 105 determines whether the ejection timing and ejection volume of the dispenser 11 are within the appropriate range based on the width 184 of the droplet 10 provided by the droplet detection unit 103 and the trajectory information of the droplet 10 provided by the droplet tracking unit 104.

[0252] In step S149, the parameter determination unit 105 determines whether to change the control parameters of the distributor 11. For example, when it is determined that the spray timing or spray volume of the distributor 11 is not within an appropriate range, the parameter determination unit 105 determines to change the control parameters.

[0253] When it is determined in step S149 that the control parameters need to be changed, the process proceeds to step S150, and the parameter determination unit 105 generates control information for correcting the parameters as feedback control information and outputs the control information to the distributor 11.

[0254] However, if it is determined in step S149 that the control parameters will not be changed, the processing in step S150 is skipped.

[0255] In step S151, the control device 13 determines whether to terminate the control. For example, the parameter determination unit 105 of the control device 13 determines to terminate the control when it detects that a control termination operation has been performed, and determines not to terminate the control in other cases.

[0256] If it is determined in step S151 that the control has not yet ended, the process returns to step S141 and repeats the above steps S141 to S151.

[0257] However, when the control is determined to be complete in step S151, the process ends. Figure 29 Droplet control processing.

[0258] For ease of description, the droplet control process has been described. Figure 29 Each step in the process is processed sequentially; however, in reality, each step is executed in parallel, and the results are output sequentially to subsequent stages.

[0259] <11. Application of DNN>

[0260] In the above droplet control processing, the event image and the reconstruction image have been generated based on the event data from the EVS camera 12, the position, the speed, the moving direction, and the like of the droplet 10 have been calculated from the generated event image or reconstruction image, and it has been determined whether the control parameter of the dispenser 11 is within the normal range.

[0261] However, for example, the processing of identifying the control parameter of the dispenser 11 can be performed using a deep neural network (DNN) to determine whether the control parameter of the dispenser 11 is within the normal range.

[0262] As shown in Figure 30 , the DNN obtains an identification result by performing feature extraction processing to extract a feature amount on input data and performing identification processing based on the extracted feature amount.

[0263] For example, the control device 13 can give at least one of the event image and the reconstruction image as input data to the DNN and, by learning, generate and use the DNN that outputs the control parameter of the dispenser 11 as an identification result.

[0264] Figure 31 A flowchart of droplet control processing in a case where the DNN performs identification processing using the event image and the reconstruction image as input data is shown.

[0265] Figure 31 Steps S201 to S205 and S207 to S209 in Figure 29 are the same processing as steps S141 to S145 and S149 to S151 in Figure 29 , and Figure 31 the processing in steps S146 to S148 in is changed to the DNN control parameter identification processing in step S206 in

[0266] In step S206, for example, the DNN processing unit of the control device 13 identifies the control parameter of the dispenser 11 based on the event image and the reconstruction image supplied from the preprocessing unit 101 and outputs the control parameter to the parameter determination unit 105. In step S207, the parameter determination unit 105 determines whether to change the control parameter based on the control parameter as an identification result.

[0267] Alternatively, the event data from the EVS camera 12 can be directly given as input data to the DNN, and such a DNN that outputs the control parameter of the dispenser 11 can be generated and used by learning as an identification result.

[0268] Figure 32A flowchart of the droplet control processing in a case where the DNN uses event data as input data to perform recognition processing is shown.

[0269] Figure 32 Steps S221 and S223 to S225 in Figure 29 are the same processing as steps S141 and S149 to S151 in Figure 29 the processing in steps S142 to S148 in Figure 32 is changed to the control parameter recognition processing by the DNN in step S222 in

[0270]

[0271] In a case where event data is output from the EVS camera 12 as data compressed in a predetermined compression format, the event data compressed in the predetermined compression format can be learned as is as input data, so that the control parameter can be recognized.

[0272] For machine learning for performing recognition processing, a spiking neural network (SNN) can be used in addition to a DNN.

[0273] According to the first embodiment described above, the EVS camera 12 detects a change in luminance as an event based on the droplet 10 ejected from the dispenser 11, and outputs event data to the control device 13. The control device 13 generates control information for controlling ejection of the droplet 10 by the dispenser 11 based on the event data output from the EVS camera 12, and outputs the control information to the dispenser 11. As a result, the droplet from the dispenser 11 can be accurately detected, and ejection of the droplet can be controlled with high precision.

[0274] <12. Second Embodiment of Dispenser Control System>

[0275] Figure 33 A configuration example of a second embodiment of a dispenser control system to which the present technology is applied is shown.

[0276] In Figure 33 , parts corresponding to those of the first embodiment shown in Figure 1 are denoted by the same reference numerals, and the description of these parts will be omitted as appropriate, focusing on the description of different parts.

[0277] In Figure 33In the dispenser control system 1, an RGB camera 201 that captures an image of the droplet 10 and outputs an RGB image (color image) is further added. For example, as shown in FIG. 10, the RGB camera 201 is arranged side by side with the EVS camera 12 in the moving direction of the droplet 10. A range different from the image capturing range of the EVS camera 12 is set as the image capturing range of the RGB camera 201. Figure 33

[0278] The control device 13 detects and tracks the droplet 10 by performing marker processing, template matching, or the like on the RGB image supplied from the RGB camera 201. Further, the control device 13 analyzes the color, shape, or the like of the droplet 10 based on the RGB image supplied from the RGB camera 201. The control device 13 acquires a correspondence relation with the trajectory information of the droplet 10 based on the event data of the EVS camera 12, and integrates the detection information of the droplet 10 of the event image or the reconstructed image based on the RGB image and the detection information of the droplet 10. The control device 13 can determine whether the control parameter is normal based on the integrated detection information, and performs feedback.

[0279] Note that, regarding the arrangement of the EVS camera 12 and the RGB camera 201, for example, in addition to the configuration in which they are arranged side by side in the moving direction of the droplet 10 as shown in FIG. 10, a configuration in which they are arranged to capture images of the same image capturing range via a half mirror 211 as shown in FIG. 11 can be employed. Figure 33 Figure 34

[0280] As shown in FIG. 12, in a case where the EVS camera 12 and the RGB camera 201 are arranged to have different image capturing ranges, different illumination conditions can be set for the EVS camera 12 and the RGB camera 201 so that illumination suitable for each camera can be used. Figure 33

[0281] However, in a case where the EVS camera 12 and the RGB camera 201 are arranged to capture the same image capturing range as shown in FIG. 13, it is easy to identify the correspondence relation of the droplet 10 detected by each camera. Figure 34

[0282] According to the above-described second embodiment, the EVS camera 12 detects a luminance change as an event based on the droplet 10 ejected from the dispenser 11, and outputs event data to the control device 13. The control device 13 generates control information for controlling the ejection of the droplet 10 by the dispenser 11 based on the event data output from the EVS camera 12, and outputs the control information to the dispenser 11. As a result, the droplet from the dispenser 11 can be accurately detected, and the ejection of the droplet can be controlled with high precision. Further, the droplet 10 can be accurately detected by using the RGB image captured by the RGB camera 201, and the ejection of the droplet can be controlled. ​​​​​

[0283] <13. Third embodiment of the dispenser control system>

[0284] Figure 35 A configuration example of a third embodiment of the dispenser control system to which the present technology is applied is shown.

[0285] In Figure 35 , parts corresponding to those of the first embodiment shown in Figure 1 are denoted by the same reference numerals, and the different parts are focused on, and the description thereof is omitted as appropriate.

[0286] Comparing Figure 35 the third embodiment shown in Figure 1 with the first embodiment shown in , the dispenser control system 1 according to the third embodiment has a configuration in which the EVS camera 12 and the control device 13 according to the first embodiment are replaced with an EVS camera 300.

[0287] The EVS camera 300 is an imaging device including an event sensor and a processing unit that executes the functions of the control device 13 in the first embodiment. That is, the EVS camera 300 detects a change in luminance as an event based on the liquid droplets 10 ejected from the dispenser 11, and generates event data. Further, the EVS camera 300 generates feedback control information for controlling the ejection of the liquid droplets 10 by the dispenser 11 based on the event data, and outputs the feedback control information to the dispenser 11. Further, the EVS camera 300 generates a display image to be monitored by the worker based on the event data, and causes the display 14 to display the display image.

[0288] <EVS camera configuration example>

[0289] Figure 36 is a block diagram showing a configuration example of the EVS camera 300 in Figure 35

[0290] The EVS camera 300 includes an optical unit 311, an imaging element 312, a recording unit 313, and a control unit 314.

[0291] The optical unit 311 collects light from an object and causes the light to enter the imaging element 312. The imaging element 312 is an event sensor that outputs event data indicating occurrence of an event in a case where an event occurs with a change in luminance in a pixel as the event.

[0292] The imaging element 312 photoelectrically converts incident light incident via the optical unit 311 to generate event data, and causes the recording unit 313 to record the event data. Further, the imaging element 312 generates feedback control information for controlling the ejection of the liquid droplets 10 by the dispenser 11 based on the event data, and outputs the feedback control information to the dispenser 11. Also, the imaging element 312 generates a display image based on the event data, and outputs the display image to the display 14.

[0293] The recording unit 313 records and accumulates the event data, the event image, and the like provided from the imaging element 312 into a predetermined recording medium. The control unit 314 controls the imaging element 312. For example, the control unit 314 instructs the imaging element 312 of the start and end of imaging, and specifies the frame rate of the event image, and the like.

[0294] Figure 37 is a perspective view showing a schematic configuration example of the imaging element 312.

[0295] The imaging element 312 has a layered structure in which the light-receiving chip 321 and the detection chip 322 are joined and layered. For example, the light-receiving chip 321 and the detection chip 322 are electrically connected via a connection portion such as a via, a Cu-Cu bonding, or a bump.

[0296] Figure 38 is a plan view showing a configuration example of the light-receiving chip 321.

[0297] The light-receiving chip 321 includes a light-receiving portion 341 formed in a chip center portion, and one or a plurality of via arrangement portions 342 formed in an outer peripheral portion outside the light-receiving portion 341. In the example of Figure 38 In the example of, three vias are arranged in the corner portions of the chip periphery.

[0298] In the light-receiving portion 341, a plurality of photodiodes 351 are arranged in a two-dimensional lattice pattern. The photodiodes 351 photoelectrically convert incident light to generate a photoelectric current. Each of the photodiodes 351 is assigned a pixel address including a row address and a column address, and is regarded as a pixel. In the via arrangement portion 342, a via that is electrically connected to the detection chip 322 is arranged.

[0299] Figure 39 is a plan view showing a configuration example of the detection chip 322.

[0300] The detection chip 322 includes one or a plurality of via arrangement portions 361, an address event detection unit 362, a row drive circuit 363, a column drive circuit 364, and a signal processing circuit 365.

[0301] The via arrangement portion 361 is provided at a position corresponding to the via arrangement portion 342 of the light-receiving chip 321, and is electrically connected to the light-receiving chip 321 through a via. In Figure 39 , the via arrangement portion 361 is separately provided at positions corresponding to the three via arrangement portions 342 in Figure 38 , and a total of three via arrangement portions 361 are formed on the detection chip 322.

[0302] The address event detection unit 362 generates a detection signal from a photoelectric current of each of the plurality of photodiodes 351 of the light-receiving chip 321, and outputs the detection signal to the signal processing circuit 365. The detection signal is a 1-bit signal indicating whether or not the fact that the light quantity of incident light exceeds a predetermined threshold is detected as an address event.

[0303] The row drive circuit 363 selects a predetermined row address of the address event detection unit 362, and outputs the detection signal of the selected row address to the signal processing circuit 365.

[0304] The column drive circuit 364 selects a predetermined column address of the address event detection unit 362, and outputs the detection signal of the selected column address to the signal processing circuit 365.

[0305] The signal processing circuit 365 performs predetermined signal processing on the detection signal output from the address event detection unit 362. Further, for example, the signal processing circuit 365 generates event data based on the detection signal output from the address event detection unit 362, and further performs processing of generating an event image, reconstructing an image, and displaying an image. Further, the signal processing circuit 365 determines whether or not the control parameter of the dispenser 11 is within a normal range based on the generated event image and reconstructed image, and generates and outputs feedback control information in a case where it is determined that the control parameter is outside the normal range. Thus, in the third embodiment, the processing performed by the control device 13 in Figure 1 is performed by the signal processing circuit 365.

[0306] Figure 40 is a plan view showing details of the address event detection unit 362.

[0307] In the address event detection unit 362, a plurality of address event detection circuits 371 are arranged in a two-dimensional grid pattern. The address event detection circuits 371 are arranged on the light-receiving chip 321, for example, in a one-to-one correspondence with the photodiodes 351. Each address event detection circuit 371 is electrically connected to the corresponding photodiode 351 through a via, a Cu-Cu bonding, or the like.

[0308] Figure 41 is a block diagram showing a configuration example of the address event detection circuit 371.

[0309] The address event detection circuit 371 includes a current-voltage conversion circuit 381, a buffer 382, a subtracter 383, a quantizer 384, and a transfer circuit 385.

[0310] The current-voltage conversion circuit 381 converts a photocurrent from the corresponding photodiode 351 into a voltage signal. The current-voltage conversion circuit 381 generates a voltage signal corresponding to a logarithmic value of the photocurrent, and outputs the voltage signal to the buffer 382.

[0311] The buffer 382 buffers the voltage signal from the current-voltage conversion circuit 381, and outputs the voltage signal to the subtracter 383. The buffer 382 makes it possible to ensure isolation of noise accompanying switching operation in the subsequent stage, and makes it possible to improve a driving force for driving the subsequent stage. Note that the buffer 382 can be omitted.

[0312] The subtracter 383 reduces a level of the voltage signal from the buffer 382 in accordance with a row drive signal from the row drive circuit 363. The subtracter 383 outputs the reduced voltage signal to the quantizer 384.

[0313] The quantizer 384 quantizes the voltage signal from the subtracter 383 into a digital signal, and supplies the digital signal as a detection signal to the transfer circuit 385. The transfer circuit 385 transfers (outputs) the detection signal to the signal processing circuit 365 in accordance with a column drive signal from the column drive circuit 364.

[0314] Figure 42 is a circuit showing a detailed configuration of the current-voltage conversion circuit 381. Although the current-voltage conversion circuit 381 is arranged on the detection chip 322, Figure 42 The photodiode 351 of the light-receiving chip 321 connected to the current-voltage conversion circuit 381 is also shown.

[0315] The current-voltage conversion circuit 381 includes FETs 411 to 413. As the FET 411 and the FET 413, for example, an N-type metal oxide semiconductor (NMOS) FET can be employed, and as the FET 412, for example, a P-type metal oxide semiconductor (PMOS) FET can be employed.

[0316] The photodiode 351 of the light-receiving chip 321 receives incident light, performs photoelectric conversion, and generates and allows a flow of a photocurrent as an electric signal. The current-voltage conversion circuit 381 converts the photocurrent from the photodiode 351 into a voltage (hereinafter, also referred to as a photo-voltage) Vo corresponding to a logarithm of the photocurrent, and outputs the voltage Vo to the buffer 382 Figure 41 ).

[0317] The source of the FET 411 is connected to the gate of the FET 413, and a photocurrent from the photodiode 351 flows through the connection point of the source of the FET 411 and the gate of the FET 413. The drain of the FET 411 is connected to the power supply VDD, and the gate thereof is connected to the drain of the FET 413.

[0318] The source of the FET 412 is connected to the power supply VDD, and the drain thereof is connected to the connection point between the gate of the FET 411 and the drain of the FET 413. A predetermined bias voltage Vbias is applied to the gate of the FET 412. The source of the FET 413 is grounded.

[0319] The drain of the FET 411 is connected to the power supply VDD side, and is a source follower. The photodiode 351 is connected to the source of the FET 411 which is a source follower, and this connection allows a photocurrent caused by electric charges generated by photoelectric conversion of the photodiode 351 to flow through the FET 411 (from the drain to the source). The FET 411 operates in a subthreshold region, and a photo voltage Vo corresponding to the logarithm of the photocurrent flowing through the FET 411 appears at the gate of the FET 411. As described above, in the photodiode 351, the photocurrent from the photodiode 351 is converted into the photo voltage Vo corresponding to the logarithm of the photocurrent by the FET 411.

[0320] The photo voltage Vo is output from the connection point of the gate of the FET 411 and the drain of the FET 413 to the subtracter 383 via the buffer 382.

[0321] Figure 43 A detailed configuration of the subtracter 383 and the quantizer 384 is shown.

[0322] The subtracter 383 calculates, with respect to the photo voltage Vo from the current voltage conversion circuit 381, a difference between a current photo voltage and a photo voltage of timing of a minute time difference from the current time, and outputs a difference signal Vout corresponding to the difference.

[0323] The subtracter 383 includes a capacitor 431, an operational amplifier 432, a capacitor 433, and a switch 434. The quantizer 384 includes a comparator 451.

[0324] One end of the capacitor 431 is connected to the output end of the buffer 382 Figure 41 ), and the other end thereof is connected to an input end of the operational amplifier 432. Thus, the photo voltage Vo is input to the (inverted) input end of the operational amplifier 432 via the capacitor 431.

[0325] An output end of the operational amplifier 432 is connected to a non-inverted input end (+) of the comparator 451 of the quantizer 384.

[0326] One end of the capacitor 433 is connected to an input terminal of the operational amplifier 432, and the other end is connected to an output terminal of the operational amplifier 432.

[0327] The switch 434 is connected to the capacitor 433 to turn on / off the connection between both ends of the capacitor 433. The switch 434 turns on / off the connection between both ends of the capacitor 433 by turning on / off according to a row drive signal of the row drive circuit 363.

[0328] The capacitor 433 and the switch 434 constitute a switched capacitor. When the switch 434 that has been turned off is temporarily turned on and turned off again, the capacitor 433 is reset to a state in which the electric charge is discharged and the electric charge can be re-accumulated.

[0329] The photo-voltage Vo of the capacitor 431 on the photodiode 351 side when the switch 434 is turned on is denoted by Vinit, and the capacitance (static capacitance) of the capacitor 431 is denoted by Cl. In the case where the switch 434 is turned on, the input terminal of the operational amplifier 432 is virtually grounded, and the electric charge Qinit accumulated in the capacitor 431 is denoted by Equation (1),

[0330] Qinit = Cl x Vinit · · · (1).

[0331] Further, in the case where the switch 434 is turned on, both ends of the capacitor 433 are short-circuited, so that the electric charge accumulated in the capacitor 433 becomes 0.

[0332] Thereafter, when the photo-voltage Vo of the capacitor 431 on the photodiode 351 side in the case where the switch 434 is turned off is denoted by Vafter, the electric charge Qafter accumulated in the capacitor 431 when the switch 434 is turned off is denoted by Equation (2),

[0333] Qafter = Cl x Vafter · · · (2).

[0334] When the capacitance of the capacitor 433 is denoted by C2, then the electric charge Q2 accumulated in the capacitor 433 is denoted by Equation (3) by using the difference signal Vout that is the output voltage of the operational amplifier 432,

[0335] Q2 = -C2 x Vout · · · (3).

[0336] The total amount of electric charge of the electric charge of the capacitor 431 and the electric charge of the capacitor 433 does not change before and after the switch 434 is turned off, so that Equation (4) is established,

[0337] Qinit = Qafter + Q2 · · · (4).

[0338] When the equations (1) to (3) are substituted into the equation (4), the equation (5) is obtained,

[0339] Vout = -(C1 / C2) x (Vafter - Vinit)...(5).

[0340] According to the equation (5), the subtracter 383 subtracts the photovoltage Vafter and Vinit, that is, calculates a difference signal Vout corresponding to a difference (Vafter - Vinit) of the photovoltage Vafter and Vinit. According to the equation (5), the subtraction gain of the subtracter 383 is C1 / C2. Therefore, the subtracter 383 outputs, as the difference signal Vout, a voltage obtained by multiplying the change of the photovoltage Vo after the reset of the capacitor 433 by C1 / C2.

[0341] The subtracter 383 outputs the difference signal Vout by turning on and off the switch 434 together with the row drive signal output from the row drive circuit 363.

[0342] The comparator 451 of the quantizer 384 compares the difference signal Vout from the subtracter 383 with a predetermined threshold voltage Vth input to the inverting input terminal (-) and outputs the comparison result as a detection signal to the transfer circuit 385.

[0343] Note that, in the configuration example of Figure 42 , only the photodiode 351 of the light-receiving section 341 is arranged on the light-receiving chip 321, and the address event detection circuit 371 including the current-voltage conversion circuit 381 is arranged on the detection chip 322. In this case, the circuit scale in the detection chip 322 can increase as the number of pixels increases. Therefore, a part of the address event detection circuit 371 can be arranged in the light-receiving section 341 of the light-receiving chip 321.

[0344] For example, as Figure 44 indicated, a part of the current-voltage conversion circuit 381 of the address event detection circuit 371 (for example, the FET 411 and the FET 413 configured with N-type MOS (NMOS) FETs) can be arranged in the light-receiving section 341 of the light-receiving chip 321. In this case, the light-receiving chip 321 includes only N-type MOS FETs, and the current-voltage conversion circuit 381 of the detection chip 322 includes only P-type MOS FETs. Therefore, the number of steps of forming transistors can be reduced compared to the case of mixing P-type and N-type. Therefore, the manufacturing cost of the light-receiving chip 321 can be reduced.

[0345] According to the above-described third embodiment, the EVS camera 300 can detect the droplet 10 of the dispenser 11 based on the event data generated by itself, generate control information for controlling the ejection of the droplet 10, and output the control information to the dispenser 11. As a result, the droplet from the dispenser 11 can be accurately detected, and the ejection of the droplet can be controlled with high precision.

[0346] <14. Conclusion>

[0347] According to each of the above-described embodiments of the dispenser control system 1, by using the EVS camera 12 configured to detect a change in brightness as an event and to output asynchronously, the amount of calculation and the amount of communication can be greatly reduced compared to a high-speed camera with a frame rate of about 1000 fps, and the image of the droplet 10 can be captured at high speed. Furthermore, since the event image or the reconstructed image generated based on the event data is a binary image, the amount of calculation is small, and it is also possible to reduce power consumption.

[0348] According to the droplet control processing of the dispenser control system 1, the trajectory information, the size (width), the volume, and the like of the droplet 10 can be detected at high speed from the event image or the reconstructed image generated based on the event data, and the control parameter of the dispenser 11 can be controlled with high precision. Therefore, the droplet 10 from the dispenser 11 can be accurately detected and the ejection of the droplet 10 can be controlled with high precision. It is possible to determine in real time whether the ejection of the droplet 10 by the dispenser 11 is good or bad.

[0349] <15. Computer Configuration Example>

[0350] The control processing of the droplet 10 performed by the above-described control device 13 can be performed by hardware or software. In the case where a series of processes is performed by software, a program configuring the software is installed in a computer. Here, examples of the computer include, for example, a microcomputer built into a dedicated hardware, a general-purpose personal computer capable of performing various functions by installing various programs, and the like.

[0351] Figure 45 is a block diagram showing a configuration example of a computer as an information processing device that performs a series of droplet control processes described above by a program.

[0352] In the computer, a central processing unit (CPU) 501, a read only memory (ROM) 502, and a random access memory (RAM) 503 are connected to each other by a bus 504.

[0353] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a storage unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0354] Input unit 506 includes a keyboard, mouse, microphone, touch panel, input terminals, etc. Output unit 507 includes a display, speaker, output terminals, etc. Storage unit 508 includes a hard disk, RAM disk, non-volatile memory, etc. Communication unit 509 includes a network interface, etc. Driver 510 drives a removable recording medium 511 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0355] In the computer configured as described above, for example, the CPU 501 loads the program recorded in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and performs the series of processes described above. The RAM 503 also appropriately stores the data required by the CPU 501 to perform various processes.

[0356] The program executed by the computer (CPU 501) can be provided by recording it on a removable recording medium 511, such as a packaging medium. Alternatively, the program can be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0357] In a computer, a program can be installed in the storage unit 508 via the input / output interface 505 by attaching the removable recording medium 511 to the drive 510. Alternatively, the program can be received by the communication unit 509 via a wired or wireless transmission medium and installed in the storage unit 508. Furthermore, the program can be pre-installed in the ROM 502 and the storage unit 508.

[0358] It should be noted that a program executed by a computer may be a program that performs processing in chronological order according to the order described in this specification, or it may be a program that performs processing in parallel or as needed (e.g., when a call is made).

[0359] The embodiments of this technology are not limited to the above embodiments, and various modifications can be made without departing from the scope of this technology.

[0360] For example, it is possible to appropriately adopt all or part of the combinations of the various embodiments described above.

[0361] Furthermore, each step described in the flowchart above can be performed by one device, or it can be shared and performed by multiple devices.

[0362] Furthermore, in cases where a step includes multiple processes, the multiple processes included in a step can be executed by a single device, and can also be shared and executed by multiple devices.

[0363] It should be noted that the effects described in this specification are merely examples and are not limited thereto, and effects other than those described in this specification may exist.

[0364] Note that the present technology can have the following configurations. (1)

[0366] An information processing apparatus comprising:

[0367] an event sensor including a pixel configured to photoelectrically convert a light signal and output a pixel signal, the event sensor being configured to output, as an event signal, a temporal luminance change of the light signal based on the pixel signal; and

[0368] a processor configured to detect a droplet ejected from a dispenser based on the event signal. (2)

[0370] The information processing apparatus according to (1) above, in which

[0371] the event sensor is disposed such that a reading direction of the pixel signal coincides with a moving direction of the droplet. (3)

[0373] The information processing apparatus according to any one of (1) to (3) above, in which

[0374] the event sensor detects the droplet from two orthogonal directions. (4)

[0376] The information processing apparatus according to any one of (1) to (3) above, in which

[0377] the two orthogonal directions are detected by different event sensors. (5)

[0379] The information processing apparatus according to any one of (1) to (3) above, in which

[0380] the two orthogonal directions are detected by one of the event sensors. (6)

[0382] The information processing apparatus according to any one of (1) to (5) above, in which

[0383] the processor further generates and outputs control information for controlling ejection of the droplet by the dispenser. (7)

[0385] The information processing apparatus according to (6) above, in which

[0386] the processor generates an event image in which event signals from the event sensor are collected for each predetermined integration time, and generates the control information based on the generated event image. (8)

[0388] The information processing apparatus according to (7) above, wherein

[0389] The processor generates a first event image based on the event signals of the first events that are luminance changes of a first polarity, as an event image; and generates a second event image based on the event signals of the second events that are luminance changes of a second polarity, the second polarity being a polarity opposite to the first polarity. (9)

[0391] The information processing apparatus according to (7) or (8) above, wherein

[0392] The event image is a binary image in which a pixel value of a pixel in which the event signal occurs is set to 1 and a pixel value of a pixel in which the event signal does not occur is set to 0. (10)

[0394] The information processing apparatus according to any one of (7) to (9) above, wherein

[0395] The predetermined integration time is shorter than a one-frame period corresponding to the frame rate. (11)

[0397] The information processing apparatus according to any one of (8) to (10) above, wherein

[0398] The processor further generates a display image in which a pixel value of a pixel in which the first event occurs is set to a first luminance value, a pixel value of a pixel in which the second event occurs is set to a second luminance value, and a pixel value of another pixel is set to a third luminance value. (12)

[0400] The information processing apparatus according to any one of (8) to (11) above, wherein

[0401] The processor calculates trajectory information of the droplet based on the first event image, and generates control information based on the calculated trajectory information. (13)

[0403] The information processing apparatus according to any one of (8) to (12) above, wherein

[0404] The processor performs noise removal processing that removes noise of the first event image. (14)

[0406] The information processing apparatus according to any one of (1) to (13) above, wherein

[0407] The processor generates a reconstructed image of an estimated luminance value based on the event signals for each frame rate period corresponding to the frame rate. (15)

[0409] The information processing apparatus according to (14) above, wherein

[0410] The processor generates a reconstructed image by using all of the event signals in the past time. (16)

[0412] The information processing apparatus according to (14) or (15) above, wherein

[0413] The processor calculates a width of the droplet based on the reconstructed image, and generates control information for controlling the dispenser to eject the droplet based on the calculated width. (17)

[0415] The information processing apparatus according to any one of (14) to (16) above, wherein

[0416] The processor calculates a volume of the droplet based on the reconstructed image, and generates control information for controlling the dispenser to eject the droplet based on the calculated volume. (18)

[0418] The information processing apparatus according to any one of (14) to (17) above, wherein

[0419] The processor performs noise removal processing of setting pixel values of pixels in which no event has occurred within a certain period of time before a timing of generation of the reconstructed image to zero. (19)

[0421] The information processing apparatus according to any one of (1) to (17) above, wherein

[0422] The processor detects the droplet also based on an RGB image obtained by capturing an image of the droplet using the image sensor. (20)

[0424] An information processing system including:

[0425] A dispenser configured to eject a predetermined liquid;

[0426] An event sensor including a pixel configured to photoelectrically convert a light signal and output a pixel signal, the event sensor being configured to output, as an event signal, a temporal luminance change of the light signal based on the pixel signal; and

[0427] A processor configured to detect a droplet ejected from the dispenser based on the event signal.

[0428] Reference symbol list

[0429] 1 Dispenser control system

[0430] 10 Droplet

[0431] 11 Dispenser

[0432] 12 EVS camera

[0433] 13 control device

[0434] 14 display

[0435] 21 substrate

[0436] 22 conveyor

[0437] 61 illuminating device

[0438] 101 preprocessing unit

[0439] 102 image output unit

[0440] 103 droplet detection unit

[0441] 104 droplet tracking unit

[0442] 105 parameter determination unit

[0443] 111 framing processing unit

[0444] 112 noise removal processing unit

[0445] 121 first frame processing unit

[0446] 122 second frame processing unit

[0447] 201 RGB camera

[0448] 300 EVS camera

[0449] 312 imaging element

[0450] 365 signal processing circuit

[0451] 501 CPU

[0452] 502 ROM

[0453] 503 RAM

[0454] 506 input unit

[0455] 507 output unit

[0456] 508 storage unit

[0457] 509 communication unit

Claims

1. An information processing apparatus, comprising: An event sensor includes pixels configured to photoelectrically convert light signals and output pixel signals, and the event sensor is configured to output a time-varying brightness change of the light signal based on the pixel signals as an event signal; as well as The processor is configured to detect droplets ejected from the dispenser based on the event signal; The processor further generates and outputs control information for controlling the ejection of the droplets through the dispenser; The processor generates an event image by collecting event signals from the event sensor in units of predetermined integration time, and generates the control information based on the generated event image.

2. The information processing apparatus according to claim 1, wherein... The event sensor is configured such that the reading direction of the pixel signal is consistent with the movement direction of the droplet.

3. The information processing apparatus according to claim 1, wherein... The event sensor detects the droplets from two orthogonal directions.

4. The information processing apparatus according to claim 3, wherein The two orthogonal directions are detected by different event sensors.

5. The information processing apparatus according to claim 3, wherein The two orthogonal directions are detected by one of the event sensors.

6. The information processing apparatus according to claim 1, wherein The processor generates a first event image and a second event image as the event images. The first event image is based on an event signal of a first event, which is a brightness change of a first polarity. The second event image is based on an event signal of a second event, which is a brightness change of a second polarity, which is opposite to the first polarity.

7. The information processing apparatus according to claim 6, wherein The event image is a binary image, in which the pixel value of the pixel that has generated the event signal is set to 1 and the pixel value of the pixel that has not generated the event signal is set to 0.

8. The information processing apparatus according to claim 1, wherein The predetermined integration time is shorter than one frame period corresponding to the frame rate.

9. The information processing apparatus according to claim 6, wherein The processor also generates a display image in which the pixel value of the pixel where the first event occurred is set to a first brightness value, the pixel value of the pixel where the second event occurred is set to a second brightness value, and the pixel values ​​of all pixels other than the pixel where the first event occurred and the pixel where the second event occurred are set to a third brightness value.

10. The information processing apparatus according to claim 6, wherein The processor calculates the trajectory information of the droplet based on the first event image, and generates the control information based on the calculated trajectory information.

11. The information processing apparatus according to claim 6, wherein The processor performs noise removal processing to remove noise from the first event image.

12. The information processing apparatus according to claim 1, wherein The processor generates a reconstructed image based on the estimated brightness values ​​of the event signal, in units corresponding to each frame rate period.

13. The information processing apparatus according to claim 12, wherein The processor generates the reconstructed image by using all the event signals from the past time.

14. The information processing apparatus according to claim 12, wherein The processor calculates the width of the droplet based on the reconstructed image, and generates control information based on the calculated width for controlling the ejection of the droplet through the dispenser.

15. The information processing apparatus according to claim 12, wherein The processor calculates the volume of the droplet based on the reconstructed image, and generates control information based on the calculated volume for controlling the ejection of the droplet through the dispenser.

16. The information processing apparatus according to claim 12, wherein The processor performs noise removal processing to set the pixel values ​​of pixels that did not experience any events during a specific time period prior to the generation timing of the reconstructed image to zero, wherein, The specific time period is set to be longer than the passage time of the object and shorter than the ejection interval of the droplet, and the passage time of the object is the time it takes for the object to travel a distance in the direction of movement of the droplet.

17. The information processing apparatus according to claim 1, wherein The processor also detects the droplets based on RGB images obtained by capturing images of the droplets using an image sensor.

18. An information processing system, comprising: The dispenser is configured to spray a predetermined liquid; An event sensor includes pixels configured to photoelectrically convert light signals and output pixel signals, and the event sensor is configured to output a time-varying brightness change of the light signal based on the pixel signals as an event signal; as well as The processor is configured to detect droplets ejected from the dispenser based on the event signal; The processor further generates and outputs control information for controlling the ejection of the droplets through the dispenser; The processor generates an event image by collecting event signals from the event sensor in units of predetermined integration time, and generates the control information based on the generated event image.

Images