Imaging device

Abstract

[Problem] To address the problem that, when a moving body is extracted from a captured image, a huge amount of image information outputted from an imaging device must be processed, resulting in enlargement of a system and increase of power consumption. [Solution] Provided is an imaging device which generates a difference signal of pixel signals different in image-capturing time, makes adjustment so as to prevent generation of the difference signal in a stationary object, determines the contour end of the moving body on the basis of the difference signal, generates a binarized masking signal, takes a logical product from a pixel signal output which is a current image-capturing signal and the masking signal, and outputs only an image of a moving object for each reading line.

Description

Imaging device

[0001] The present invention relates to a method for acquiring motion information, and more particularly to an imaging device that eliminates background still image information based on contour information of a moving object and acquires and displays only the moving object. Here, the imaging device refers to a solid-state imaging device or an imaging camera using such a device.

[0002] In imaging devices such as solid-state imaging devices and imaging cameras that use such devices, there is a motion vector display method that displays the direction and speed of moving objects (hereinafter referred to as moving objects) in an image. A block matching method is known as a conventional method for obtaining such motion vectors.

[0003] Patent Document 1 applies this block matching technique to track a moving object, and Patent Document 2 discloses a technique for extracting feature points on the contour curve of a moving object and estimating a motion vector using the extracted feature points.

[0004] Furthermore, Patent Document 3 discloses a method for extracting feature points representing predetermined characteristics from an image of one frame captured by a camera, extracting feature points from a region adjacent to the feature points in an image of a temporally subsequent frame, and extracting a motion vector from both feature points with a small amount of calculation.

[0005] Furthermore, a common method for detecting the movement of a subject on the image capture screen of a solid-state imaging device is to compare signal outputs at the same pixel level between previous and next frames, but there is also a method disclosed in Patent Document 4 in which adjacent pixels are grouped into pairs, the integration time is changed, the gain is adjusted, and the movement is detected based on the differential signal output. Patent Document 5 also discloses a method in which adjacent pixels with different sensitivities are grouped into pairs, the integration time is changed, and the movement is detected based on the differential signal output. The former method of comparing frames is called the inter-frame difference method, and the latter method of comparing pixel pairs is called the inter-pixel difference method.

[0006] The latter patent document 5 discloses that a differential signal for determining motion detection is directly output, and only contour information of a moving object (hereinafter referred to as a moving object) is output, thereby eliminating the process of extracting the contour of the moving object from an image, and thus enabling application to a human-machine interface. For this contour information of a moving object, differences in the amount of photoelectrically converted charge occur in pixels according to changes in motion, and these differences are output as contour information corresponding to the pixel position.

[0007] Patent Document 6 describes a concept of reducing the power consumption of a system by controlling the ON / OFF of the power supply of a downstream system based on the results of motion detection, and a concept of reducing the amount of information by controlling the image signal output from a solid-state imaging device.The motion detection methods in Patent Documents 2 and 3 describe cases where motion detection is possible using pixel differences without using a frame memory.

[0008] Patent Document 7 discloses a method for acquiring motion vectors that significantly reduces the load of processing image information in subsequent circuits. The method combines a difference circuit, a differentiation circuit, a logical product, and a logical sum to acquire and output motion vector information based on contour information of a moving object obtained from a solid-state imaging device.

[0009] Patent No. 3037383, Patent No. 3674084, Patent No. 6624841, Patent No. 3521109, Patent No. 5646421, Patent No. 5604700, Patent No. 7025073

[0010] In the conventional motion vector acquisition methods shown in Patent Documents 1, 2, and 3, the amount of image information to be processed is large, which results in a large calculation scale and increased power consumption. Furthermore, Patent Documents 5 and 6 make it possible to easily acquire motion contour information, but do not disclose a method for acquiring or outputting motion vectors.

[0011] In Patent Document 7, the load of processing image information in a downstream circuit for obtaining motion vectors is significantly reduced, but moving objects are displayed using motion vectors that are mainly based on outline information, and information about the moving objects themselves is missing.

[0012] The object of the present invention is to significantly reduce the load of image processing performed by subsequent circuits when acquiring only image information of a moving object by removing background still image information. The method involves acquiring masking information based on contour information of the moving object obtained from an imaging device, and acquiring and outputting moving object information in a simple manner.

[0013] To solve the above problem, we provide an imaging device that acquires contour information of a moving object from a solid-state imaging device or an imaging camera, deletes background still image information to reduce the amount of information, then acquires masking information of the moving object from the position and size of the moving object based on this contour information, and takes the logical product (hereinafter referred to as AND) of the masking information and image information for each line, thereby outputting information of only the moving object from the image information.

[0014] In this way, the present invention does not handle still images among the captured images, but only moving objects. Furthermore, by displaying only the images of moving objects and eliminating the still images in the background, the amount of information is significantly reduced. This has the advantage of making it easier to visually grasp the movement information.

[0015] The proposed imaging device (solid-state imaging device, imaging camera) outputs only image information of moving objects, which significantly reduces the amount of information, and makes it possible to provide an imaging device (moving object output sensor, moving object output camera) that outputs only moving objects without having to perform real-time calculations on the vast amount of image information output from normal camera images.

[0016] The present invention provides an imaging device that acquires masking information for moving objects from contour information and outputs information on only the moving objects from image information by performing an AND operation on the masking information and image information line by line. The masking information referred to here refers to a masking waveform for extracting only the moving objects. To retain the moving objects using a logical AND, the masking information is binarized, with a waveform that corresponds to a 1 when outputting information on only the moving objects and a 0 when outputting a still image of the background. Solid-state imaging devices generally include area sensors with two-dimensionally arranged pixels and linear sensors with one-dimensionally arranged pixels. By performing an AND operation line by line, both linear and area sensors can be used. Furthermore, when memory is used to acquire contour information on moving objects, line memory and area memory are generally used, corresponding to linear and area sensors, respectively. Since the term frame memory is more commonly used than area memory, the term frame memory will be used in the following description. Linear sensors are also sometimes called line sensors, but the term linear sensor will be used here. Another method for obtaining contour information of a moving object is to obtain it from pixel differences between pixel pairs with different exposure times without using a memory.

[0017] According to the present invention, background still images are deleted during the process of acquiring moving contour information, significantly reducing the amount of information to be calculated. Furthermore, by ANDing a masking pattern created from contour information with the original image and outputting only moving objects, the amount of information from the imaging device can be reduced. Furthermore, by performing calculations to extract only moving objects line by line along the scan line, and performing this in real time in parallel with the pixel output for each line, a significant reduction in the calculation load can be achieved.

[0018] Conventional imaging cameras have a large amount of image information because they undergo various processing steps in post-processing circuits, but the present invention limits the amount of information output to moving objects only, reducing the computational load and enabling a significant reduction in the amount of information output from the imaging device.

[0019] 1 is a block diagram showing a configuration using an imaging camera as an imaging device according to a first embodiment of the present invention. FIG. 1 is a block diagram showing the schematic configuration of a motion contour extraction unit using a linear sensor according to a first embodiment-1. FIG. 2 shows the position of a customer's hand captured at each timing of the linear sensor camera. FIG. 3 shows pixel signal output from a linear sensor and a reconstructed image of a customer's hand at each timing of FIG. 3. FIG. 4 shows the pixel signal output of FIG. 2 and a difference mechanism using a line memory, and motion contour information, at the timings excerpted from FIG. 4. FIG. 5 shows the actual pixel signal output of a linear sensor with background light, at each timing of FIG. 4. FIG. 6 shows the pixel signal output of FIG. 2 and a difference mechanism using a line memory, and motion contour information, at the timings excerpted from FIG. 6. FIG. 7 shows the motion contour information acquired by the motion contour extraction unit of FIG. 2 at each timing of FIG. 6. FIG. 8 is a block diagram showing the schematic configuration of a masking extraction unit using a linear sensor according to a first embodiment-1. FIG. 9 shows the masking signal acquired by the masking extraction unit of FIG. 9 at the timings excerpted from FIG. 7. FIG. 10 is a block diagram showing the schematic configuration of a masking circuit using a linear sensor according to a first embodiment-1. 17 is an explanatory diagram of a masking pixel signal acquired by the masking circuit of FIG. 11 at selected timings. FIG. 18 is an explanatory diagram of a masking signal acquired by the masking extraction unit of FIG. 9 at each timing. FIG. 19 is an explanatory diagram of a masking pixel signal acquired by the masking circuit of FIG. 11 at each timing. FIG. 20 is an explanatory diagram of a masking pixel signal at each timing under different criteria for determining the edge of a motion contour. FIG. 21 is a block diagram showing the schematic configuration of a motion contour extraction unit using an area sensor according to a first embodiment-2. FIG. 22 shows the position of a customer's hand captured by each scanning line of the area sensor camera of FIG. 16. FIG. 23 shows the scanning line pixel signal output of the area sensor and an image of the customer's hand, which is a moving object, according to FIG. 27. FIG. 24 is an explanatory diagram of each scanning line pixel signal output of the area sensor, a mechanism for difference using a frame memory, and motion contour information. FIG. 25 is an explanatory diagram of motion contour information, masking signals, and masking pixel signals for each scanning line. FIG. 26 is a block diagram of another motion contour extraction method using a linear sensor according to a first embodiment-3. FIG. 27 is an explanatory diagram of motion contour information, masking signals, and masking pixel signals using FIG. 21 at selected timings. 21 is an explanatory diagram of motion contour information and masking signals at each timing using FIG. 21. FIG. 22 is a block diagram of another motion contour extraction method using an area sensor as the first embodiment-4. FIG. 23 is a diagram showing the position of a customer's hand captured by each scanning line of the area sensor camera in FIG. 24.29. Scanning line pixel signal output of the area sensor of FIG. 24 when there is no customer's hand (moving object). An explanatory diagram of motion contour information, masking signals, and masking pixel signals using FIG. 24 for selected scanning lines. An explanatory diagram of motion contour information and masking signals using FIG. 24 for each scanning line. A block diagram of another motion contour extraction method using a special area sensor as the first embodiment-5. A diagram explaining the pixel arrangement and pixel output characteristics of the area sensor of FIG. 29. A diagram explaining another pixel arrangement and pixel output characteristics of the area sensor of FIG. 29. A block diagram showing a configuration using a solid-state imaging device as an imaging device of the second embodiment of the present invention. A block diagram of a solid-state imaging device using a special area sensor as the second embodiment-5. A block diagram of another solid-state imaging device using the special area sensor of FIG. 33.

[0020] Several selectable cases are possible for embodiments of the present invention. Examples of such combinations include: 1) an imaging camera or a solid-state imaging device is used as the imaging device; 2) an area sensor or a linear sensor is used as the solid-state imaging device; and 3) a method for extracting the contours of a moving object is performed by difference detection using memory (frame memory or line memory), or by difference detection between pixel pairs with different exposure times. Since imaging devices according to embodiments of the present invention are configured using a combination of these cases, the following description focuses on representative combinations and refers to the drawings, with other combinations being explained in more detail.

[0021] In the following description, the first embodiment uses the case of an imaging camera as an imaging device that outputs only moving object information, and describes the functions of each block constituting the imaging device of the present invention. In the case of a solid-state imaging device of the second embodiment, these blocks correspond to a case mounted on the solid-state imaging device, so the description of the functions of each block is the same.

[0022] First Embodiment FIG. 1 illustrates the blocks of an imaging device according to a first embodiment of the present invention, using an imaging camera 1 as an example. The block configuration of the imaging camera 1 and the signals of each block are described below. A lens 2 constitutes an optical system that forms a subject image on a solid-state imaging device 4 in a camera housing 3. A pixel signal output 5 output from the solid-state imaging device 4 is split into two, one input to a motion contour extraction unit 6 and the other input to a masking circuit 10. The motion contour extraction unit 6 extracts motion contour information 7 from the pixel signal output 5. The extracted motion contour information 7 is input to a masking extraction unit 8. The masking extraction unit 8 uses the motion contour information 7 to determine the motion contour edge corresponding to the outermost periphery of a moving object, and creates and outputs a masking signal 9 that is used to output only moving objects. The masking signal 9 is input to the masking circuit 10, where, together with the pixel signal output 5 also input, a masking pixel signal output 11 is created and output. This masking pixel signal output is an output of only pixels corresponding to an image of a moving object, and is an imaging device that outputs only moving object information, which is the object of the present invention.

[0023] A typical method for outputting only moving object information involves using an area sensor as the solid-state imaging device 4, capturing the image in a frame memory, processing the image frame by frame, detecting moving objects, identifying the areas of movement, and then outputting only the moving object information. This has the drawback of requiring a complex calculation system to process a huge amount of image information frame by frame, and performing high-speed calculations in real time, resulting in enormous power consumption. The greatest feature of the present invention is that the pixel signal output 5 from the solid-state imaging device 4 is processed for each readout line, and a masking pixel signal output 11 is generated in real time.

[0024] The masking circuit 10 receives the masking signal 9 and the pixel signal output 5, and generates a masking pixel signal output 11. As will be described later, the masking circuit 10 is an AND circuit that obtains the AND (logical product) of the masking signal 9 and the pixel signal output 5, and outputs the masking pixel signal output 11, which is an output of only pixels corresponding to images of moving objects, in real time from the imaging camera 1. In Fig. 1, to emphasize this real time, the pixel signal output 5 that is output from the solid-state imaging device 4 and input to the masking circuit 10 is indicated by a thick line. Each block that constitutes the imaging camera 1 of the first embodiment of the present invention shown in Fig. 1 will be described below in order with reference to the drawings.

[0025] First Embodiment-1 First, the configuration of the motion contour extraction unit 6 of the imaging camera 1 in Figure 1 will be described with reference to Figure 2. The location where an optical system that forms a subject image on the solid-state imaging device 4 in the camera housing 3 using the lens 2 is configured is the same as in Figure 1. First, for simplicity's sake, a linear sensor 4' will be used as the solid-state imaging device 4 (this will be referred to as First Embodiment-1). For this reason, the lens 2 and camera housing 3 will be referred to as a linear sensor camera 1'. In Figure 1, the lens 2 and camera housing 3, together with the motion contour extraction unit 6, masking extraction unit 8, and masking circuit 10, are included in the imaging camera 1, but for convenience, the lens 2 and camera housing 3 with the linear sensor 4' incorporated therein will be referred to as a linear sensor camera 1'. The same is true for the area sensor camera 1" in Figure 16. The pixel signal output 5 from the linear sensor 4' is divided into two, one input to a masking circuit 10 and the other input to a motion contour extraction unit 6. In the motion contour extraction unit 6, the pixel signal output 5 is further divided into two and taken into a line memory 12. The pixel signal output 5 coming out of the line memory 12 is the line output of the previous line, and a difference between this and the current pixel signal output 5 is taken in a difference circuit 13, and motion contour information 7 is obtained through an absolute value circuit 14. The time corresponding to one line is the cycle time of the linear sensor 4', and will be represented as Δt in the following drawings. In other words, the difference between adjacent lines is taken in the difference circuit 13. The cycle time here is the time required for one period to read out all of the pixels of the solid-state image pickup device 4, and is the time required for the solid-state image pickup device 4 to read out one line with the linear sensor 4'.

[0026] The pixel signal output 5 represents a signal output corresponding to one line of the linear sensor, which is read out from all pixels of the linear sensor 4' and has idle feed before and after, and is synchronized for each line. The pixel difference referred to in the following explanation refers to the difference synchronized between these lines, and refers to the difference between pixels with different exposure timings that image the same position on a stationary subject. Generally, the cycle time of the linear sensor 4' includes the exposure time for photoelectric conversion at the pixels, and the exposure time is determined by the time difference between the start and end of photoelectric conversion at the pixels. The set of the start and end times of this photoelectric conversion is called the shooting time. This is also true for the area sensor, which will be described later.

[0027] <Moving object recognition method in first embodiment-1> Before explaining why the output from the absolute value circuit 14 is called motion contour information 7, it is necessary to explain how a moving object is recognized using the linear sensor camera 1'. As an example of moving object recognition using the linear sensor 4', a case will be explained in which the linear sensor camera 1' monitors the front part of a product display shelf and recognizes the hand movements of a customer performing tasks associated with the ingress and egress of products (picking up, returning).

[0028] FIG. 3 shows a case in which a linear sensor camera 1' installed on the upper edge of a product display shelf 15 monitors the state of a customer's hand 16 reaching out to grab an item displayed on the product display shelf 15. The solid-state imaging device 4 used in this linear sensor camera 1' is a linear sensor 4', which observes the front portion of the product display shelf 15. On the front of the product display shelf 15, a customer's hand 16 is shown performing an action associated with placing or removing an item. Because the pixels of the linear sensor 4' are arranged in a straight line, the range that can be captured is limited to the plane in front of the product display shelf 1. When explaining an application system as in FIG. 3, the linear sensor camera 1' is used to mean an imaging camera 1 that uses a linear sensor 4'. The same applies to the area sensor camera 4'' in FIG. 17.

[0029] This plane is called t = T ０ The timing is indicated by a dashed line extending from the linear sensor camera 1′. ０At the timing t=T, the customer's hand 16 has not yet reached this plane. The customer's hand 16 approaches to pick up a product (not shown) from the product display shelf 15. For the sake of explanation, we will determine which part of the customer's hand 16 crosses the dashed line extending from the linear sensor camera 1' (the part that can be photographed by the linear sensor camera 1') at the timing t=T. ０ , t=T １ , t=T ２ , t=T ３ , t=T ４ Shown in.

[0030] In FIG. 3, the part of the customer's hand 16 that can be photographed by the linear sensor camera 1′ is determined at timing t=T ０ , t=T １ , t=T ２ , t=T ３ , t=T ４ The pixel output signals 5 from the linear sensor 4' at each timing are shown in Figures 4(a), (b), (c), (d), and (e). Here, the pixel signal output 5 is the output of the pixel row of the linear sensor 4', with the horizontal axis representing the pixel position and the vertical axis representing the pixel signal output at each pixel position. For reference, the reproduced image is indicated to which part of the customer's hand 16 the image corresponds.

[0031] A feature of the linear sensor 4' is that when there is no movement, the output of the background remains constant over time, and for convenience, this is set to zero (see FIG. 4(a)). When there is movement (here, movement of the customer's hand 16), the output of the linear sensor 4' changes. Corresponding to the movement of the hand, t=T ０ The image is captured on the plane indicated by the dashed line extending from the linear sensor camera 4 at the timing shown in FIG. 4, and changes occur in the pixel output waveform corresponding to the part of the customer's hand 16 that crosses this plane (see FIGS. 4(b), (c), (d), and (e)).

[0032] A reconstructed image of the customer's hand 16 can be obtained by arranging the pixel signal outputs of the linear sensor 4' in chronological order. Figure 4 shows a reconstructed image obtained from pixel output waveforms corresponding to the parts of the customer's hand 16. The reconstructed image expands and contracts depending on the speed of the movement of the customer's hand 16, but because the aspect ratio (length to width) of a human hand is approximately constant, correction of the reconstructed image can be easily performed. Furthermore, even if the customer's hand 16 grabs a product from the product display shelf 15 and removes it, i.e., moves in a direction away from the product display shelf 15, a reconstructed image can easily be obtained. This concludes the explanation of how a moving object is recognized using the linear sensor camera 1'.

[0033] <Motion contour extraction method in first embodiment-1 (case where there is no background light)> Figure 2 shows a signal processing circuit that processes pixel signal output 5 from linear sensor camera 1' to extract only the area corresponding to the customer's hand 16 and set the background output to zero. The signal processing circuit shown in Figure 2 forms an image of the subject on linear sensor 4' inside camera housing 3 using lens 2, and outputs the obtained pixel signal output 5 from linear sensor camera 1'. From the output pixel signal output 5, motion contour information 7 is obtained in motion contour extraction unit 6.

[0034] The pixel signal output 5 is taken into a line memory 12. The pixel signal output 5 coming out of the line memory 12 is the line output of the previous line, and a difference circuit 13 takes the difference between this and the current pixel signal output 5, and the difference is passed to an absolute value circuit 14 to obtain moving contour information 7. The time corresponding to one line is the cycle time Δt of the linear sensor 4' (the time required to read one line). In other words, the difference circuit 13 takes the difference between adjacent lines.

[0035] Of course, the line memory 12 does not have to be a line memory equivalent to a one-line delay, and a delay equivalent to two lines may be generated, and the difference between every other line may be taken. The amount of line delay is determined by the balance between the speed at which the customer's hand 16 is inserted and removed and the cycle time of the linear sensor 4'. For system simplicity, the delay time of the line memory is generally an integral multiple of the cycle time, but the following explanation will be given for the case of a difference between adjacent lines equivalent to a one-line delay (Δt).

[0036] For example, in the case of a difference between lines every other line, which corresponds to a two-line delay (2Δt), the line memory 12 in FIG. 2 requires a capacity equivalent to two lines to delay two lines. However, in general, when a two-line delay is required in a line memory, the speed of the moving object is slow (in this case, the customer's hand 16 moves in and out slowly), so the timing of monitoring the image can be slow. In other words, an image every other line can be used, and by adding a mechanism (not shown) that discards the data of the image signal output 5 of the line corresponding to every other line, the line memory 12 only needs to store the pixel signal output of two lines before, and only requires a capacity equivalent to one line. This is also true for a delay of three or more lines and the frame memory 22 in FIG. 16.

[0037] The line difference method shown in Fig. 2 will be explained using Fig. 5 based on each signal waveform. For simplicity, the timing of the position of the customer's hand 16 will be first assumed to be t=T shown in Figs. 4(a), (b), and (d), which is the case where there is no background light. ０ , t=T １ , t=T ３ First, let us consider the case where t=T ０ At this timing, as shown in FIG. 5(a), there is no output of the customer's hand 16 in the waveform of the pixel signal output 5 of the linear sensor camera 1'. ０-Δｔ 5(a'), there is no output from the customer's hand 16. The difference between the two is taken by the difference circuit 13, but it is zero, and even after passing through the absolute value circuit 14, the motion contour information 7 remains zero as shown in FIG. 5(a'').

[0038] Next, t=T shown in FIG. １ At this timing, as shown in FIG. 5B, the output of the customer's hand 16 appears in the waveform of the pixel signal output 5. At this time, the previous line (T ０-Δｔ) also shows the output of the customer's hand 16 as shown in Figure 5(b'). The two waveforms differ slightly depending on the insertion position of the customer's hand 16, and differences generally appear in the outline portion of the hand output. For this reason, when the difference between the two is taken in the difference circuit 13, an output similar to a differentiated waveform appears in this outline portion. After passing through the absolute value circuit 14, the motion outline information 7 becomes as shown in Figure 5(b''). This has a shape similar to the differentiated waveform of the pixel signal output 5.

[0039] The signal output from the absolute value circuit 14, which would normally be called the pixel difference absolute value, is called motion contour information 7 because of the calculation method used to synchronize and calculate the difference between the pixel signal output of the previous line and the pixel signal output at the current time. In other words, the signal takes on a shape similar to a differential waveform, and since a stationary subject has the same pixel signal output waveform, the difference becomes zero when the difference is calculated. On the other hand, a moving subject has a different pixel signal output waveform, so a signal remains when the difference is calculated. This remaining signal is often found at the contours (periphery) of moving objects, and is therefore called motion contour information 7. Even if the center of a moving object (for example, the back of the customer's hand 16) moves, the same pattern of change occurs, and the signal obtained by difference calculation is small, making it difficult to determine that the object is moving.

[0040] In a similar sequence, the customer's hand 16 moves to t=T in FIG. ３ At this timing, the output of the entire customer's hand 16 appears as shown in FIG. 5(d). At this time, the previous line (T ３-Δｔ ) are the outputs of the entire customer's hand 16, as shown in FIG. 15(d'), and the two waveforms are subtly different. When the difference between the two is taken by the difference circuit 13, in this case, not only the contours (periphery) but also the areas between the fingers (finger contours) appear subtly. After passing through the absolute value circuit 14, the motion contour information 7 becomes as shown in FIG. 5(d'').

[0041] The motion contour information 7 shown in FIG. 5(d") has a complex shape, but this is because the moving object is a hand, which is a complex moving object with gaps between the fingers, and the motion contour information 7 is also complex. When it comes to the fingertip in FIG. 4(b) or the wrist in FIG. 4(e), the motion contour information 7 becomes simpler.

[0042] <Motion Contour Extraction Method in First Embodiment-1 (Case with Background Light)> For simplicity, Figures 4 and 5 have been described assuming a case where there is no background light, but in reality, the background light from the indoor illumination never becomes zero, and the pixel signal output waveform of the background light from the linear sensor camera 1' will actually be as shown in Figure 6(a). However, the background is stationary, and the pattern is considered to be constant over the short period of time in which the movement of the customer's hand is captured. Therefore, the actual pixel output signal from the linear sensor camera 1' at each timing corresponding to Figures 4(a) to (e) will be like the output waveforms shown in Figures 6(a) to (e).

[0043] It should be noted here that the magnitude of the pixel signal output at the position corresponding to the customer's hand 16 is not superimposed on the output waveform of the stationary background, but rather the pixel signal output of the background at the hand position is blocked by the customer's hand 16, the pixel signal output of the background corresponding to the hand becomes zero, and the pixel signal output corresponding to the part of the customer's hand 16 appears. This situation is represented by dotted lines in Figures 6(b) to (e).

[0044] Next, in the case where there is background light, t=T shown in Figs. 6(b) and 6(d) corresponds to Figs. 5(b) and 5(d). １ , t=T ３ The timing of t=T will be explained with reference to FIG. １ At this timing, as shown in FIG. 7B, the output of the customer's hand 16 appears in the waveform of the pixel signal output 5. At this time, the previous line (T １-Δｔ 7(b'), the output of the customer's hand 16 also appears in the waveform of the pixel signal output of FIG. 7(b), and the two waveforms are slightly different. For this reason, when the difference between the two is taken in the difference circuit 13, the background light area disappears because it is the same, and an output like a differentiated waveform appears in the outline of the hand, and after passing through the absolute value circuit 14, the motion outline information 7, which corresponds to the absolute value of the pixel difference, becomes as shown in FIG. 7(b''). This is similar to the case where there is no background light, as shown in FIG. 5(b'').

[0045] In a similar sequence, t=T ３ At the timing of , the whole hand appears as shown in FIG. 7(d), and the previous line (T ３-Δｔ) are outputs of the entire hand, as shown in Figure 7(d'), and the two are subtly different. When the difference between the two is taken using the difference circuit 13, the areas between the fingers as well as the outline appear subtly, and the motion outline information 7, which corresponds to the absolute value of the pixel difference, is as shown in Figure 7(d''). This is also similar to the case where there is no background light in Figure 5(d'').

[0046] The above is the timing t = T １ , t=T ３ However, other timings are also included in the timing t=T shown in FIG. ０ , t=T １ , t=T ２ , t=T ３ , t=T ４ The waveforms of the pixel signal output 5 shown in FIGS. 6(a), (b), (c), (d), and (e) correspond to the waveforms of the motion contour information 7, which are shown in FIGS. 8(a), (b), (c), (d), and (e).

[0047] <Masking Extraction Method in First Embodiment-1> Fig. 9 explains the masking extraction unit 8, which is part of a signal processing circuit that processes signals from the output of the linear sensor camera 1' of the imaging camera 1 according to the first embodiment-1 of the present invention, extracts only the area corresponding to the customer's hand 16, and sets the background output to zero. The masking extraction unit 8 extracts a masking signal 9 from the motion contour information 7. The motion contour extraction unit 6 has been explained separately for the cases with and without background light, but the motion contour information 7 from the motion contour extraction unit 6 has similar waveforms in both cases, as shown in Figs. 5(a"), (b"), and (d") and Figs. 7(a"), (b"), and (d"), and therefore the masking extraction unit 8 will be explained without making any distinction.

[0048] The motion contour extraction unit 6 obtains motion contour information 7 by calculating the difference between the timing at which the movement of the customer's hand 16 occurs and the timing one line before. Based on this motion contour information 7, the masking extraction unit 8 generates a masking signal 9 for extracting the timing at which the movement of the customer's hand 16 occurs (which means both the time at which the movement occurs and the output timing of pixels corresponding to the location of the movement) and the pixel output timing corresponding to the movement.

[0049] Next, the configuration of the masking extraction unit 8 will be described. From the motion contour information 7, the motion contour end determination unit 17 extracts the outer contour portion 7' ​​of the customer's hand 16. As explained in FIG. 7(d"), the difference circuit 13 detects not only the contour portion of the hand but also the area between the fingers, and the motion contour information 7 has a complex waveform. The motion contour end determination unit 17 can extract the outer contour portion of the customer's hand 16 by tracking the progress of the waveform of the motion contour information 7.

[0050] The method will be explained below, assuming that the customer's hand 16 can be captured as a series of movements from the time it enters the plane in front of the product display shelf 15 monitored by the linear sensor camera 1' until it leaves. This assumption holds true because, in general, a customer's hand 16 performs a series of movements to insert, grab, and remove a product (not shown) from the product display shelf 15 (because the customer will not stop inside the display shelf while grasping a product, and will not insert anything other than their hand (such as their head or foot) into the display shelf).

[0051] The linear sensor camera 1' captures the customer's hand 16 from the tip, gradually expanding to the width of the hand. As shown in FIG. 4(c), the gap between the fingers also appears, but it remains within the width of the hand, allowing the outer contour of the hand to be traced. As shown in FIG. 4(d), a gap appears between the thumb and the hand, making it indistinguishable from a new moving object. However, as shown in FIG. 4(e), the thumb and hand are connected and can be recognized as a single object. Because the size of the customer's hand 16 is small compared to the size of the product display shelf 15 observed by the linear sensor camera 1', establishing a criterion that a new moving object (fingertip) occurring near the hand belongs to the hand makes it easy to determine the outermost contour of the hand (movement contour end determination). The movement contour end information 18 corresponding to the outer contour of the customer's hand 16 obtained by the movement contour end determination unit 17 is compared with a threshold value 19 in the masking signal generation circuit 20, binarized, and a masking signal 9 is obtained, which is then output from the masking extraction unit 8.

[0052] The signal processing in the masking extraction unit 8 shown in Fig. 9 will be explained based on each signal waveform with reference to Fig. 10. The timing of the position of the customer's hand 16 is t=T １ , t=T ３This corresponds to the motion contour information 7. Here, a case where the motion contour information 7 is created with background light will be described.

[0053] Customer's hand 16 is t=T １ At this timing, the motion contour information 7 shown in FIG. 10(b) is obtained. The motion contour end information 18 at this time is as shown in FIG. 10(b'). The motion contour end information 18 is the end of the contour of one moving object, and each pair of information is a pair, and the area of ​​one moving object is determined by the pair of information. The motion contour end information 18 is compared with a threshold value 19 in a masking signal generating circuit 20, and a masking area is set and binarized. This masking area is the area of ​​one moving object. If the motion contour end information 18 exceeds the threshold value 19 and is recognized as an area to be masked, a value of 1 is assigned, and otherwise a value of 0 is assigned, and a binarized masking signal 9 is obtained. The masking signal 9 obtained based on the motion contour end information 18 of FIG. 10(b') is as shown in FIG. 10(b''').

[0054] The period during which this masking signal 9 is 1 corresponds to the timing when the movement of the customer's hand 16 occurs and the pixel output timing corresponding to the position of the customer's hand 16, and is the period during which masking should be performed from the pixel signal output 5. In other words, the pixel signal output 5 during the timing when the masking signal 9 is generated (is 1) corresponds to observing only the movement of the customer's hand 16.

[0055] In a similar sequence, the customer's hand 16 is at t=T ３ At timing (d), the motion contour information 7 of FIG. 10(d) is obtained, just like FIG. 8(d), and the motion contour end information 18 is deleted as the portion between the fingers is determined not to be an end, resulting in the information shown in FIG. 10(d'). At this time, the gap between the thumb and the other fingers has not yet been determined to be an inter-finger gap, and the thumb is determined to be the appearance of a new moving object. At the timing of the base of the thumb, both are determined to be the same moving object. The masking signal 9 obtained based on the motion contour end information 18 of FIG. 10(d') is shown in FIG. 10(d''').

[0056] Here, the gaps between the fingers are determined based on the initial edge information determined from the fingertips at the tip of the customer's hand 16, and any new moving object that occurs near this edge, within a distance equivalent to the thickness of the finger, is defined as another fingertip and recognized as one hand. If this is expanded further and a criterion is established that any new moving object (fingertip) that occurs near a distance equivalent to the width of the hand belongs to the hand, then even the thumb, which was determined to be a new moving object in Figure 10 (d'''), will not be determined to be a new moving object, making it easier to determine the outermost contour of the hand. This is the criterion mentioned in paragraph 51.

[0057] <Masking Method in First Embodiment-1> The signal processing circuit performs signal processing on the output of the linear sensor camera 1' of the imaging camera 1 according to the first embodiment-1 of the present invention, extracts only the area corresponding to the customer's hand 16, and zeros the background output. The final masking signal 9 is used in the signal processing circuit to zero out the background and obtain a masking pixel signal output 11 that outputs only the area corresponding to the customer's hand 16 from the pixel signal output 5. The masking circuit will be described with reference to Figure 11. Because the motion contour information 7 has the same waveform in both cases, there is no background light and there is, the description will be omitted. The period during which the binarized masking signal 9 is 1 corresponds to the timing of the movement of the customer's hand 16 and the pixel output timing corresponding to the position of the customer's hand 16, and is the period that should be masked from the pixel signal output 5.

[0058] In order for the masking circuit 10 to extract the timing of the movement of the customer's hand 16 and the corresponding pixel output timing, it is sufficient to output the pixel signal output 5 from the linear sensor camera 1' only when the binarized masking signal 9 is 1. When the masking signal is 0, the pixel signal output 5 is not output. The masking circuit 10 that achieves this simply takes the AND of the two and is composed of a simple AND circuit. The masking circuit 10 receives two inputs: the masking signal 9 from the masking extraction unit 8 and the pixel signal output 5 from the solid-state imaging device 4, and outputs a masking pixel signal output 11. In this example, the background disappears and only the area corresponding to the customer's hand 16 is output.

[0059] In this case, if a delay occurs due to the calculation processing time between the motion contour extraction unit 6 and the masking extraction unit 8, causing problems in extracting only the movement of the customer's hand 16 from the pixel signal output 5, the pixel signal output 5 can be delayed by a time equivalent to this calculation processing time using a delay circuit 21 to achieve synchronization (timing adjustment) and obtain the masking pixel signal output 11 from the masking circuit 10. This delay circuit 21 does not need to be inserted if it is small enough that problems due to delays in the calculation processing time do not occur.

[0060] The signal processing in the masking circuit 10 shown in Fig. 11 will be explained based on each signal waveform with reference to Fig. 12. The timing of the position of the customer's hand 16 is t=T １ , t=T ３ This corresponds to the pixel signal output 5 and the masking signal 9. Here, the case where the motion contour information 7 is created with background light will be explained. １ The pixel signal output 5 and the masking signal 9 at the timing are shown in FIG. 12(b) and FIG. 12(b'''), respectively, and the masking pixel signal output 11 is shown in FIG. 12(b''''). That is, the pixel signal output 5 at the timing when the masking signal 9 is generated (is 1) corresponds to observing only the movement of the customer's hand 16. The masking pixel signal output 11 shown in FIG. 12(b'''') as １ This is a signal from the customer's hand 16 at the timing.

[0061] In a similar sequence, the customer's hand 16 moves to the time t=T in FIG. ３ At this timing t=T, the pixel signal output 5 is the same as that shown in FIG. ３ 10(d'''), the masking signal is shown in FIG. 12(d'''), and the masking pixel output 11 is shown in FIG. 12(d''''). When there is no background as shown in FIG. 4(d), t=T ３ This is a signal from the customer's hand 16 at the timing.

[0062] The above is the timing t = T １ , t=T ３However, other timings are also included, including the timing t=T ０ , t=T １ , t=T ２ , t=T ３ , t=T ４ The waveforms of the pixel signal output 5 shown in FIGS. 6(a), (b), (c), (d), and (e) that correspond to the above-mentioned waveforms are shown in FIGS. 8(a), (b), (c), (d), and (e), and based on this, the waveforms of the motion contour information 7 and the motion contour end information 18 and the masking signal 9 are shown in FIGS. 13(a), (b), (c), (d), and (e).

[0063] The timing t=T shown in FIG. ０ , t=T １ , t=T ２ , t=T ３ , t=T ４ Based on the waveforms of the pixel signal output 5 shown in Figures 6(a), (b), (c), (d), and (e) corresponding to the timing t, and the waveforms of the masking signal 9 shown in Figures 13(a), (b), (c), (d), and (e), the masking pixel signal output 11 obtained by the masking circuit 10 is shown in the right diagrams of Figures 14(a), (b), (c), (d), and (e) at the respective timings. The waveform of the original pixel signal output 5 is also shown in the left diagrams. ０ , t=T １ , t=T ２ , t=T ３ , t=T ４ corresponds to the dashed area on the customer's hand 16.

[0064] The case of paragraph 56, in which a criterion is set that a new moving object (fingertip) occurring in the vicinity of the width of the hand belongs to the hand, will be explained with reference to FIG. 15. ０ , t=T １ , t=T ２ , t=T ３ , t=T ４The waveforms of the pixel signal output 5 shown in Figures 6(a), (b), (c), (d), and (e) that correspond to the motion contour information 7 are shown in Figures 8(a), (b), (c), (d), and (e), and are shown again in the left diagrams of Figures 15(a), (b), (c), (d), and (e). The waveforms of the motion contour end information 18 that are determined to be ends by the motion contour end determination unit 17 based on the new determination criteria from the waveform of the motion contour information 7 are the differentiated waveforms at both ends of several differentiated waveforms in the waveforms of the motion contour information shown on the left side of Figures 15(a), (b), (c), (d), and (e) at each timing.

[0065] The masking signal 9 that is determined to be 1 by the masking extraction unit 8 from the point in time when it exceeds the threshold value 19 based on the waveform of the motion contour end information 18 obtained by the motion contour end determination unit 17 using the new determination criterion is the masking signal waveform shown in the center of Figures 15(a), (b), (c), (d), and (e). The masking signal waveform corresponds to the size of the customer's hand 16.

[0066] The timing t=T shown in FIG. ０ , t=T １ , t=T ２ , t=T ３ , t=T ４ 6(a), (b), (c), (d), and (e) corresponding to the waveforms of the pixel signal output 5 and the masking signal 9 shown in FIGS. 15(a), (b), (c), (d), and (e), the masking pixel signal output 11 obtained by the masking circuit 10 is shown at each timing in the right diagrams of FIGS. 15(a), (b), (c), (d), and (e). Comparing these waveforms with the waveforms of the masking pixel signal output 11 shown in the right diagrams of FIGS. 14(a), (b), (c), (d), and (e), it is clear that at timing t=T ３There is a difference between the right diagram of FIG. 15(d) and the right diagram of FIG. 14(d), which correspond to the above. That is, in the waveform of the masking signal 9 obtained using the new criterion, the area between the fingers is recognized as part of the customer's hand 16 and is adopted as an image, so the masking signal is 1. Therefore, in the masking pixel signal output 11 of FIG. 15(d), the background image appears in the area between the fingers. This is different from the masking pixel signal output 11 shown in FIG. 14(d), which was obtained when the area between the fingers in FIG. 13(d) also had a masking signal of zero.

[0067] <First Embodiment-2> Next, a case will be described in which an area sensor 4" is used for the solid-state image pickup device 4 of the imaging camera 1 of FIG. 1 (this will be referred to as First Embodiment-2). The configuration of the motion contour extraction unit 6 of FIG. 1 is shown in FIG. 16. The location where the lens 2 configures the optical system that forms an image of a subject on the solid-state image pickup device 4 in the camera housing 3 is the same as in FIG. 1. Because an area sensor 4" is used as the solid-state image pickup device 4, the lens 2 and camera housing 3 will be referred to as an area sensor camera 1". The pixel signal output 5 from the area sensor 4" is divided into two, one of which is input to a masking circuit 10 and the other to a motion contour extraction unit 6. In the motion contour extraction unit 6, the pixel signal output 5 is further divided into two and taken into a frame memory 22. The pixel signal output 5 coming out of the frame memory 22 is the scanning line output of the previous frame, and a difference circuit 13 takes the difference from the pixel signal output 5 corresponding to the scanning line output in question of the current frame, and the difference is passed through an absolute value circuit 14 to obtain motion contour information 7. The time corresponding to one frame is the cycle time of the area sensor 4", and is denoted as ΔT in the following drawings. That is, the difference between pixels corresponding to the same location on the subject between adjacent frames is taken in the differential circuit 13. Here, the cycle time is the time required for one period to read out all the pixels of the solid-state imaging device 4, and is the time required for the solid-state imaging device 4 to read out one frame in the area sensor 4". Here, the motion contour extraction unit 6 uses a frame memory 22 instead of the line memory 12 of the first embodiment-1 shown in FIG. 2, but both have the same constituent elements as image memories.

[0068] To facilitate understanding, the pixel signal output 5 from the area sensor 4" will be explained. One image captured by the two-dimensionally arranged pixels in the area sensor 4" is called a frame. The pixel signal output 5 is composed of one frame of signal output obtained by reading out all the pixels of the area sensor 4" and followed by an idle period before and after. One frame of screen is configured by sequentially displaying one line of signals called a scanning line from top to bottom, like a TV screen, to form an area-shaped screen. When this one frame is displayed, synchronization is maintained between the frames, and in the case of a still image, the frame is displayed in the same position each time, so the screen is displayed in a still state. The pixel signal output 5 output from the frame memory 22 is the scanning line output of the previous frame, and when the difference is taken with the pixel signal output 5 corresponding to the corresponding scanning line output of the current frame, since synchronization is maintained between frames, the difference is the difference between pixel outputs at the same pixel position (the same position on the subject) on the line corresponding to the same scanning line position, with a time difference of one frame. Generally, the cycle time of the area sensor 4" includes the exposure time for photoelectric conversion at the pixels, and as with the linear sensor 4', the time difference between the time when photoelectric conversion at the pixels starts and the time when photoelectric conversion ends determines the exposure time. The set of the time when photoelectric conversion starts and ends is called the shooting time. The cycle time of the linear sensor 4' (time required to read one line) is represented as Δt (line period), and the cycle time of the area sensor 4'' (time required to read one frame) is represented as ΔT (frame period).

[0069] Of course, the images stored in the frame memory 22 do not have to be frame images corresponding to a delay of one frame period; a delay equivalent to two frame periods may be generated and the difference taken by skipping one frame. For system simplicity, the delay time of the images stored in the frame memory 22 is generally an integer multiple of the cycle time, but the following explanation will be given using the case of a difference between adjacent frames corresponding to a delay of one frame (ΔT). The amount of this frame delay is determined by the balance between the speed at which the customer's hand 16 is inserted and removed and the cycle time of the area sensor 4". This is the same as the situation with the line memory 12 when a linear sensor 4' is used for the solid-state imaging device 4. Therefore, the situation described in paragraph 36 also applies to the frame memory 22.

[0070] When focusing on one scanning line, whether it is pixel signal output 5 from area sensor 4" or pixel signal output 5 from linear sensor 4', the difference between the pixel outputs being compared differs in whether the time change corresponds to the difference between one frame or the difference between one line (Δt), but the fact remains that the time change at the same location on the subject is being viewed. In other words, the greatest feature of the present invention, in that pixel signal output 5 from solid-state imaging device 4 is processed for each readout line and masking pixel signal output 11 is created in real time, is the same whether it is an area sensor or a linear sensor.

[0071] <Moving object recognition method in first embodiment-2> The following describes how a moving object is recognized using an area sensor 4". As an example of moving object recognition using an area sensor 4", a case will be described in which, as in Figure 3, an area sensor camera 1" that monitors the front part of a product display shelf recognizes the hand movements of a customer performing tasks associated with the ingress and egress of products (picking up, returning).

[0072] FIG. 17 shows a case where an area sensor camera 1" installed at the upper front of the product display shelf 15 monitors the state of a customer's hand 16 reaching out to grab a product displayed on the product display shelf 15. The solid-state imaging device 4 used in this area sensor camera 1" is an area sensor 4", which observes the front part of the product display shelf 15. On the front of the product display shelf 15, a customer's hand 16 is shown performing an operation to take a product in or out. Since the area sensor 4" has pixels arranged in an area, the area that can be photographed is such that the entire surface of the customer's hand in front of the product display shelf 1 can be photographed in one frame. Therefore, in FIG. 3, t=T ０ ~T ４ , but in FIG. 17, the scanning line S of the area sensor 4″ is ０ ~S ４ The pixel signal output on each scanning line is as shown in Figures 18(a) to 18(e) (background light is assumed to be zero for convenience), and the scanning line corresponds to which part of the customer's hand 16, which is a moving image, is shown in the moving image on the right side of Figure 18. The horizontal axis of the scanning line pixel signal output waveforms in Figures 18(a) to 18(e) is the scanning line (S ０ ~S ４) and the vertical axis corresponds to the pixel signal output on each scanning line.

[0073] A feature of the area sensor 4" is that when there is no movement, the output of the background image is constant, but when there is movement (here, movement of the customer's hand 16), the output of the area sensor 4" changes. Corresponding to the movement of the hand, the scanning line pixel signal output changes (see Figures 18(b), (c), (d), and (e)). In this case, the motion contour information 7 obtained by the motion contour extraction unit 6 in Figure 16 will be explained next.

[0074] <Motion Contour Extraction Method in First Embodiment-2 (Case with Background Light)> In reality, the background light in a room never becomes zero, so the case with background light will be described below. The pixel signal output of each scanning line of the area sensor 4" is shown in the left diagrams of Figures 19(a) to (e). The background image is stationary, and during the short period in which the movement of the customer's hand 16 is captured, a constant pattern is considered for each frame. The motion contour information 7 obtained by the motion contour extraction unit 6 of Figure 16 is obtained by taking the difference between the pixel signal output of each scanning line of the area sensor camera 1" shown in the left diagrams of Figures 19(a) to (e) and the previous frame using a difference circuit 13, and then taking the absolute value using an absolute value circuit 14, resulting in a waveform similar to that of Figures 8(a) to (e). The motion contour information 7 has an output waveform like that shown in the right diagrams of Figures 19(a) to (e). Both are differential waveforms, and the time difference is the pixel signal output difference between the line period Δt or the frame period (ΔT). For convenience, the motion contour information shown in the right diagrams of FIGS. 19(a) to 19(e) is set to be the same as the motion contour information of the first embodiment-1 shown in the right diagrams of FIGS. 8(a) to 8(e).

[0075] The magnitude of the pixel signal output at the position corresponding to the customer's hand 16 is not superimposed on the output waveform of the background, but the pixel signal output of the background at the position of the hand is blocked by the customer's hand 16, the pixel signal output of the background corresponding to the hand becomes zero, and the pixel signal output corresponding to the part of the customer's hand 16 appears, just as in the case of linear sensor 4'. This situation is similarly represented by dotted lines in Figures 19(b) to 19(e).

[0076] <Masking extraction method in first embodiment-2> Since the motion contour information 7 from each scanning line pixel signal output of the area sensor camera 1" is similar to the motion contour information 7 of the linear sensor camera 1', the masking extraction unit 8 is the same as that shown in FIG. 9. The explanation of the masking extraction unit 8 for extracting the masking signal 9 from the motion contour information 7 is the same as in the case of the linear sensor camera 1'.

[0077] In paragraph 56, a case where a judgment criterion is established that a new moving object (fingertip) occurring in a vicinity corresponding to the width of a hand belongs to the hand is described with reference to FIG. 20. The right diagrams of FIG. 19(a), (b), (c), (d), and (e) show the motion contour information 7 corresponding to the waveforms of pixel signal outputs 5 on each scanning line shown in the left diagrams of FIG. 19(a), (b), (c), (d), and (e), and this is shown again in the left diagrams of FIG. 20(a), (b), (c), (d), and (e). The waveform of motion contour end information 18 determined to be an end by the motion contour end determination unit 17 based on the waveform of motion contour information 7 according to the new judgment criterion is the differential waveform at either end of several differential waveforms in the motion contour information waveforms shown on the left sides of FIG. 20(a), (b), (c), (d), and (e) at each timing.

[0078] The masking signal 9 that is determined to be 1 by the masking extraction unit 8 from the point in time when it exceeds the threshold value 19 based on the waveform of the motion contour end information 18 obtained by the motion contour end determination unit 17 using the new determination criterion is the masking signal waveform shown in the center of Figures 20(a), (b), (c), (d), and (e). The masking signal waveform corresponds to the size of the customer's hand 16.

[0079] <Masking Method in First Embodiment-2> The signal processing circuit processes the output of area sensor camera 1" to extract only the area corresponding to customer's hand 16 and zero the background output. The final masking signal 9 is used in the signal processing circuit to zero the background from pixel signal output 5 and obtain masking pixel signal output 11 that outputs only the area corresponding to customer's hand 16. The masking circuit 10 in Figure 11 can be used as is. The case where there is background light will be described.

[0080] The scanning line S of the area sensor 4" of the area sensor camera 1" in FIG. ０ ~S４ The pixel signal output on each scanning line corresponds to the left diagram (scanning line pixel signal output) of Figure 19(a) to (e). The horizontal axis of the scanning line pixel signal output waveform of Figure 19(a) to (e) is the scanning line (S ０ ~S ４ ), and the vertical axis corresponds to the pixel position on each scanning line. From this, motion contour information 7 is extracted via motion contour extraction unit 6 shown in Figure 16, and the masking signal 9 obtained by masking extraction unit 8 is shown in the center of Figures 20(a), (b), (c), (d), and (e), as described above. The period during which the binarized masking signal 9 is 1 corresponds to the timing when the movement of the customer's hand 16 occurs and the pixel output timing corresponding to the position of the customer's hand 16, and is the period that should be masked from the pixel signal output 5.

[0081] In the masking circuit 10, in order to extract the timing at which the movement of the customer's hand 16 occurs and the corresponding pixel output timing, it is sufficient to pass the signal through an AND circuit that outputs pixel signal output 5 from area sensor camera 1" only when the binarized masking signal 9 is 1, and does not output pixel signal output 5 when the masking signal is 0. The output of masking circuit 10 that has done this is masked pixel signal output 11, and is shown in the right diagrams of Figures 20(a), (b), (c), (d), and (e). This is similar to linear sensor camera 1', in that the background disappears and only the area corresponding to the customer's hand 16 is output.

[0082] <First Embodiment-3> Another configuration of the motion contour extraction unit 6 and masking extraction unit 8 of the imaging camera 1 of FIG. 1 will be described with reference to FIG. 21. This configuration, which uses a linear sensor 4' as the solid-state imaging device 4, is referred to as First Embodiment-3. The configuration of the linear sensor camera 1' is the same as that of First Embodiment-1 of FIG. 2, with the pixel signal output 5 from the linear sensor 4' being split into two, one input to a masking circuit 10 and the other input to a motion contour extraction unit 6'. In the motion contour extraction unit 6', the pixel signal output 5 is further split into two and loaded into a line memory unit 24 via a switch circuit 23. The pixel signal output 5 emerging from the line memory unit 24 is the line output (referred to as the line memory output) at the timing loaded into the line memory unit 24, and a difference circuit 13 calculates the difference between this and the current pixel signal output 5, and the difference is passed through an absolute value circuit 14 to obtain motion contour information 7'. The line storage unit 24 stores one line of pixel signal output 5, just like the line memory 12, but because the signal is taken in via the switch circuit 23, it stores one line of pixel signal output 5 at the timing when the switch is turned on. The line storage unit 24 also differs from the line memory 12 of the first embodiment-1 in FIG. 2 in that it repeatedly outputs the stored one line of pixel signal output 5.

[0083] Here, the line storage unit output (pixel signal output 5) which is the output of the line storage unit 24 is the pixel signal output 5 at t=T in FIG. ０1 shows the waveform of the pixel signal output of background light from the linear sensor camera 4' at timing 1. When storing pixel signal output 5 of background light, which does not include the output of the customer's hand 16, the switch circuit 23 turns ON and stores it in the line memory unit 24. When storing pixel signal output 5 of background light, which does not include the output of the customer's hand 16, the switch is turned OFF and it is not stored in the line memory unit 24. In this way, the difference circuit 13 can remove background light. The switch circuit 23 is added as a new component to the motion contour extraction unit 6'. The line memory output of background light (pixel signal output 5) is stored in the line memory unit 24 and used repeatedly, so it is distinguished as the motion contour extraction unit 6'. The difference circuit 13 outputs positive or negative signals depending on the magnitude relationship of the pixel signal outputs between the lines being compared. Therefore, the output of the difference circuit 13 is input into the absolute value circuit 14 to obtain motion contour information 7'. The output of the motion contour extraction unit 6' is also distinguished as motion contour information 7'.

[0084] The motion contour information 7' ​​obtained by the motion contour extraction unit 6' is input to a masking extraction unit 8'. In the masking extraction unit 8', the motion contour information 7 is compared with a threshold value 19 in a masking signal generation circuit 20 and binarized to obtain a masking signal 9. The masking extraction unit 8' does not have the motion contour end determination unit 17 and motion contour end information 18 that were present in the masking extraction unit 8 in Fig. 9. For this reason, it is referred to as a masking extraction unit 8'.

[0085] The masking signal 9 output from the masking signal generating circuit 20 is a signal for extracting the timing at which the movement of the customer's hand 16 occurs and the corresponding pixel output timing. Furthermore, from this masking signal 9 and the pixel signal output 5 output from the linear sensor camera 4', a masking circuit 10 obtains a masking pixel signal output 11 that corresponds only to the movement of the customer's hand 16.

[0086] If the calculation time required by the motion contour extraction unit 6' and the masking extraction unit 8' causes problems in extracting only the movement of the customer's hand 16 from the pixel signal output 5, the pixel signal output 5 can be delayed by a time corresponding to this calculation time using the delay circuit 21 present in the masking circuit 10 of FIG. 11 to obtain the masked pixel signal output 11 from the masking circuit 10. The term "masking" used here does not refer to the commonly used meaning of hiding data, but rather to hiding everything other than the data and highlighting the data to reveal it. That is, in the masking circuit 10, if the binarized masking signal 9 is 1, the pixel signal output 5 is output, and if it is 0, the pixel signal output 5 is not output. This situation also has the significance of using logical product (AND) in the masking circuit 10, as described in paragraph 16. The purpose of the present application is to hide unnecessary background, eliminate unnecessary data processing, extract essential data (e.g., the customer's hand 16), and simplify signal processing.

[0087] Further, to add an explanation of the switches of the switch circuit 23, when the masking signal 9 is 0, there is no movement of the customer's hand 16, which corresponds to background light, so the switch circuit stores the line as ON, and when the masking signal 9 is 1, there is movement of the customer's hand 16, which corresponds to the movement of the customer's hand 16 that should be captured, so the switch circuit stores the line as OFF and does not store the line. Because the background light rarely changes, there is no need to use the masking signal 9 every time to control the ON / OFF of the switch circuit 23; it is sufficient to turn on the switch circuit 23 by periodic sampling and capture background light with no movement.

[0088] The masking signal 9 has a waveform of 1 and 0 within one line, and the masking circuit 10 extracts a moving object (customer's hand 16) from the pixel signal output 5. However, when the masking signal 9 that controls the switch circuit 23 is said to be 0 or 1, it means that if the masking signal is always 0 within one line of the masking signal, it is said to be 0, and if there is even a small moment when the masking signal becomes 1, it is said to be 1. To perform this operation, a selection unit 25 is located between the masking signal 9 and the switch circuit 23. The selection unit 25 also has a function similar to the timer that performs the periodic sampling described above, and periodically stores lines of background light with little change.

[0089] The operation of the motion contour extraction unit 6', masking extraction unit 8', and masking circuit 10 shown in Fig. 21 will be explained with reference to Fig. 22 based on the waveforms of each signal. The timing of the position of the customer's hand 16 is t=T ０ , t=T １ , t=T ３ First, the background image corresponds to t=T ０ At the timing of t=T, as shown in FIG. 6(a), the same waveform as the waveform of the pixel signal output 5 of the background light of the linear sensor camera 4' is captured in advance in the line memory unit 24 with no output from the customer's hand 16 and the switch circuit turned on. This line memory unit output is ０ This waveform is stored in the line storage unit 24, so it appears as a background image at other times as well. In Figures 22(b) and 22(d), it is expressed as line storage unit output.

[0090] Next, the customer's hand 16 is t=T in FIG. １ At this timing t=T, the waveform shown in FIG. 6(b) is obtained, and the pixel signal output is shown in FIG. 22(b'). １ Then, the difference circuit 13 calculates the pixel signal output and t=T ０ 22(b) is subtracted from the line memory output corresponding to the background image shown in FIG. 22(b) at the timing t=T １This is the motion information itself, but since the pattern end becomes the motion contour information, it is referred to as the motion contour information 7' ​​in relation to the first embodiment-1. ０ From then on, t=T １ If the customer's hand 16 is not recognized at the previous timing, the pixel signal output will have the same waveform as that shown in Figure 22 (b), and the motion contour information 7' ​​that passes through the difference circuit 13 and absolute value circuit 14 will remain zero because there is no moving object and no contour.

[0091] This timing t=T １ Then, the motion contour information 7' ​​is compared with the threshold value 19 in the masking extraction unit 8' and binarized. If the motion contour information 7' ​​exceeds the threshold value 19, 1 is assigned, and if not, 0 is assigned, and a masking signal 9 is obtained. １ 1 shows the masking signal 9 at the timing.

[0092] The period when the masking signal 9 is 1 corresponds to the timing when the movement of the customer's hand 16 occurs and the pixel output timing corresponding to the position of the customer's hand 16, and is the period when the signal should be masked and output from the pixel signal output 5. In other words, the pixel signal output 5 when the masking signal 9 is generated (is 1) corresponds to observing only the movement of the customer's hand 16, and is shown in FIG. 22(b'''') as the masking pixel signal output 11. This is the same as the period when there is no background, t=T, as shown in FIG. 4(b). １ This is a signal from the customer's hand 16 at the timing.

[0093] In a similar sequence, the customer's hand 16 moves to the time t=T in FIG. ３ At this timing t=T, the pixel signal output is as shown in FIG. 22(d'), just like in FIG. 6(d). ３ Then, the difference circuit 13 calculates the pixel signal output and t=T ０ 22(d) is subtracted from the line storage output (same as in FIG. 22(b)) corresponding to the background image at the timing of t=T shown in FIG. 22(d) and passes through the absolute value circuit 21. ３Similarly, the masking signal is shown in FIG. 22(d'''), and the masking pixel signal output 11 is shown in FIG. 12(d''''). ３ This is a signal from the customer's hand 16 at the timing.

[0094] The above is the timing t = T ０ , t=T １ , t=T ３ However, other timings are also included, including the timing t=T ０ , t=T １ , t=T ２ , t=T ３ , t=T ４ The left, center, and right diagrams of Figures 23(a), (b), (c), (d), and (e) show how the motion contour information 7' ​​and masking signal will behave for the pixel signal output waveforms shown in Figures 6(a), (b), (c), (d), and (e), which correspond to the above.

[0095] The masking signals 9 shown in the right diagrams of Figures 23(a), (b), (c), (d), and (e) are the same as the masking signals shown in Figures 13(a), (b), (c), (d), and (e), and the masking pixel signal output 11 obtained through the masking circuit 10 of Figure 21 is the same as the masking pixel signal output 11 obtained by the masking circuit 10 of Figure 11 at each timing, as shown in the right diagrams of Figures 14(a), (b), (c), (d), and (e). This is the same as the pixel signal output when there is no background light, as shown in Figures 4(a), (b), (c), (d), and (e), and the background light can be eliminated, resulting in an image of only the customer's hand 16, which is a moving object. The left diagrams of Figures 14(a), (b), (c), (d), and (e) also include the waveforms of the original pixel signal output 5. Timing t=T ０ , t=T １ , t=T ２ , t=T ３ , t=T ４ corresponds to the dashed area on the customer's hand 16.

[0096] <First Embodiment-4> Another configuration of the motion contour extraction unit 6 and masking extraction unit 8 of the imaging camera 1 of Fig. 1 will be described with reference to Fig. 24. Another configuration using an area sensor 4" as the solid-state imaging device 4 will be referred to as First Embodiment-4. The configuration of the area sensor camera 1" is the same as that of First Embodiment-2 of Fig. 16, and the pixel signal output 5 from the area sensor 4" is divided into two, one of which is input to a masking circuit 10 and the other is input to a motion contour extraction unit 6'. In the motion contour extraction unit 6', the pixel signal output 5 is further divided into two and taken into a frame storage unit 26 via a switch circuit 23'. The pixel signal output 5 coming out of the frame storage unit 26 is the frame output at the timing taken into the frame storage unit 26 (frame storage unit output The difference between this and the pixel signal output 5, which is the current frame output, is calculated by the difference circuit 13, and motion contour information 7' ​​is obtained via the absolute value circuit 14. The frame storage unit 26, like the frame memory 22, stores the pixel signal output 5 of one frame, but because it is taken in via the switch circuit 23', it stores the pixel signal output 5 of one frame at the timing when the switch is turned on. The frame storage unit 26 also differs from the frame memory 22 of the first embodiment-2 in FIG. 16 in that it repeatedly outputs the stored pixel signal output 5 of one frame. This is the same as in the first embodiment-3. The motion contour extraction unit 6' is distinguished as the motion contour extraction unit 6' because the switch circuit 23' is added as a new component, and the frame storage unit output (pixel signal output 5) of the background light is stored in the frame storage unit 26 and used repeatedly. The output of the motion contour extraction unit 6' is also distinguished as motion contour information 7'.

[0097] Here, the frame storage unit output (pixel signal output 5) which is the output of the frame storage unit 26 is a still image of the background light in the state where the customer's hand 16 is not present (shown by the broken line in FIG. 25). ０ ~S ４26(a), (b), (c), (d), and (e) show the waveforms of the scanning line pixel signal output of the background light of the area sensor 4" in this case. When storing the pixel signal output 5 of the background light when the customer's hand 16 does not appear, the switch in the switch circuit 23' is turned ON and the background light pattern shown in FIGS. 26(a) to (e) is taken into the frame memory unit 26. When the customer's hand 16 appears, the switch in the switch circuit 23' is turned OFF and the pixel signal output 5 is not taken into the frame memory unit 26. In this way, the background light can be removed by the difference circuit 13. The frame memory output (pixel signal output 5) of background light is stored in a frame memory 26 and is used repeatedly in the same way as the line memory 24. The difference circuit 13 outputs a positive or negative signal depending on the magnitude relationship of the pixel signal outputs between the frames being compared. Therefore, the output of the difference circuit 13 is input to an absolute value circuit 14 to obtain motion contour information 7'. As in FIG. 21, FIG. 24 also uses a motion contour extraction unit 6' and motion contour information 7' ​​to distinguish it from the first embodiment-2.

[0098] The motion contour information 7' ​​obtained by the motion contour extraction unit 6' is input to a masking extraction unit 8'. In the masking extraction unit 8', the motion contour information 7' ​​is compared with a threshold value 19 in a masking signal generation circuit 20, and binarized for each scanning line to obtain a masking signal 9. The masking extraction unit 8' does not have a motion contour end determination unit 17 or motion contour end information 18, as in FIG. 21 .

[0099] The masking signal 9 output from the masking signal generating circuit 20 is a signal for extracting the timing for outputting the pixel signal output 5 corresponding to the pattern of the customer's hand 16. To achieve this, a masking circuit 10 removes the background from this masking signal 9 and the pixel signal output 5 output from the area sensor camera 4" to obtain a masking pixel signal output 11 corresponding only to the customer's hand 16, which is a moving object.

[0100] If the calculation processing time of the motion contour extraction unit 6' and the masking extraction unit 8' causes problems in extracting only the movement of the customer's hand 16 from the pixel signal output 5, the pixel signal output 5 can be delayed by a time equivalent to this calculation time using the delay circuit 21 (not shown) present in the masking circuit 10 of Fig. 11, and the masked pixel signal output 11 can be obtained in the masking circuit 10. In the masking circuit 10, if the binarized masking signal 9 is 1, the pixel signal output 5 is output, and if it is 0, the pixel signal output 5 is not output.

[0101] Further, to add an explanation of the switches of the switch circuit 23, when the masking signals 9 for each scanning line are all 0, there is no movement of the customer's hand 16 and this corresponds to background light, so the switch circuit is turned ON and stores the frame, but if there is a portion where the masking signal 9 is 1 on even one scanning line (i.e., there is movement of the customer's hand 16), it is not stationary background light and the switch circuit is turned OFF and does not store the frame. Because the background light rarely changes, there is no need to use the masking signal 9 every time to control the ON / OFF of the switch circuit 23; it is sufficient to turn the switch circuit 23 ON by periodic sampling and capture stationary background light, as in the first embodiment-3 of FIG. 21.

[0102] The masking signal 9 has a waveform with regions of 1 and 0 within one frame, and the masking circuit 10 extracts a moving object (customer's hand 16) from the pixel signal output 5. However, when the masking signal 9 controlling the switch circuit 23' is referred to as 0 or 1, it is referred to as 0 if the masking signal is always 0 within one frame of the masking signal, and as 1 if there is even a slight moment when the masking signal becomes 1. To perform this operation, a selector 25' is located between the masking signal 9 and the switch circuit 23'. The selector 25' also has a timer-like function that performs the periodic sampling described above, and periodically stores frames of the background light, which does not change much. To distinguish between line storage and frame storage, the switch circuit 23' and selector 25' are used.

[0103] The operation of the motion contour extraction unit 6', masking extraction unit 8', and masking circuit 10 shown in Figure 24 will be described by taking as an example a case in which a customer's hand 16 reaches out to pick up a product displayed on the display shelf 15, and the situation is monitored by an area sensor camera 1" installed at the upper front of the product display shelf 15. As a background light pattern without the customer's hand 16, the pixel signal output of each scanning line of the area sensor 4" is shown in Figures 26(a) to (e), and is recorded in the frame memory unit 26. On the other hand, as shown in Figure 17, when a customer's hand 16 reaches out to pick up a product displayed on the product display shelf 15, the pixel signal output of the scanning line S of the area sensor 4" ０ ~S ４ The pixel signal output on each scanning line corresponds to the left diagram of Figure 19(a) to (e). The horizontal axis of the scanning line pixel signal output waveform of Figure 18(a) to (e) is the scanning line (S ０ ~S ４ ) and the vertical axis corresponds to the pixel signal output on each scanning line.

[0104] FIG. 27 shows the scanning line S of the area sensor 4″. １ , S ３ 27 is taken as an example, and the waveforms of the frame memory output (background light), the scanning line pixel signal output, and the motion contour information 7', masking signal 9, and masking pixel signal output 11, which correspond to the outputs of the motion contour extraction unit 6', masking extraction unit 8', and masking circuit 10, respectively, are explained. １ , the right figure shows the scanning line S ３ First, (b), (b'), (b"), (b"'), and (b"") in FIG. 27 correspond to the scanning lines S １ 27(d), (d'), (d"), (d"'), and (d"") correspond to the frame memory output (background light), the scanning line pixel signal output, the motion contour information 7', the masking signal 9, and the masking pixel signal output 11. ３ corresponds to the one in

[0105] In FIG. 27(b), (b'), (b"), (b"'), and (b""), the scanning lines S １ 27(b) shows the waveform of the scanning line S １27(b') shows the waveform of the frame memory output of the background light without the customer's hand 16 corresponding to the position of the scanning line S １ 27(b''). When the difference between the two is taken in the difference circuit 13 of the motion contour extraction unit 6' in FIG. 24 and the absolute value is taken in the absolute value circuit 14, motion contour information 7' ​​shown in FIG. 27(b'') is obtained, which is compared with the threshold value 19 in the masking signal generation circuit 20 of the masking extraction unit 8' in FIG. 24, and a binarized masking signal 9 is obtained as shown in FIG. 27(b'''). This and the scan line S from the area sensor 4'' are compared. １ When the AND of the pixel signal output 5 corresponding to the scan line is taken by the masking circuit 10, １ The masking pixel signal output 11 corresponding to the scan line S is obtained and is shown in FIG. ３ For the above, the frame memory output (background light), scanning line pixel signal output, motion contour information 7', masking signal 9, and masking pixel signal output 11 are obtained, which are shown in Figure 27(d), (d'), (d"), (d"'), and (d""), respectively.

[0106] The masking pixel signal output 11 shown in (b'') and (d'') of FIG. 27 is the pixel signal output of the scanning line S among the pixel signal outputs of the scanning line without background light shown in FIG. １ , S ３ 18(b) and (d) corresponding to the above. That is, even if a subject has background light, if there is a moving object (here, the customer's hand 16), passing it through the circuit shown in the first embodiment-4 of FIG. 24 makes it possible to extract and output only the moving object for each scanning line.

[0107] The above is the scanning line S １ , S ３ The scanning line S of the area sensor 4" of the area sensor camera 1" in FIG. ０ ~S ４Corresponding to the above, when background light is present, the pixel signal output on each scanning line is as shown in the left diagrams of Fig. 19(a), (b), (c), (d), and (e) (scanning line pixel signal output), which are shown again in the left diagrams of Fig. 28(a), (b), (c), (d), and (e). From this, motion contour information 7' ​​extracted via the motion contour extraction unit 6 shown in Fig. 24 is shown in the center of Fig. 28(a), (b), (c), (d), and (e), and further, masking signal 9 extracted via the masking extraction unit 8' of Fig. 24 is shown in the right diagrams of Fig. 28(a), (b), (c), (d), and (e).

[0108] This masking signal 9 is the same as the masking signals obtained in the first embodiment-3, as shown in Fig. 23(a), (b), (c), (d), and (e), and is also the same as the masking signals shown in Fig. 13(a), (b), (c), (d), and (e). That is, the masking pixel signal output 11 obtained through the masking circuit 10 in Fig. 24 is the same as the right diagrams in Fig. 14(a), (b), (c), (d), and (e), which correspond to the masking pixel signal output 11 obtained by the masking circuit 10 in Fig. 11 at each timing. This is the same as the pixel signal output when there is no background light, as shown in Fig. 4(a), (b), (c), (d), and (e). This timing is the same as the pixel signal output when there is no background light, as shown in Fig. 4(a), (b), (c), (d), and (e). In the area sensor 4'', ０ ~S ４ , which corresponds to the scanning line pixel signal output without background light in FIG.

[0109] <First Embodiment - 5> Another configuration of the motion contour extraction unit 6 and the masking extraction unit 8 of the imaging camera 1 in FIG. 1 will be described with reference to FIG. 29. As the solid-state imaging device 4, another configuration using an area sensor 4''' having a special pixel array structure described later is referred to as the First Embodiment - 5. The configuration of the area sensor camera 1'' is the same as that of the First Embodiment - 2 in FIG. 16. The pixel signal output 5 from the area sensor 4''' is split into two. One is input to the masking circuit 10, and the other is input to the motion contour extraction unit 6''. In the motion contour extraction unit 6'', the pixel signal output 5 is further split into two and captured into the bit memory 27. The pixel signal output 5 coming out of the bit memory 27 is the pixel signal output 5 from one bit before, and the difference from the current pixel signal output 5 is taken by the difference circuit 13. This corresponds to obtaining the difference between adjacent pixels. Further, the differentiated signal passes through the absolute value circuit 14 to obtain the motion contour information 7''. Since the motion contour extraction unit 6'' uses the bit memory 27, it is distinguished as the motion contour extraction unit 6''. Also, the output of the motion contour extraction unit 6'' is distinguished as the motion contour information 7''.

[0110] FIG. 29 shows a case where the bit memory 27 is used as a method for obtaining pixel differentiation. In the difference circuit 13, by taking the difference between the current pixel signal and the adjacent pixel signals of the same color, the difference operation between adjacent pixels of the same color can be performed, and pixel differentiation can be taken. Since it is a difference, positive and negative values are taken depending on light and darkness (having a sign), so through the absolute value circuit 14, it is converted into absolute value information to obtain the motion contour information 7''.

[0111] As a method for performing pixel differentiation operation between adjacent pixels of the same color, there are a method of taking the difference between adjacent pixels of the same color in the left - right direction on the same scanning line, and a method of simultaneously reading adjacent scanning lines and taking the difference between adjacent pixels of the same color in the up - down direction. In the former case, since the difference between the same - color pixels on the same scanning line is taken, the bit memory 27 shown in FIG. 29 is used. In the latter case, since adjacent scanning lines come out simultaneously, the bit memory 27 becomes unnecessary. The motion edge extraction unit 6'' shown in FIG. 29 does not have the frame memory 22 that was in the motion contour extraction unit 6 in FIG. 16. Therefore, the motion contour information 7'' is an output with respect to the image information based on the most recent (not frame - based) scanning line captured by the area sensor 4'''.

[0112] As a first embodiment-5, a method of extracting a motion contour using a bit memory 27 without using a frame memory, as shown in FIG. 29, will be described. Acquiring motion contour information from the difference between adjacent pixel signals of the same color is not possible with a normal pixel structure. Therefore, the pixel structure of the area sensor 4''' used in this case is special, and is composed of two types of pixels 28 with different exposure times, as shown in FIGS. 30 and 31. For this reason, it is distinguished from the area sensor 4" in FIGS. 16 and 24 and is referred to as area sensor 4'' in FIG. 29.

[0113] The configuration of the pixels 28 of the area sensor 4"' in FIG. 29 is shown in FIG. 30(a), and the pixel output characteristics are shown in FIG. 30(b). In the pixel configuration of the area sensor 4"' shown in FIG. 30(a), short exposure pixels STS and long exposure pixels STL are arranged adjacent to each other in the horizontal direction, corresponding to two types of pixels 28 with different exposure times, and output alternately on one scanning line. As shown in FIG. 30(b), the pixel output characteristics of both pixels 28 show the same slope (corresponding to sensitivity) of the pixel output increasing with the passage of time, and a pixel output commensurate with the exposure time can be obtained. STS and STL are read out at the same timing and output as pixel signal output 5 of the area sensor 4"'. In FIG. 30(b), the exposure times of the short exposure pixel STS and the long exposure pixel STL are respectively represented as T Ｓ , T Ｌ (T Ｌ >T Ｓ )

[0114] Of the pixel signal outputs 5 input to the motion end extraction unit 6" in Fig. 29, the short exposure pixel STS enters the bit memory 27, is delayed by a time equivalent to one pixel on a scanning line, and is then input to the difference circuit 13. This timing is the same as the input to the difference circuit 13 of the adjacent long exposure pixel STL, and the difference between adjacent pixels on one scanning line can be obtained. At this time, since the pixel outputs of the short exposure pixel STS and the long exposure pixel STL differ in magnitude by the difference in exposure time as shown in Fig. 30(b), gain adjustment is performed so that the outputs are aligned. Specifically, in addition to delaying by the time equivalent to one pixel on a scanning line (1 bit) in the bit memory 27, the signal is also delayed by the exposure time T Ｓ , TＬ In order to correct the difference in output proportional to Ｌ / T Ｓ This gain adjustment function is also built into the bit memory 27 (not shown). By adjusting the gain and taking the difference, the outputs are aligned for a still image and the output of the difference circuit 13 becomes zero, whereas for a moving subject, an output remains at the edge of the moving object, and after passing through the absolute value circuit 14, the output remains as motion contour information 7".

[0115] As such, in still areas, the difference is the same signal level, so it becomes zero. In areas where there is movement and the output value of the moving object changes, the difference does not become zero when it is acquired. Furthermore, if the brightness is uniform within the moving object pattern, the inside of the moving object pattern becomes zero, similar to a still image, and the outline pattern of the moving object remains. In this way, the moving outline information 7" includes outline information of the moving object. Paragraph 26 defines "pixel difference" as "the difference between pixels with different exposure timings that image the same position on the subject." Therefore, while the difference between adjacent pixels is strictly different from the difference between pixels defined in the present application, since the number of pixels is typically several thousand for linear sensors and one to ten million for area sensors, it is safe to assume that adjacent pixels are capturing the same position on the subject. Furthermore, as an additional function of the second embodiment-5 shown in Figure 33, averaging pixels of the same color in the vertical and horizontal directions and taking the difference between the pixel outputs of adjacent short-exposure pixels STS and long-exposure pixels STL can reduce noise that occurs at the edges of still objects, enabling more accurate outline extraction of moving objects. In this case, it can be further assumed that adjacent pixels are photographing the same position (area) of the subject.

[0116] In the area sensor 4''' shown in FIG. 30(a), the short exposure pixels STS and the long exposure pixels STL are arranged adjacent to each other in the horizontal direction. However, when they are adjacent to each other in the vertical direction, the two lines of the short exposure pixels STS and the long exposure pixels STL are read out simultaneously as one scanning line. In this case, there is no need for a delay of the time equivalent to one pixel (1 bit) in the bit memory 27, and the exposure time T Ｓ , T Ｌ In order to compensate for the difference in output proportional to Ｌ / T Ｓ The gain-adjusted short-exposure pixel STS and long-exposure pixel STL are input to the differential circuit 13, and the difference is calculated.

[0117] Another configuration of the pixel 28 of the area sensor 4"' in FIG. 29 is shown in FIG. 31(a), and its pixel output characteristics are shown in FIG. 31(b). In the configuration of the pixels 28', 28" of the area sensor 4"' shown in FIG. 31(a), a short-exposure pixel 28' (shown as STS in the figure) and a long-exposure pixel 28" (shown as STL in the figure) are arranged, but the two pixels have different sizes and therefore different sensitivities. If the sensitivity of the short-exposure pixel STS is SS and the sensitivity of the long-exposure pixel STL is SL, then in the case where the sensitivity ratio between them; SS / SL is the reciprocal of the exposure time ratio between them; TS / TL, the pixel output characteristics will be as shown in FIG. 31(b), and the pixel outputs of the short-exposure pixel STS and the long-exposure pixel STL will be the same.

[0118] The short-time exposure pixel STS and the long-time exposure pixel STL are adjacent to each other in the vertical direction in the pixel arrangement shown in FIG. 31(a). These two lines are read out simultaneously as one scanning line. As shown in FIG. 31(b), the exposure time T Ｓ : T Ｌ The sensitivity ratio of both is T Ｌ : T Ｓ Therefore, the pixel outputs of the short exposure pixel STS and the long exposure pixel STL are aligned. The aligned pixel outputs of the short exposure pixel STS and the long exposure pixel STL are input to the difference circuit 13, which takes the difference. In a still image, the outputs are aligned and the output of the difference circuit 13 is zero, whereas in a moving subject, an output remains at the edge of the moving object, and after passing through the absolute value circuit 14, the output remains as motion contour information 7".

[0119] In the pixel output characteristics shown in Figure 31(b), the pixel outputs of the short-exposure pixel STS and the long-exposure pixel STL are aligned, eliminating the need for gain adjustment in the bit memory 27. However, since only the exposure time ratio (TS / TL) determined by the sensitivity ratio SS / SL can be selected, gain adjustment is still desirable to increase the flexibility of the exposure time. In this way, the bit memory 27 is used as a general term for circuits that perform bit delays and gain adjustments, if necessary, between pairs of adjacent pixels with different exposure times so that the outputs are aligned in a still image and become zero when the difference is calculated.

[0120] By adjusting the gain for pixel pairs with different exposure times shown in FIG. 30 to make the outputs uniform, or by adjusting the pixel sensitivity to match the exposure time for pixel pairs with different exposure times shown in FIG. 31 to make the outputs uniform, motion contour information 7" is prevented from being generated, and the still image in the background can be eliminated.

[0121] In the first embodiment-5 of FIG. 29, an area sensor 4"' is used and the pixel structure of the area sensor shown in FIGS. 30 and 31 has been described, but this pixel structure also holds true for a linear sensor. The greatest feature of the present invention is that the pixel signal output 5 from the solid-state imaging device 4 is processed for each readout line to create a masking pixel signal output 11 in real time. For this reason, there is no difference between an area sensor and a linear sensor. As described above, the final masking pixel signal output 11 is the same in the first embodiment-1 (linear sensor), first embodiment-2 (area sensor), first embodiment-3 (linear sensor), first embodiment-4 (area sensor), and first embodiment-5 (area sensor).

[0122] <Second Embodiment> The imaging camera 1 has been described above as an imaging device, but next, as a second embodiment of the imaging device, a solid-state imaging device 4 shown in Fig. 32 will be described as an imaging device. Fig. 32 describes the block configuration of the solid-state imaging device 4 of the present invention and the signals of each block.

[0123] A lens 2 forms an image of a subject on a pixel section 28 of a solid-state imaging device 4. A pixel signal output 5 output from the pixel section is split into two: one is input to a motion contour extraction section 6, and the other is input to a masking circuit 10. The motion contour extraction section 6 extracts motion contour information 7 from the pixel signal output 5. The extracted motion contour information 7 is input to a masking extraction section 8. The masking extraction section 8 generates and outputs a masking signal 9 based on the motion contour information 7, which is used to output only moving objects. The masking signal 9 is input to the masking circuit 10, and together with the pixel signal output 5 also input, the masking circuit generates and outputs a masking pixel signal output 11. This masking pixel signal output is an output of only pixels corresponding to an image of a moving object, making this an imaging device that outputs only moving object information, which is the objective of the present invention. From the pixel signal output 5 onwards, the block configuration and signals are the same as those of the imaging camera 1 of the first embodiment shown in FIG. 1, and the block configuration is formed (on-chip) on the solid-state imaging device 4 where the pixel section is located.

[0124] That is, in the second embodiment, the circuit blocks of the imaging camera 1 of the first embodiment shown in FIG. 1 , which are located after the pixel signal output 5 of the first embodiment-1 (linear sensor), first embodiment-2 (area sensor), first embodiment-3 (linear sensor), first embodiment-4 (area sensor), and first embodiment-5 (area sensor), are formed on a solid-state imaging device 4, resulting in second embodiment-1 (linear sensor), second embodiment-2 (area sensor), second embodiment-3 (linear sensor), second embodiment-4 (area sensor), and second embodiment-5 (area sensor), respectively. The functions and operations are as described above, but here, the circuit configuration of the solid-state imaging device 4"' will be described with reference to FIG. 33, taking as an example second embodiment-5 (area sensor) which has a special pixel structure.

[0125] <Second embodiment-5> In the imaging camera 1 of FIG. 1, the pixel signal output 5 output from the area sensor 4''' in FIG. 29 passes through a bit memory 27 having a gain adjustment function, a difference circuit 13, and an absolute value circuit 14 in the motion contour extraction unit 6'', to obtain motion contour information 7''. There are several types of pixel arrangements in the area sensor 4''', as shown in FIG. 30(a) and FIG. 31(a), so in FIG. 33, the pixel signal passes through a separation circuit 29 and a gain adjustment circuit 29', and the difference is taken in the difference circuit 13 from the short exposure pixel STS and the long exposure pixel STL, and the absolute value is obtained. 33. In this embodiment, motion contour information 7" is obtained via a value circuit 14. The processing method is the same as in the first embodiment-5, and since the functions of the separation circuit 29 and gain adjustment circuit 29' are included in the bit memory 27, they are denoted by the same numbers. In FIG. 33, the method of obtaining the masking pixel signal output 11 from the motion contour information 7" via the masking extraction unit 8" and masking circuit 10 is the same as in the first embodiment-5 of FIG. 29. The masking circuit 10 is an AND circuit 30 (which has not been assigned a number until now).

[0126] In the circuit configuration of the solid-state imaging device 4"' shown in FIG. 33, more specific circuit blocks have been added. In the pixel arrays shown in FIGS. 30(a) and 31(a), processing using the short-exposure pixels STS and the long-exposure pixels STL was performed using the difference between adjacent pixel signals of the same color. In order to obtain a color image, color video information 32 is obtained via a color processing unit 31. The pixel unit 28 is made up of two types of pixels with different exposure times, as shown in FIGS. 30(a) and 31(a).

[0127] Describing the pixel section 28 of the solid-state imaging device 4"' in FIG. 33, color filters in a Bayer arrangement are formed in each pixel for color imaging. These are indicated in the figure by green (Gr, Gb), red (R), and blue (B). Exposure control, vertical scanning, and horizontal scanning are performed at the pixels based on a master clock generated by a timing generation circuit 33. As an exposure control method, exposure times TL and TS for the long exposure pixel STL and short exposure pixel STS described in FIGS. 30 and 31 are generated by an exposure time generation circuit 34, and the pixel section 28 is controlled via a vertical scanning circuit 35.

[0128] The vertical parallel control circuit 36 ​​can also vertically combine (average) pixel signals of the same color arranged vertically by simultaneously turning on multiple pixel drive lines arranged in the vertical direction. Above the pixel section 28, circuits that simultaneously read out pixel signals from multiple vertical signal lines output from the pixel section 28 are comprised of a horizontal scanning circuit 37 that scans the horizontal readout, and a horizontal combining circuit 38 that receives the vertical signal lines and combines the signals in the horizontal direction. The horizontal combining circuit 38 can also horizontally combine (average) pixel signals of the same color. This pixel combination of the same color is performed between pixels with the same exposure time, which reduces noise that occurs at the edges of stationary objects and enables more accurate contour extraction of moving objects.

[0129] As shown in Figures 30 and 31, short exposure pixels STS and long exposure pixels STL alternately repeat for each scanning line, but the synthesis (averaging) of pixel signals of the same color and with the same exposure time is performed across a plurality of scanning lines, so long exposure information and short exposure pixel information of approximately the same area can be acquired, and the motion contour information 7" obtained by subtracting in the difference circuit 13 is a comparison of approximately the same area, which has the advantage of improving the accuracy of the motion information. This is because when the difference between the short exposure pixel STS and the long exposure pixel STL is taken on a pixel-by-pixel basis, a problem occurs in which the difference does not become zero at the contour parts even in the case of a stationary object (edge ​​noise), but by averaging, the edge noise at the contour parts can be significantly suppressed, which has the advantage. This is the reason why the contours of moving objects can be extracted with higher accuracy, as mentioned above.

[0130] Combining (averaging) pixel signals of the same color with the same exposure time has the advantage of significantly reducing the amount of information, although this depends on the number of pixels combined in the vertical and horizontal directions. For example, averaging 10 pixels in each direction can reduce the amount of information by two orders of magnitude (1 / 100). In conventional motion analysis, processing the massive amount of image information output from a solid-state imaging device in real time, pixel by pixel, requires a heavy image processing load on the downstream processing circuit, making battery operation difficult. With this method, the amount of information corresponding to the number of pixels is significantly reduced (by two orders of magnitude in the above example). Furthermore, by processing the binarized motion contour information 9, the amount of information per pixel is also significantly reduced (e.g., from 8 bits to 1 bit), which is another major advantage. This, along with outputting only moving objects, contributes to the load reduction achieved by reducing the amount of information, which is a feature of the imaging device of the present invention.

[0131] In the solid-state imaging device 4"' in FIG. 33, a column-type noise cancellation circuit (CDS) 39 to which pixel signals of a plurality of vertical signal lines output from the pixel section 28 are input, a column-type analog-to-digital converter circuit (A-D conversion) 40, a line memory 41, and a horizontal pixel scanning circuit 42 are added below the pixel section 28. Based on a plurality of lines of pixels in a Bayer array (2 × 2 pixels) (which is why the line memory 41 is necessary), color processing is performed in a color processing section 31 to generate color video information 32. The line memory 41 has the same name as the line memory 12 in FIG. 2, but it is a line memory for performing color processing with an area sensor, and so although the name is the same, different numbers are used. In the explanation of the symbols, it is referred to as line memory (for color processing).

[0132] 33, the parts related to color processing may be omitted, and only the masking pixel signal output 11 may be output. Omitting the parts related to color processing means omitting the CDS circuit 39, ADC circuit 40, line memory 41, horizontal pixel scanning circuit 42, and color processing unit 31.

[0133] In Figure 33, we have explained a method for acquiring motion vector information while reducing the amount of information and edge noise by utilizing the vertical parallel control circuit 36 ​​and horizontal synthesis circuit 38 to vertically synthesize (average) pixel signals of the same color arranged vertically, and horizontally synthesize (average) pixel signals of the same color in the horizontal direction. The horizontal synthesis circuit 38 and horizontal scanning circuit 37 are added to the parts related to color processing in a normal solid-state imaging device. The present invention can also be applied to the output of normal color processing, as will be explained in the next Figure 34.

[0134] In Figure 34, the horizontal scanning circuit 37 and horizontal synthesis circuit 38 are omitted. For color imaging, color filters (green (Gr, Gb), red (R), and blue (B) in the figure) are formed in each pixel in a Bayer arrangement, and from the output of a line memory 41 for color processing these, a motion contour extraction unit 6" extracts motion contour information 7" from STS and STL of the same color. The rest is the same as in Figure 33.

[0135] <Relationship between claims, embodiments, and corresponding drawings> Claim 1: The first and second embodiments of the present invention, corresponding to FIGS. 1 and 32, respectively. Claim 2: The first embodiment of the present invention, corresponding to FIGS. 2 and 3. Claim 3: The first embodiments-3 and-4 of the present invention, corresponding to FIGS. 21-24 and 24-28, respectively. Claim 4: The first embodiments-3 and-4 of the present invention, corresponding to FIGS. 21 and 24, respectively. Claim 5: The first embodiments-1 and -2 of the present invention, corresponding to FIGS. 2-15 and 16-20, respectively. Claim 6: The first embodiment-5 and second embodiment-5 of the present invention, corresponding to FIGS. 29-31 and 33-34, respectively. Claim 7: The second embodiment-5 of the present invention, corresponding to FIG. 33. Claim 8: The first embodiments-1 and -3 of the present invention, corresponding to FIGS. 2 and 21, respectively. Although not shown in the drawings, it also corresponds to the second embodiments-1 and -3 of the present invention. Claim 9: The first embodiments-2 and -4 of the present invention, corresponding to FIGS. 16 and 24, respectively. Although not shown, this also corresponds to second embodiment-2 and -4 of the present invention. Claim 10: This is the first embodiment of the present invention and corresponds to Figures 1 to 31. Claim 11: This is the second embodiment of the present invention and corresponds to Figures 32 to 34.

[0136] REFERENCE SIGNS LIST 1 Imaging camera 1' Linear sensor camera 1" Area sensor camera 2 Lens 3 Camera housing 4 Solid-state imaging device 4' Linear sensor 4", 4"' Area sensor 5 Pixel signal output 6, 6', 6" Motion contour extraction unit 7, 7', 7" Motion contour information 8, 8', 8" Masking extraction unit 9 Masking signal 10 Masking circuit 11 Masking pixel signal output 12 Line memory 13 Difference circuit 14 Absolute value circuit 15 Product display shelf 16 Customer's hand 17, 17" Motion contour end determination unit 18, 18" Motion contour end information 19 Threshold value 20 Masking signal generation circuit 21 Delay circuit 22 Frame memory 23, 23' Switch circuit 24, 24' Line storage unit 25, 25' Selection unit 26 Frame storage unit 27 Bit memory 28, 28', 28" Pixels 29 Separation circuit 29' Gain adjustment circuit 30 AND circuit 31 Color processing section 32 Color image information 33 Timing generation circuit 34 Exposure time generation circuit 35 Vertical scanning circuit 36 ​​Vertical parallel control circuit 37 Horizontal scanning circuit 38 Horizontal synthesis circuit 39 CDS circuit 40 ADC circuit 41 Line memory (for color processing) 42 Horizontal pixel scanning circuit

Claims

1. Means for capturing an image of a subject using a pixel section having a plurality of pixels, which are photoelectric conversion element units that convert the optical signal of the subject imaged by an optical imaging means into an electrical signal, and generating a differential signal between the two captured images, at least one of which is a pixel signal output that is the current captured image signal and which corresponds to the difference between pixels corresponding to the same location in a stationary subject, from a still subject; means for adjusting the pixel signal output of the two captured images that constitute the differential signal so that the differential signal is not generated from the captured image signal from a still subject; means for taking the absolute value of the differential signal and acquiring moving contour information related to contour information of a moving subject for each readout line; means for acquiring a binary masking signal for each readout line, which is obtained from the moving contour information by comparing it with a threshold value and sets moving subject areas to 1 and other areas to 0; means for performing a logical AND operation on the binarized masking signal and a pixel signal output, which is a current shooting signal, from the pixel unit, and outputting only the pixel signal output where the masking signal is 1 as an image of a moving subject for each readout line.

2. The imaging device described in claim 1, characterized in that a memory is used as a means for generating a differential signal between two imaging signals captured by the pixel section at different imaging times, one of which is the current imaging signal, and the time interval between the different imaging times is an integer multiple of the cycle time required to read out all pixels in the pixel section, thereby performing image difference.

3. The imaging device according to claim 1, characterized in that a means for generating a difference signal between two imaging signals captured by the pixel section at different times, one of which is the current imaging signal, is realized by repeatedly using a still imaging signal with no movement as the imaging signal for performing pixel difference, thereby performing image difference.

4. The imaging device according to claim 3, characterized in that a binarized masking signal is utilized to determine which still-still imaging signals should be stored in memory, and an imaging signal whose masking signal is all 0 is determined to be a still-still imaging signal with no movement, and this imaging signal is temporarily stored in memory on a regular basis.

5. The imaging device described in claim 2, characterized in that the time interval between different shooting times is the cycle time required to read out all pixels in the pixel section, image difference is performed, and a binarized masking signal is obtained by comparing with a threshold value after passing through motion contour end information corresponding to information on the outer contour of a moving object.

6. The imaging device according to claim 1, characterized in that the means for generating a differential signal between two photographing signals photographed at different times by the pixel section is realized by: an exposure time control means for dividing the pixels arranged in the pixel section into at least two and controlling the exposure time length to be different; a means for generating a differential signal between the photographing signals having the different exposure times, adjusted so that no differential signal is generated from the photographing signal of a stationary subject; and a means for generating a differential signal between the two photographing signals by the difference between adjacent pixels, both of which are pixel signal outputs that are photographing signals at the current time and which are considered to correspond to the same location of the subject.

7. In an imaging device that outputs only images of moving subjects, the imaging device described in claim 6 is characterized in that the means for generating a difference signal between two shooting signals captured by the pixel section at different shooting times is a means for averaging shooting signals of multiple adjacent pixels under the same conditions and generating a difference signal between the averaged shooting signals.

8. The imaging device according to claim 1, wherein the pixel section is a linear sensor in which pixels are linearly arranged, the memory used is a line memory, and the cycle time is a line period.

9. The imaging device according to claim 1, wherein the pixel section is an area sensor in which pixels are arranged in an area, the memory used is a frame memory, and the cycle time is a frame period.

10. The imaging device according to claim 1, wherein the imaging device is an imaging camera.

11. The imaging device according to claim 1, wherein the imaging device is a solid-state imaging device.

Images