Imaging apparatus

Abstract

To solve the problem in which, when extracting a moving object from a captured image, it is necessary to process a vast amount of image information output from an imaging apparatus, making a system large, and increasing power consumption.SOLUTION: The imaging apparatus is configured to generate a differential signal from pixel signals with different capture times, adjust so as not to generate the differential signal for a still subject, determine outline edges of a moving object on the basis of the differential signal, generate a binary masking signal, take a logical product from the pixel signal output, which is the current image signal, and the masking signal, and output only the image of the moving subject for each readout line.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for acquiring motion information, and more particularly to an imaging device which is characterized in that it erases background still image information based on contour information of a moving object, and acquires and displays only the moving object. Here, the imaging device refers to a solid-state imaging device or an imaging camera using the same. [Background technology]

[0002] In imaging devices such as solid-state imaging devices and imaging cameras using such devices, there are motion vector display methods that display the direction and speed of moving objects (hereinafter referred to as moving objects) in an image. As a method for acquiring such motion vectors, a block matching method is known as a conventional technique.

[0003] Patent Document 1 applies this block matching technique to track a moving object, while Patent Document 2 shows a technique for extracting feature points on the contour curve of a moving object and estimating a motion vector using the extracted feature points.

[0004] Furthermore, Patent Document 3 discloses a method for extracting feature points representing predetermined characteristics from an image of one frame captured by an imaging device, extracting feature points from a region adjacent to the feature points of an image of a temporally subsequent frame, and extracting a motion vector from both feature points with a small amount of calculation.

[0005] A common method for detecting the movement of a subject on the image capture screen of a solid-state imaging device is to compare signal outputs at the same pixel level between previous and next frames, but there is also a method in which adjacent pixels are treated as pairs, integral times are changed, and gain is adjusted to detect movement based on differential signal output, as shown in Patent Document 4. There is also a method in which adjacent pixels with different sensitivities are treated as pairs, integral times are changed, and movement is detected based on differential signal output, as shown in Patent Document 5. The former method of comparing frames is called the inter-frame difference method, and the latter method of comparing pixel pairs is called the inter-pixel difference method.

[0006] The latter, Patent Document 5, shows that a difference signal for determining motion detection is directly output, and only the contour information of a moving object (hereinafter referred to as a moving body) is output, eliminating the process of extracting the contour of the moving body from an image, and that this can be applied to a human-machine interface. For this contour information of a moving body, differences in the amount of photoelectrically converted charge occur in pixels according to changes in motion, and the differences are output as contour information corresponding to the pixel positions.

[0007] Patent Document 6 shows a concept of reducing the power consumption of a system by controlling the power supply of a downstream system to be turned on / off based on the results of motion detection, and a concept of reducing the amount of information by controlling the image signal output from a solid-state imaging device. The motion detection methods in Patent Documents 2 and 3 show cases where motion detection is possible using pixel differences without using a frame memory.

[0008] Patent Document 7 shows a method for obtaining motion vectors that significantly reduces the load of image information arithmetic processing in subsequent circuits. The method combines a difference circuit, a differentiation circuit, logical AND, and logical OR to obtain and output motion vector information based on the contour information of a moving object obtained from a solid-state imaging device. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent No. 3037383 [Patent Document 2] Patent No. 3674084 [Patent Document 3] Patent No. 6624841 [Patent Document 4] Patent No. 3521109 [Patent Document 5] Patent No. 5646421 [Patent Document 6] Patent No. 5604700 [Patent Document 7] Patent No. 7025073 Summary of the Invention [Problem to be solved by the invention]

[0010] In the conventional motion vector acquisition methods shown in Patent Documents 1, 2, and 3, the amount of image information to be processed is large, leading to a large calculation scale and increased power consumption. In addition, Patent Documents 5 and 6 show that it is possible to easily acquire motion contour information, but do not show a method for acquiring or outputting a motion vector.

[0011] In Patent Document 7, the load of arithmetic processing of image information in a downstream circuit is significantly reduced with regard to obtaining motion vectors, but moving objects are displayed using motion vectors based mainly on contour information, and information on the moving objects themselves is missing.

[0012] The object of the present invention is to significantly reduce the load of processing image information in a downstream circuit when acquiring only image information of a moving object by deleting still image information of the background. The method involves acquiring masking information based on contour information of the moving object obtained from an imaging device, and acquiring and outputting information on the moving object in a simple manner. [Means for solving the problem]

[0013] In order to solve the above problems, an imaging device is provided which acquires contour information of a moving object from a solid-state imaging device or an imaging camera, reduces the amount of information by deleting background still image information, acquires masking information for the moving object from the position and size of the moving object based on this contour information, and takes the logical product (hereinafter referred to as AND) of the masking information and image information for each line, thereby outputting information only about the moving object from the image information.

[0014] In this way, the present invention does not handle still images among the captured images, but only moving objects. Furthermore, by displaying only the images of moving objects and erasing the still images in the background, the amount of information is significantly reduced. This has the advantage of making it easier to visually grasp the movement information.

[0015] With the proposed imaging device (solid-state imaging device, imaging camera), only image information of moving objects is output, so the amount of information is significantly reduced, and it is possible to provide an imaging device (moving object output sensor, moving object output camera) that outputs only moving objects without having to perform real-time calculations on the vast amount of image information output from normal camera images.

[0016] In the present invention, an imaging device is provided that obtains masking information for moving objects from contour information, and outputs only information on moving objects from image information by taking the AND of the masking information and image information for each line. The masking information referred to here is a masking waveform for extracting only moving objects, and in order to leave moving objects by logical product (AND), the masking information is binarized, and the masking waveform is a waveform that corresponds to 1 when outputting information on only moving objects and 0 when outputting a still image of the background. Generally, solid-state imaging devices are classified into area sensors with pixels arranged two-dimensionally and linear sensors with pixels arranged one-dimensionally, but by processing AND for each line, either a linear sensor or an area sensor is possible. When memory is used as a method for acquiring contour information of a moving object, line memory and area memory are generally used for linear sensors and area sensors, respectively. Since the name frame memory is more common than area memory, it will be referred to as frame memory in the following explanation. Linear sensors are also sometimes called line sensors, but here they will be referred to as linear sensors. As a method for acquiring contour information of a moving object, there is also a method for acquiring the contour information from pixel differences between pixel pairs with different exposure times without using a memory. Effect of the Invention

[0017] According to the present invention, the amount of information to be calculated is significantly reduced by deleting background still images during the process of acquiring motion contour information. Furthermore, the amount of information from the imaging device can be reduced by ANDing the masking pattern created from the contour information with the original image and outputting only moving objects. Furthermore, the calculation to extract only moving objects is performed line by line along the scanning line, and is performed in real time in parallel with the pixel output for each line, thereby achieving a significant reduction in the calculation load.

[0018] Conventional imaging cameras have a large amount of image information because they undergo various processing steps in the post-processing circuitry. On the other hand, the present invention limits the output to moving objects only, reducing the amount of information, which reduces the computational load and significantly reduces the amount of information output from the imaging device. [Brief description of the drawings]

[0019] [Figure 1] 1 is a block diagram showing a configuration in which an imaging camera is used as an imaging device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a schematic configuration of a motion contour extraction unit using a linear sensor according to the first embodiment. [Diagram 3] A diagram showing the position of the customer's hand captured at each point in time by the linear sensor camera in Figure 2. [Figure 4] 4 is a diagram showing pixel signal outputs of a linear sensor and reconstructed images of a customer's hand at each timing in FIG. 3. [Diagram 5] 5 is an explanatory diagram of the pixel signal output of FIG. 2, a difference mechanism using a line memory, and motion contour information at the timing excerpted from FIG. 4; [Figure 6] 5A and 5B are diagrams showing the actual pixel signal output of a linear sensor with background light at each timing in FIG. 4; [Figure 7] 7 is an explanatory diagram of the pixel signal output in FIG. 2, a difference mechanism using a line memory, and motion contour information at the timing excerpted from FIG. 6; [Figure 8] 7 is an explanatory diagram of motion contour information acquired by the motion contour extraction unit in FIG. 2 at each timing in FIG. 6; [Figure 9]FIG. 2 is a block diagram showing a schematic configuration of a masking extraction unit using a linear sensor according to the first embodiment. [Figure 10] 8 is an explanatory diagram of a masking signal acquired by the masking extraction unit in FIG. 9 at the timing excerpted from FIG. 7; [Figure 11] FIG. 2 is a block diagram showing a schematic configuration of a masking circuit using a linear sensor according to the first embodiment. [Figure 12] 12A to 12C are explanatory diagrams of masking pixel signals acquired by the masking circuit of FIG. 11 at selected timings. [Figure 13] 10 is an explanatory diagram of a masking signal acquired by the masking extraction unit in FIG. 9 at each timing. [Figure 14] 12A to 12C are explanatory diagrams of masking pixel signals acquired by the masking circuit of FIG. 11 at each timing. [Figure 15] 11A to 11C are explanatory diagrams of masking pixel signals at different timings and with different criteria for determining the edge of a motion contour. [Figure 16] FIG. 11 is a block diagram showing a schematic configuration of a motion contour extraction unit using an area sensor according to the first embodiment-2. [Figure 17] FIG. 17 is a diagram showing the positions of a customer's hands captured by each scanning line of the area sensor camera in FIG. 16. [Figure 18] 18 is a diagram showing scanning line pixel signal outputs of the area sensor in FIG. 17 and an image of a customer's hand, which is a moving object. [Figure 19] 4 is an explanatory diagram of pixel signal output of each scanning line of an area sensor, a difference mechanism using a frame memory, and motion contour information. [Figure 20] 4A to 4C are explanatory diagrams of motion contour information, masking signals, and masking pixel signals for each scanning line. [Figure 21] FIG. 11 is a block diagram of another motion contour extraction method using a linear sensor according to the first embodiment-3. [Figure 22] 22 is an explanatory diagram of motion contour information, a masking signal, and a masking pixel signal using FIG. 21 at selected timings. [Figure 23] FIG. 21 is an explanatory diagram of motion contour information and masking signals at each timing. [Figure 24]FIG. 11 is a block diagram of another motion contour extraction method using an area sensor according to the first embodiment-4. [Diagram 25] FIG. 25 is a diagram showing the positions of the customer's hands captured by each scanning line of the area sensor camera in FIG. 24. [Figure 26] Scanning line pixel signal output of the area sensor in FIG. 24 when there is no customer's hand (moving object). [Figure 27] 25 is an explanatory diagram of motion contour information, masking signals, and masking pixel signals using FIG. 24 in a selected scanning line. [Figure 28] An explanatory diagram of motion contour information and masking signals using Figure 24 for each scanning line. [Figure 29] FIG. 11 is a block diagram of another motion contour extraction method using a special area sensor as the first embodiment-5. [Diagram 30] 30 is a diagram for explaining the pixel arrangement and pixel output characteristics of the area sensor in FIG. 29. [Diagram 31] 30 is a diagram for explaining another pixel arrangement of the area sensor of FIG. 29 and its pixel output characteristics. [Diagram 32] FIG. 11 is a block diagram showing a configuration in which a solid-state imaging device is used as an imaging device according to a second embodiment of the present invention. [Diagram 33] FIG. 11 is a block diagram of a solid-state imaging device according to a second embodiment 5, which uses a special area sensor. [Diagram 34] A block diagram of another solid-state imaging device using the special area sensor of FIG. 33. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] As an embodiment of the present invention, several selection cases are possible. The combination cases are: 1) the imaging device is an imaging camera or a solid-state imaging device; 2) the solid-state imaging device is an area sensor or a linear sensor; and 3) the method of extracting the contour of a moving object is performed by difference using memory (frame memory or line memory), or by difference between pixel pairs with different exposure times. Since the imaging device according to the embodiment of the present invention is configured with a combination of each of the cases, the following will focus on representative combinations and explain them with reference to the drawings, while the other combinations will be supplemented with words.

[0021] In the following description, the case of an imaging camera is used as an imaging device that outputs only moving object information in the first embodiment, and the functions of each block constituting the imaging device of the present invention will be described. In the case of the solid-state imaging device of the second embodiment, these blocks correspond to a case mounted on the solid-state imaging device, so the description of the functions of each block is the same.

[0022] <First embodiment> 1 shows blocks of an imaging device of the present invention, taking an imaging camera 1 as an example of a first embodiment of the imaging device of the present invention. The block configuration of the imaging camera 1 and signals of each block will be described below. The lens 2 configures an optical system that forms an image of a subject on a solid-state imaging device 4 in a camera housing 3. A pixel signal output 5 output from the solid-state imaging device 4 is split into two, one of which is input to a motion contour extraction unit 6 and the other to a masking circuit 10. The motion contour extraction unit 6 extracts motion contour information 7 from the pixel signal output 5. The extracted motion contour information 7 is input to a masking extraction unit 8. The masking extraction unit 8 performs a motion contour end determination corresponding to the outermost periphery of a moving object from the motion contour information 7, and creates and outputs a masking signal 9 that is output only as a moving object. The masking signal 9 is input to the masking circuit 10, where, together with the pixel signal output 5 that was also input, a masking pixel signal output 11 is created and output by the masking circuit. This masking pixel signal output is an output of only pixels corresponding to an image of a moving object, and results in an imaging device that outputs only moving object information, which is the objective of the present invention.

[0023] Normally, to output only moving object information, an area sensor is used as the solid-state imaging device 4, and after capturing the image in a frame memory, image processing is performed on a frame-by-frame basis, moving object detection is performed to identify the area of ​​movement, and only the moving object information is output. This has the drawback that the calculation system becomes complicated because a huge amount of image information is processed on a frame-by-frame basis, and high-speed calculations are performed in real time, resulting in huge power consumption. The most distinctive feature of the present invention is that the pixel signal output 5 from the solid-state imaging device 4 is processed for each readout line, and a masking pixel signal output 11 is generated in real time.

[0024] The masking circuit 10 receives the masking signal 9 and the pixel signal output 5 to generate a masking pixel signal output 11, which, as will be described later, is an AND circuit that obtains the AND (logical product) of the masking signal 9 and the pixel signal output 5, and outputs the masking pixel signal output 11, which is an output of only pixels corresponding to an image of a moving object, in real time from the imaging camera 1. In Fig. 1, in order to emphasize this real time, the pixel signal output 5 output from the solid-state imaging device 4 and input to the masking circuit 10 is shown in bold. Each block constituting the imaging camera 1 according to the first embodiment of the present invention shown in FIG. 1 will be described below in sequence with reference to the drawings.

[0025] <First embodiment-1> First, the configuration of the motion contour extraction unit 6 of the imaging camera 1 in FIG. 1 will be described with reference to FIG. 2. The location where the lens 2 forms an optical system for forming a subject image on the solid-state imaging device 4 in the camera housing 3 is the same as in FIG. 1. First, for ease of explanation, a linear sensor 4' will be used as the solid-state imaging device 4. (This will be referred to as the first embodiment-1.) For this reason, the lens 2 and the camera housing 3 will be referred to as a linear sensor camera 1'. In FIG. 1, the lens 2 and the camera housing 3 are configured to include the motion contour extraction unit 6, masking extraction unit 8, and masking circuit 10 to form the imaging camera 1, but here, for convenience, the lens 2 and the camera housing 3 with the linear sensor 4' inserted therein will be referred to as a linear sensor camera 1'. This is also true for the area sensor camera 1'' in FIG. 16. A pixel signal output 5 from the linear sensor 4' is split into two, one of which is input to a masking circuit 10 and the other to a motion contour extraction unit 6. In the motion contour extraction unit 6, the pixel signal output 5 is further split into two and taken into a line memory 12. The pixel signal output 5 coming out of the line memory 12 is the line output of the previous line, and a difference circuit 13 takes the difference between this and the current pixel signal output 5, and the difference circuit 14 obtains motion contour information 7. The time corresponding to one line is the cycle time of the linear sensor 4', and is denoted as Δt in the following drawings. In other words, the difference between adjacent lines is taken in the difference circuit 13. Here, the cycle time refers to the time required for one period to read out all the pixels of the solid-state imaging device 4, and is the time required for the solid-state imaging device 4 to read out one line in the linear sensor 4'.

[0026] The pixel signal output 5 represents a signal output corresponding to one line of the linear sensor that reads out all pixels of the linear sensor 4' and has idle feed before and after, and is synchronized for each line. The pixel difference that will be mentioned in the following explanation is the difference synchronized between these lines, and means the difference between pixels with different exposure timings that image the same position on the subject.

[0027] <Moving object recognition method in the first embodiment-1> Before explaining why the output from the absolute value circuit 14 is called the motion contour information 7, it is necessary to explain how a moving object is recognized using the linear sensor camera 1'. As an example of motion recognition using a linear sensor 4', we will explain the case where a linear sensor camera 1' monitoring the front part of a product display shelf is used to recognize the hand movements of customers performing tasks associated with bringing in and out products (picking up, returning them).

[0028] FIG. 3 shows a case where a customer's hand 16 reaching out to grab an item displayed on a product display shelf 15 is monitored by a linear sensor camera 1' installed on the upper edge of the product display shelf 15. The solid-state imaging device 4 used in this linear sensor camera 1' is a linear sensor 4', which observes the front portion of the product display shelf 15. A customer's hand 16 is shown on the front of the product display shelf 15 performing an operation to place an item in or out of the product display shelf 15. Since the pixels of the linear sensor 4' are arranged in a straight line, the range that can be captured is limited to the flat surface in front of the product display shelf 1. When explaining the application system as in FIG. 3, the linear sensor camera 1' is used to mean an imaging camera 1 using a linear sensor 4'. The same applies to the area sensor camera 4'' in FIG. 17.

[0029] This plane is called t=T 0 The dashed line extending from the linear sensor camera 1' at the timing t=T 0 At the timing t=T, the customer's hand 16 has not yet reached this plane. The customer's hand 16 approaches to pick up a product (not shown) from the product display shelf 15. For the sake of explanation, the part of the customer's hand 16 that crosses the dashed line extending from the linear sensor camera 1' (the part that can be photographed by the linear sensor camera 1') is determined at the timing t=T. 0 , t=T 1 , t=T 2 , t=T 3 , t=T 4 As shown in.

[0030] In FIG. 3, which part of the customer's hand 16 can be photographed by the linear sensor camera 1′ is determined at a timing t=T 0 , t=T 1 , t=T 2 , t=T 3 , t=T 4 The pixel output signals 5 from the linear sensor 4' at each timing are shown in Figures 4(a), (b), (c), (d) and (e). Here, the pixel signal output 5 is the output of the pixel row of the linear sensor 4', with the horizontal axis indicating the pixel position and the vertical axis indicating the pixel signal output at each pixel position. For reference, the reproduced image is indicated to which part of the customer's hand 16 the image corresponds.

[0031] A feature of the linear sensor 4' is that when there is no movement, the output of the background remains constant over time, and for convenience, this is set to zero (see FIG. 4(a)). When there is movement (here, the movement of the customer's hand 16), the output of the linear sensor 4' changes. 0 At this timing, an image is captured on the plane indicated by the dashed line extending from the linear sensor camera 4, and changes occur in the pixel output waveform in response to the parts of the customer's hand 16 that cross this plane (see Figs. 4(b), (c), (d), and (e)).

[0032] A reconstructed image of the customer's hand 16 can be obtained by arranging the pixel signal outputs of the linear sensor 4' in chronological order. Figure 4 shows a reconstructed image obtained from pixel output waveforms corresponding to parts of the customer's hand 16. The reconstructed image expands and contracts depending on the speed at which the customer's hand 16 moves, but since the aspect ratio (length and width) of a human hand is approximately constant, it is easy to correct the reconstructed image. In addition, even if the customer's hand 16 grabs a product from the product display shelf 15 and comes out, that is, even if it moves in a direction away from the product display shelf 15, a reconstructed image can be easily obtained. The above is an explanation of how a moving object is recognized using the linear sensor camera 1'.

[0033] <Motion Contour Extraction Method in First Embodiment-1 (No Background Light)> Fig. 2 shows a signal processing circuit that processes pixel signal output 5 of linear sensor camera 1' to extract only the area corresponding to the customer's hand 16 and set the background output to zero. The signal processing circuit shown in Fig. 2 focuses an image of the subject on linear sensor 4' inside camera housing 3 via lens 2, and outputs the obtained pixel signal output 5 from linear sensor camera 1'. From the output pixel signal output 5, motion contour information 7 is obtained in motion contour extraction unit 6.

[0034] The pixel signal output 5 is taken into a line memory 12. The pixel signal output 5 coming out of the line memory 12 is the line output of the previous line, and a difference circuit 13 takes the difference between this and the current pixel signal output 5, and moving contour information 7 is obtained via an absolute value circuit 14. The time corresponding to one line is the cycle time Δt of the linear sensor 4' (the time it takes to read one line). That is, the difference between adjacent lines is taken in the difference circuit 13.

[0035] Of course, the line memory 12 does not have to be a line memory equivalent to a one-line delay, and a delay equivalent to two lines may be generated and the difference between every other line may be taken. The amount of line delay is determined by the speed at which the customer's hand 16 is inserted and removed and the cycle time of the linear sensor 4'. For the sake of simplicity of the system, the delay time of the line memory is generally set to an integer multiple of the normal cycle time, but hereinafter, an explanation will be given of the case of a difference between adjacent lines equivalent to a one-line delay (Δt).

[0036] For example, in the case of a difference between lines every other line, which corresponds to a delay of two lines (2Δt), the line memory 12 in FIG. 2 needs a capacity for two lines to perform a delay of two lines. However, in general, in the case where a delay of two lines is required in a line memory, the speed of the moving object is slow (here, the customer's hand 16 moves in and out slowly), so the timing of monitoring the image can be slow. In other words, an image every other line can be used, and if a mechanism (not shown) is added that discards the data of the image signal output 5 of the line corresponding to the every other line, the line memory 12 only needs to store the pixel signal output of two lines before, and only needs a capacity for one line. This is the same for a delay of three or more lines and for the frame memory 22 in FIG. 16.

[0037] The line difference method shown in FIG. 2 will be explained with reference to FIG. 5 based on each signal waveform. For simplicity, the timing of the position of the customer's hand 16 will be first assumed to be t=T 0 , t=T 1 , t=T 3This will be explained using the case of First, t=T shown in Fig. 4(a) 0 At this timing, as shown in FIG. 5(a), there is no output from the customer's hand 16 in the waveform of the pixel signal output 5 of the linear sensor camera 1'. 0-Δt 5(a'), there is no output from the customer's hand 16. The difference between the two is taken by the difference circuit 13, but it is zero, and even after passing through the absolute value circuit 14, the motion contour information 7 remains zero as shown in FIG. 5(a").

[0038] Next, t=T shown in Figure 4(b) 1 At this timing, as shown in FIG. 5(b), the output of the customer's hand 16 appears in the waveform of the pixel signal output 5. At this time, the previous line (T 0-Δt ), the output of the customer's hand 16 appears in the waveform of the pixel signal output from the sensor 13, as shown in Figure 5(b'). The two waveforms differ slightly depending on the position of the customer's hand 16, and generally a difference appears in the outline of the hand output. For this reason, when the difference between the two is taken in the difference circuit 13, an output similar to a differentiated waveform appears in this outline. After passing through the absolute value circuit 14, the motion outline information 7 becomes as shown in Figure 5(b"). This has a shape similar to the differentiated waveform of the pixel signal output 5.

[0039] Here, the signal that comes out through the absolute value circuit 14 and is actually called the pixel difference absolute value is called motion contour information 7 because of the calculation method that synchronizes and calculates the difference between the pixel signal output of the previous line and the pixel signal output at the current time. In other words, it has a shape like a differential waveform, and since a stationary subject has the same pixel signal output waveform, taking the difference becomes zero. On the other hand, a moving subject has a different pixel signal output waveform, so a signal remains when taking the difference. This remaining signal is often the contour (periphery) of the moving object, so it is called motion contour information 7. Even if the center of a moving object (for example the back of the customer's hand 16) moves, the change will be in the same pattern, and even if the difference is calculated, the signal is small, making it difficult to determine that it is a moving object.

[0040] In a similar sequence, customer hand 16 is at t=T in Figure 4(d).3 At this timing, the output of the entire customer's hand 16 appears as shown in FIG. 5(d). At that time, the previous line (T 3-Δt 15(d'), the waveforms of the customer's hand 16 as a whole are subtly different. When the difference between the two is taken by the difference circuit 13, in this case not only the contours (periphery) but also the areas between the fingers (contours of the fingers) are subtly visible. After passing through the absolute value circuit 14, the motion contour information 7 becomes as shown in FIG. 5(d").

[0041] The motion contour information 7 shown in FIG. 5(d") has a complex shape, but this is because the moving object is a hand, which is a complex moving object with gaps between the fingers, and the motion contour information 7 is also complex. When it comes to the fingertip in FIG. 4(b) or the wrist in FIG. 4(e), the motion contour information 7 becomes simpler.

[0042] <Motion Contour Extraction Method in First Embodiment-1 (with Background Light)> For simplicity, a case where there is no background light has been described in Figures 4 and 5, but in reality, the background does not become zero due to the lighting in the room, and the pixel signal output waveform of the background light of the linear sensor camera 1' will actually be as shown in Figure 6(a). However, the background is stationary, and in the short period in which the movement of the customer's hand is captured, it is considered to be a constant pattern. Therefore, the actual pixel output signal of the linear sensor camera 1' at each timing corresponding to Figures 4(a) to (e) will be like the output waveforms shown in Figures 6(a) to (e).

[0043] It should be noted here that the magnitude of the pixel signal output at the position corresponding to the customer's hand 16 is not superimposed on the output waveform of the stationary background, but the pixel signal output of the background at the position of the hand is blocked by the customer's hand 16, the pixel signal output of the background corresponding to the hand becomes zero, and the pixel signal output corresponding to the part of the customer's hand 16 appears. This situation is represented by dotted lines in Figures 6(b) to (e).

[0044] Next, in the case of background light, t=T 1 , t=T3 The timing will be explained with reference to FIG. t=T 1 At this timing, as shown in FIG. 7(b), the output of the customer's hand 16 appears in the waveform of the pixel signal output 5. At this time, the previous line (T 1-Δt ), the output of the customer's hand 16 appears as shown in Figure 7(b'), and the two waveforms are subtly different. For this reason, when the difference between the two is taken in the difference circuit 13, the background light area disappears as it is the same, and an output like a differentiated waveform appears in the outline of the hand, and after passing through the absolute value circuit 14, the motion outline information 7 corresponding to the absolute value of the pixel difference becomes as shown in Figure 7(b"). This is similar to the case where there is no background light as shown in Figure 5(b").

[0045] In a similar sequence, t=T 3 At the timing of T , the whole hand appears as shown in FIG. 7(d), and the previous line (T 3-Δt ) is the output of the entire hand, as shown in Figure 7(d'), and the two are subtly different. When the difference between the two is taken using differential circuit 13, in addition to the contour parts, the parts between the fingers also subtly appear, and the motion contour information 7 corresponding to the absolute pixel difference values ​​looks like Figure 7(d"), which is similar to the case of no background light in Figure 5(d")

[0046] The above is the timing t=T 1 , t=T 3 We have explained the timing t=T shown in Figure 2, including other timings. 0 , t=T 1 , t=T 2 , t=T 3 , t=T 4 The motion contour information 7 corresponding to the waveforms of the pixel signal output 5 shown in Figures 6(a), (b), (c), (d), and (e) is shown in Figures 8(a), (b), (c), (d), and (e).

[0047] <Masking extraction method in the first embodiment-1> 9, the masking extraction unit 8 for extracting the masking signal 9 from the motion contour information 7 will be described, which is included in the signal processing circuit that performs signal processing on the output of the linear sensor camera 1' of the imaging camera 1 according to the first embodiment-1 of the present invention, extracts only the area corresponding to the customer's hand 16, and sets the background output to zero. The motion contour extraction unit 6 has been described separately for the case with and without background light, but the motion contour information 7 from the motion contour extraction unit 6 has the same waveform in both cases, as shown in Figs. 5(a"), (b"), (d") and 7(a"), (b"), (d"), so the masking extraction unit 8 will be described without making a distinction.

[0048] In the motion contour extraction unit 6, motion contour information 7 is obtained by calculating the difference between the timing at which the movement of the customer's hand 16 occurs and the timing of the previous line. Based on this motion contour information 7, the masking extraction unit 8 generates a masking signal 9 for extracting the timing at which the movement of the customer's hand 16 occurs (meaning both the time when the movement occurs and the output timing of pixels corresponding to the place where the movement occurs) and the pixel output timing corresponding to the movement.

[0049] Next, the configuration of the masking extraction unit 8 will be described. From the motion contour information 7, the motion contour end determination unit 17 extracts the outer contour portion 7' ​​of the customer's hand 16. As explained in FIG. 7(d"), the difference circuit 13 extracts not only the contour portion of the hand but also the portion between the fingers, and the motion contour information 7 becomes a complex waveform. The motion contour end determination unit 17 tracks the progress of the waveform of the motion contour information 7, making it possible to extract the outer contour of the customer's hand 16.

[0050] The method is explained below, assuming that the customer's hand 16 can be captured as a series of movements from when it enters the flat front surface of product display shelf 15 monitored by linear sensor camera 1' until when it is removed. This assumption holds true because, in general, a customer's hand 16 performs a series of movements to insert the hand, grab the product, and remove it in order to grab a product (not shown) on product display shelf 15 (because the customer does not stop inside the display shelf while gripping a product, and does not insert anything other than the hand (head, foot, etc.)).

[0051] The linear sensor camera 1' captures the tip of the customer's hand 16, gradually expanding to the width of the hand. As shown in Fig. 4(c), the area between the fingers also appears along the way, but it remains inside the width of the hand, and the outer contour of the hand can be traced. As shown in Fig. 4(d), a gap appears between the thumb and the hand temporarily, making it impossible to distinguish it from a new moving object, but as shown in Fig. 4(e), it can be recognized as a connected, single object. Since the size of a customer's hand 16 is small compared to the size of the product display shelf 15 observed by the linear sensor camera 1', if a criterion is set that a new moving object (fingertip) occurring near the hand belongs to the hand, it becomes easy to determine the outermost contour of the hand (determine the end of the moving contour). The motion contour end information 18 thus obtained by the motion contour end determination unit 17 and corresponding to the outer contour of the customer's hand 16 is compared with a threshold value 19 in the masking signal generation circuit 20 and binarized to obtain a masking signal 9, which is then output from the masking extraction unit 8.

[0052] The signal processing in the masking extraction unit 8 shown in FIG. 9 will be explained with reference to FIG. 10 based on each signal waveform. The timing of the position of the customer's hand 16 is t=T 1 , t=T 3 This is the case, which corresponds to the motion contour information 7. Here, a case where the motion contour information 7 is created with background light will be described.

[0053] Customer's hand 16 is t=T 1 At this point, the motion contour information 7 becomes as shown in Fig. 10(b). The motion contour end information 18 at this time becomes as shown in Fig. 10(b'). The motion contour end information 18 is the end of the contour of one moving object, and each piece of information is paired, with the two pieces of information determining the area of ​​one moving object. The motion contour end information 18 is compared with a threshold value 19 in a masking signal generating circuit 20, which sets a masking area and binarizes it. This masking area refers to the area of ​​a single moving object. If the motion contour end information 18 exceeds the threshold value 19 and is recognized as an area that should be masked, a value of 1 is assigned, otherwise a value of 0 is assigned, and a binarized masking signal 9 is obtained. The masking signal 9 obtained based on the motion contour end information 18 in Figure 10(b') looks like Figure 10(b''').

[0054] The period during which the masking signal 9 is 1 corresponds to the timing when the movement of the customer's hand 16 occurs and the pixel output timing corresponding to the position of the customer's hand 16, and is the period that should be masked from the pixel signal output 5. In other words, the pixel signal output 5 during the timing when the masking signal 9 is generated (is 1) corresponds to observing only the movement of the customer's hand 16.

[0055] In a similar sequence, customer hand 16 is at t=T 3 At this timing, the motion contour information 7 in Fig. 10(d) is the same as in Fig. 8(d), and the motion contour end information 18 is deleted as the part between the fingers is determined not to be an end, resulting in the information shown in Fig. 10(d'). At this time, the gap between the thumb and the other fingers has not yet been determined to be between the fingers, and the thumb is determined to be the appearance of a new moving object. At the timing of the base of the thumb, both are determined to be the same moving object. The masking signal 9 obtained based on the motion contour end information 18 in Fig. 10(d') is as shown in Fig. 10(d"').

[0056] Here, the gap between the fingers is judged based on the first edge information determined from the fingertip at the tip of the customer's hand 16, and any new moving object occurring in the vicinity of this edge corresponding to the thickness of the finger is defined as another fingertip and recognized as one hand. If this is expanded further and a criterion is set that any new moving object (fingertip) occurring in the vicinity corresponding to the width of the hand belongs to the hand, then even the thumb, which was judged to be a new moving object in Figure 10(d'''), will not be judged as a new moving object, and it will be easier to determine the outermost contour of the hand. This is the criterion mentioned in paragraph number 51.

[0057] <Masking method in the first embodiment-1> The signal processing circuit processes the output of the linear sensor camera 1' of the imaging camera 1 according to the first embodiment of the present invention, extracts only the area corresponding to the customer's hand 16, and makes the background output zero. The final masking signal 9 is used in the signal processing circuit to obtain a masking pixel signal output 11 that makes the background zero and outputs only the area corresponding to the customer's hand 16 from the pixel signal output 5. The masking circuit will be described with reference to Fig. 11. Since the motion contour information 7 has the same waveform whether there is background light or not, the description will be made without distinguishing between the two. The period during which the binarized masking signal 9 is 1 corresponds to the timing when the movement of the customer's hand 16 occurs and the pixel output timing corresponding to the position of the customer's hand 16, and is the period that should be masked from the pixel signal output 5.

[0058] In the masking circuit 10, in order to extract the timing of the movement of the customer's hand 16 and the corresponding pixel output timing, it is sufficient to output the pixel signal output 5 from the linear sensor camera 1' only when the binarized masking signal 9 is 1. When the masking signal is 0, the pixel signal output 5 is not output. To achieve this, the masking circuit 10 needs only to AND the two. It is composed of a simple AND circuit. The masking circuit 10 receives two inputs: the masking signal 9 from the masking extraction unit 8 and the pixel signal output 5 from the solid-state imaging device 4, and the output of the masking circuit 10 is the masking pixel signal output 11. In this example, the background disappears and only the part corresponding to the customer's hand 16 is output.

[0059] At this time, if a delay occurs due to the calculation processing time between the motion contour extraction unit 6 and the masking extraction unit 8, causing a problem in extracting only the movement of the customer's hand 16 from the pixel signal output 5, the pixel signal output 5 can be delayed by a time equivalent to this calculation processing time in a delay circuit 21 to obtain a masking pixel signal output 11 in the masking circuit 10 by synchronizing (adjusting the timing). If the delay circuit 21 is small enough that no problem occurs due to the delay in the calculation processing time, there is no need to insert it.

[0060] The signal processing in the masking circuit 10 shown in Fig. 11 will be explained with reference to Fig. 12 based on each signal waveform. The timing of the position of the customer's hand 16 is t=T 1 , t=T 3 This corresponds to the pixel signal output 5 and the masking signal 9. Here, a case where the motion contour information 7 is created with background light will be described. Customer's hand 16 is t=T 1 The pixel signal output 5 and masking signal 9 at the timing are shown in FIG. 12(b) and FIG. 12(b'''), respectively, and the masking pixel signal output 11 is shown in FIG. 12(b''"). That is, the pixel signal output 5 at the timing when the masking signal 9 is generated (is 1) corresponds to observing only the movement of the customer's hand 16. The masking pixel signal output 11 shown in FIG. 12(b''") as 1 The timing is 16 signals from the customer's hand.

[0061] In a similar sequence, customer hand 16 is at t=T in Figure 4. 3 At this timing t=T 3 10(d'''), the masking signal is shown in FIG. 12(d'''), and the masking pixel output 11 is shown in FIG. 12(d''"), and the time t=T 3 The timing is 16 signals from the customer's hand.

[0062] The above is the timing t=T 1 , t=T 3 We have explained the timing t=T shown in Figure 3, including other timings. 0 , t=T 1 , t=T 2 , t=T 3 , t=T 4The motion contour information 7 for the waveforms of the pixel signal output 5 shown in Figs. 6(a), (b), (c), (d) and (e) corresponding to the above is shown in Figs. 8(a), (b), (c), (d) and (e). Based on this, the motion contour end information 18 and the masking signal 9 are shown in Figs. 13(a), (b), (c), (d) and (e).

[0063] Timing t=T shown in Figure 3 0 , t=T 1 , t=T 2 , t=T 3 , t=T 4 Based on the waveforms of the pixel signal output 5 shown in Figures 6(a), (b), (c), (d), and (e) corresponding to the waveforms of the masking signal 9 shown in Figures 13(a), (b), (c), (d), and (e), the masking pixel signal output 11 obtained by the masking circuit 10 is as shown in the right diagrams of Figures 14(a), (b), (c), (d), and (e) at the respective timings. The waveform of the original pixel signal output 5 is also shown in the left diagrams. Also, the timing t=T 0 , t=T 1 , t=T 2 , t=T 3 , t=T 4 corresponds to the dashed area on the customer's hand 16.

[0064] The case of paragraph 56, in which a new moving object (fingertip) occurring in the vicinity of the width of the hand is determined to belong to the hand, is explained with reference to FIG. 15. 0 , t=T 1 , t=T 2 , t=T 3 , t=T 4Fig. 8(a), (b), (c), (d), and (e) show what the motion contour information 7 will be like for the waveforms of the pixel signal output 5 shown in Fig. 6(a), (b), (c), (d), and (e) corresponding to the above, and these are shown again in the left diagrams of Fig. 15(a), (b), (c), (d), and (e). The waveform of the motion contour end information 18 that has been determined to be an end by the motion contour end determination unit 17 based on the waveform of the motion contour information 7 according to the new determination criterion is one of the differential waveforms at both ends of several differential waveforms in the waveform of the motion contour information shown on the left side of Fig. 15(a), (b), (c), (d), and (e) at each timing.

[0065] Based on the waveform of the motion contour end information 18 obtained by the motion contour end determination unit 17 using the new judgment criteria, the masking signal 9 that is determined to be 1 from the point in time when the masking signal exceeds a threshold value 19 by the masking extraction unit 8 is the masking signal waveform shown in the center of Figures 15(a), (b), (c), (d) and (e). The masking signal waveform corresponds to the size of the customer's hand 16.

[0066] Timing t=T shown in Figure 4 0 , t=T 1 , t=T 2 , t=T 3 , t=T 4 Based on the waveforms of the pixel signal output 5 shown in Figures 6(a), (b), (c), (d), and (e) corresponding to the timing t, and the waveforms of the masking signal 9 shown in Figures 15(a), (b), (c), (d), and (e), the masking pixel signal output 11 obtained by the masking circuit 10 is as shown in the right diagrams of Figures 15(a), (b), (c), (d), and (e) at the respective timings. Comparing these waveforms with the waveforms of the masking pixel signal output 11 shown in the right diagrams of Figures 14(a), (b), (c), (d), and (e), it is clear that at timing t=T 3There is a difference between the right diagram of FIG. 15(d) and the right diagram of FIG. 14(d), which correspond to the above. That is, in the waveform of the masking signal 9 obtained using the new judgment criteria, the gap between the fingers is recognized as part of the customer's hand 16 and adopted as an image, so the masking signal is 1. Therefore, in the masking pixel signal output 11 of FIG. 15(d), the background image appears in the area between the fingers. This is different from the masking pixel signal output 11 shown in FIG. 14(d), which was obtained in FIG. 13(d) where the gap between the fingers also had a masking signal of zero.

[0067] <First embodiment-2> Next, a case will be described in which an area sensor 4" is used for the solid-state imaging device 4 of the imaging camera 1 in FIG. 1 (this will be referred to as first embodiment-2). FIG. 16 shows the configuration of the motion contour extraction unit 6 in FIG. 1. The location where the lens 2 configures the optical system that forms a subject image on the solid-state imaging device 4 in the camera housing 3 is the same as in FIG. 1. Since the area sensor 4" is used as the solid-state imaging device 4, the lens 2 and the camera housing unit 3 will be referred to as an area sensor camera 1". A pixel signal output 5 from the area sensor 4" is divided into two, one of which is input to a masking circuit 10 and the other is input to a motion contour extraction unit 6. In the motion contour extraction unit 6, the pixel signal output 5 is further divided into two and taken into a frame memory 22. The pixel signal output 5 coming out of the frame memory 22 is the scanning line output of the previous frame, and a difference is taken by a difference circuit 13 between this and the pixel signal output 5 corresponding to the relevant scanning line output of the current frame, and motion contour information 7 is obtained via an absolute value circuit 14. The time corresponding to one frame is the cycle time of the area sensor 4", and is denoted as ΔT in the following drawings. In other words, the difference between pixels corresponding to the same location of the subject between adjacent frames is taken by the difference circuit 13. Here, the cycle time is the time required for one period to read out all the pixels of the solid-state imaging device 4, and is the time required for the solid-state imaging device 4 to read out one frame in the area sensor 4". Here, the motion contour extraction unit 6 uses a frame memory 22 instead of the line memory 12 of the first embodiment-1 shown in FIG. 2, but both have the same components as image memories.

[0068] To aid in understanding, the pixel signal output 5 from the area sensor 4" will be explained below. A single image captured by the two-dimensionally arranged pixels in the area sensor 4" is called a frame. The pixel signal output 5 is composed of one frame of signal output obtained by reading out all the pixels in the area sensor 4" and having blank spaces before and after. The screen configuration of one frame is such that a single line of signals called a scanning line is displayed sequentially from top to bottom, like a TV screen, to form an area-shaped screen. When displaying this single frame, the frames are synchronized, and in the case of a still image, it is displayed in the same position every time, so the screen is displayed in a still state. The pixel signal output 5 coming out of the frame memory 22 is the scanning line output of the previous frame, and the difference is taken from the pixel signal output 5 corresponding to the relevant scanning line output of the current frame. Since there is synchronization between frames, the difference is between pixel outputs at the same pixel position (same position on the subject) on the line corresponding to the same scanning line position with a time difference of one frame. The cycle time of the linear sensor 4' (the time required to read out one line) is represented as Δt (line period), and the cycle time of the area sensor 4'' (the time required to read out one frame) is represented as ΔT (frame period).

[0069] Of course, the images stored in the frame memory 22 do not have to be frame images corresponding to a delay of one frame cycle, and a delay corresponding to two frame cycles may be generated to take the difference between skipping one frame. For the sake of simplicity of the system, the delay time of the images stored in the frame memory 22 is generally set to an integer multiple of the normal cycle time, but hereinafter, the case of a difference between adjacent frames corresponding to a delay of one frame (ΔT) will be described. The amount of this frame delay is determined by the balance between the speed at which the customer's hand 16 is inserted and removed and the cycle time of the area sensor 4". This is the same as the situation with the line memory 12 when a linear sensor 4' is used in the solid-state imaging device 4. Therefore, the situation described in paragraph number 36 also applies to the frame memory 22.

[0070] When focusing on one scanning line, whether it is pixel signal output 5 from area sensor 4" or pixel signal output 5 from linear sensor 4', the difference between the pixel outputs being compared differs in whether the time change corresponds to the difference between one frame or the difference between one line (Δt), but it does not change the fact that we are looking at the time change at the same location on the subject. In other words, the greatest feature of the present invention, that is, processing pixel signal output 5 from solid-state imaging device 4 for each readout line and creating masking pixel signal output 11 in real time, is the same whether it is an area sensor or a linear sensor.

[0071] <Moving object recognition method in the first embodiment-2> We will now explain how to recognize moving objects using the area sensor 4". As an example of moving object recognition using the area sensor 4", we will explain the case where an area sensor camera 1" that monitors the front part of a product display shelf, as in Figure 3, recognizes the hand movements of customers performing tasks associated with the ingress and egress of products (picking up, returning products).

[0072] FIG. 17 shows a case where an area sensor camera 1" installed at the upper front of the product display shelf 15 monitors a customer's hand 16 reaching out to grab a product displayed on the product display shelf 15. The solid-state imaging device 4 used in this area sensor camera 1" is an area sensor 4", which observes the front portion of the product display shelf 15. A customer's hand 16 is shown in front of the product display shelf 15 performing an action related to placing or removing a product. Since the pixels of the area sensor 4" are arranged in an area, the area that can be photographed is such that the entire surface of the customer's hand in front of the product display shelf 1 can be photographed in one frame. Therefore, in FIG. 3, t=T 0 ~T 4 In Figure 17, the scanning line S of the area sensor 4" is 0 ~S 4 The pixel signal output on each scanning line is as shown in Fig. 18(a)-(e) (background light is assumed to be zero for convenience), and the scanning line corresponds to which part of the customer's hand 16, which is a moving image, is shown in the moving image on the right side of Fig. 18. The horizontal axis of the scanning line pixel signal output waveforms in Fig. 18(a)-(e) is the scanning line (S 0 ~S4 ) and the vertical axis corresponds to the pixel position on each scanning line, and the vertical axis corresponds to the pixel signal output on each scanning line.

[0073] A feature of the area sensor 4" is that when there is no movement, the output of the background image is constant, but when there is movement (here, the movement of the customer's hand 16), the output of the area sensor 4" changes. Corresponding to the movement of the hand, the scanning line pixel signal output changes (see Figs. 18(b), (c), (d), and (e)). In this case, the motion contour information 7 obtained by the motion contour extraction unit 6 in Fig. 16 will be described below.

[0074] <Motion Contour Extraction Method in First Embodiment-2 (Case with Background Light)> In reality, the background does not become zero due to the lighting in the room, so the case with background light will be described. The pixel signal output of each scanning line of the area sensor 4" is shown in the left diagram of Fig. 19(a) to (e). The background image is stationary, and in the short period in which the movement of the customer's hand 16 is captured, it is considered to be a constant pattern in each frame. The motion contour information 7 obtained by the motion contour extraction unit 6 of Fig. 16 is obtained by taking the difference between the pixel signal output of each scanning line of the area sensor camera 1" shown in the left diagram of Fig. 19(a) to (e) and the pixel signal output of the previous frame in the difference circuit 13, and taking the absolute value in the absolute value circuit 14, so that it becomes a waveform similar to that of Fig. 8(a) to (e), and the motion contour information 7 becomes an output waveform as shown in the right diagram of Fig. 19(a) to (e). Both are differential waveforms, and the time difference is the pixel signal output difference between the line period Δt or the frame period (ΔT). For convenience, the motion contour information shown in the right diagrams of FIGS. 19(a) to (e) is set to be the same as the motion contour information of the first embodiment-1 shown in the right diagrams of FIGS. 8(a) to (e).

[0075] The magnitude of the pixel signal output at the position corresponding to the customer's hand 16 is not superimposed on the output waveform of the background, but the pixel signal output of the background at the position of the hand is blocked by the customer's hand 16, the pixel signal output of the background corresponding to the hand becomes zero, and the pixel signal output corresponding to the part of the customer's hand 16 appears, just like in the case of linear sensor 4'. This situation is similarly represented by dotted lines in Figures 19(b) to (e).

[0076] <Masking extraction method in the first embodiment-2> Since the motion contour information 7 from each scanning line pixel signal output of the area sensor camera 1″ is similar to the motion contour information 7 of the linear sensor camera 1′, the masking extraction unit 8 is the same as that shown in FIG. 9. The explanation of the masking extraction unit 8 for extracting the masking signal 9 from the motion contour information 7 is the same as that of the linear sensor camera 1′.

[0077] In paragraph number 56, a case where a criterion is set that a new moving object (fingertip) occurring in the vicinity of the width of the hand belongs to the hand is described with reference to FIG. 20. The right diagrams of FIG. 19(a), (b), (c), (d), and (e) show how the motion contour information 7 looks with respect to the waveforms of the pixel signal output 5 on each scanning line shown in the left diagrams of FIG. 19(a), (b), (c), (d), and (e), and this is shown again in the left diagrams of FIG. 20(a), (b), (c), (d), and (e). The waveform of the motion contour end information 18 that is determined to be an end by the motion contour end determination unit 17 based on the waveform of the motion contour information 7 according to the new criterion is the differential waveform at both ends among several differential waveforms in the waveform of the motion contour information shown on the left diagrams of FIG. 20(a), (b), (c), (d), and (e) at each timing.

[0078] Based on the waveform of the motion contour end information 18 obtained by the motion contour end determination unit 17 using the new judgment criteria, the masking signal 9 that is determined to be 1 from the point in time when the masking signal exceeds a threshold value 19 by the masking extraction unit 8 is the masking signal waveform shown in the center of Figures 20(a), (b), (c), (d) and (e). The masking signal waveform corresponds to the size of the customer's hand 16.

[0079] <Masking method in the first embodiment-2> The signal processing circuit processes the output of area sensor camera 1" to extract only the area corresponding to customer's hand 16 and makes the background output zero. The final masking signal 9 is used in the signal processing circuit which makes the background zero and obtains masking pixel signal output 11 which outputs only the area corresponding to customer's hand 16 from pixel signal output 5. The masking circuit 10 in FIG. 11 can be used as is for the method. The case where there is background light will be explained.

[0080] The scanning line S of the area sensor 4" of the area sensor camera 1" in FIG. 0 ~S 4 The pixel signal output on each scanning line corresponds to the left diagram (scanning line pixel signal output) of FIG. 19(a) to (e). The horizontal axis of the scanning line pixel signal output waveform of FIG. 19(a) to (e) is the scanning line (S 0 ~S 4 ), and the vertical axis corresponds to the pixel position on each scanning line, and the vertical axis corresponds to the pixel signal output on each scanning line. From this, motion contour information 7 is extracted through motion contour extraction unit 6 shown in Fig. 16, and the masking signal 9 obtained by masking extraction unit 8 is shown in the center of Figs. 20(a), (b), (c), (d), and (e) as described above. The period when the binarized masking signal 9 is 1 corresponds to the timing when the movement of the customer's hand 16 occurs and the pixel output timing corresponding to the position of the customer's hand 16, and is the period that should be masked from the pixel signal output 5.

[0081] In the masking circuit 10, in order to extract the timing at which the movement of the customer's hand 16 occurs and the corresponding pixel output timing, it is sufficient to pass the signal through an AND circuit that outputs pixel signal output 5 from area sensor camera 1" only when the binarized masking signal 9 is 1, and does not output pixel signal output 5 when the masking signal is 0. The output of the masking circuit 10 that has done this is masking pixel signal output 11, and is shown in the right diagrams of Figures 20(a), (b), (c), (d), and (e). This is similar to the linear sensor camera 1', in that the background disappears and only the area corresponding to the customer's hand 16 is output.

[0082] <First embodiment-3> Another configuration of the motion contour extraction unit 6 and the masking extraction unit 8 of the imaging camera 1 in FIG. 1 will be described with reference to FIG. 21. Another configuration using a linear sensor 4' as the solid-state imaging device 4 is called the first embodiment-3. The configuration of the linear sensor camera 1' is the same as that of the first embodiment-1 in FIG. 2, and the pixel signal output 5 from the linear sensor 4' is divided into two, one of which is input to the masking circuit 10, while the other is input to the motion contour extraction unit 6'. In the motion contour extraction unit 6', the pixel signal output 5 is further divided into two and taken into the line storage unit 24 via the switch circuit 23. The pixel signal output 5 coming out of the line storage unit 24 is the line output (called the line storage unit output) at the timing when it is taken into the line storage unit 24, and the difference between the pixel signal output 5 at the current time is taken by the difference circuit 13, and the motion contour information 7' ​​is obtained via the absolute value circuit 14. The line storage unit 24 stores one line of pixel signal output 5, like the line memory 12, but because the signal is taken in via the switch circuit 23, it stores one line of pixel signal output 5 at the timing when the switch is turned ON. Another difference from the line memory 12 of the first embodiment-1 in FIG. 2 is that the line storage unit 24 repeatedly outputs the stored one line of pixel signal output 5.

[0083] Here, the line storage unit output (pixel signal output 5) which is the output of the line storage unit 24 is the pixel signal output 5 at t=T 0 1 shows the waveform of the pixel signal output of the background light of the linear sensor camera 4' at the timing of FIG. 1. When storing the pixel signal output 5 of the background light where there is no output from the customer's hand 16, the switch circuit 23 turns on and takes it into the line storage unit 24, and when there is a pixel signal output 5 where there is an output from the customer's hand 16, the switch is turned off and it is not taken into the line storage unit 24. In this way, the difference circuit 13 can remove the background light. The motion contour extraction unit 6' has a switch circuit 23 added as a new component, and the line memory output of the background light (pixel signal output 5) is stored in the line memory unit 24 and used repeatedly, so it is distinguished as motion contour extraction unit 6'. The difference circuit 13 outputs positive or negative depending on the magnitude relationship of the pixel signal outputs between the lines being compared. For this reason, the output of the difference circuit 13 is input into the absolute value circuit 14 to obtain motion contour information 7'. The output of the motion contour extraction unit 6' is also distinguished as motion contour information 7'.

[0084] The motion contour information 7' ​​obtained by the motion contour extraction unit 6' is input to a masking extraction unit 8'. In the masking extraction unit 8', the motion contour information 7 is compared with a threshold value 19 in a masking signal generation circuit 20 and binarized to obtain a masking signal 9. The masking extraction unit 8' does not have the motion contour end determination unit 17 and motion contour end information 18 that were present in the masking extraction unit 8 in Fig. 9. For this reason, it is referred to as a masking extraction unit 8'.

[0085] The masking signal 9, which is the output of the masking signal generating circuit 20, is a signal for extracting the timing at which the movement of the customer's hand 16 occurs and the corresponding pixel output timing. Furthermore, from this masking signal 9 and the pixel signal output 5, which is the output from the linear sensor camera 4', a masking circuit 10 obtains a masking pixel signal output 11 that corresponds only to the movement of the customer's hand 16.

[0086] At this time, if a problem occurs in extracting only the movement of the customer's hand 16 from the pixel signal output 5 due to the calculation processing time of the motion contour extraction unit 6' and the masking extraction unit 8', the pixel signal output 5 can be delayed by the delay circuit 21 present in the masking circuit 10 of FIG. 11 for a time equivalent to this calculation time, and the masking pixel signal output 11 can be obtained in the masking circuit 10. The term "masking" used here does not mean hiding data as is commonly used, but rather means hiding everything other than data and highlighting and exposing the data. That is, in the masking circuit 10, when the binarized masking signal 9 is 1, a pixel signal output 5 is output, and when it is 0, no pixel signal output 5 is output. The above situation also has a significance for using logical product (AND) in the masking circuit 10, as described in paragraph number 16. The purpose of this application is to hide unnecessary background, eliminate unnecessary data processing, extract essential data (e.g., customer's hand 16), and simplify signal processing.

[0087] To further explain the switches of the switch circuit 23, when the masking signal 9 is 0, there is no movement of the customer's hand 16 and this corresponds to background light, so the switch circuit stores the line as ON, and when the masking signal 9 is 1, there is movement of the customer's hand 16 and this corresponds to the movement of the customer's hand 16 that should be captured, so the switch circuit does not store the line as OFF. Since the background light rarely changes, there is no need to use the masking signal 9 every time to control the ON / OFF of the switch circuit 23; it is sufficient to turn on the switch circuit 23 by periodic sampling and capture background light with no movement.

[0088] The masking signal 9 has a waveform of 1 and 0 within one line, and the masking circuit 10 extracts a moving object (customer's hand 16) from the pixel signal output 5. However, when the masking signal 9 that controls the switch circuit 23 is said to be 0 or 1, it means that the masking signal is always 0 within one line of the masking signal, and is said to be 1 if there is even the slightest timing when the masking signal becomes 1. To perform this operation, a selection unit 25 is provided between the masking signal 9 and the switch circuit 23. The selection unit 25 also has a function like a timer that performs the above-mentioned periodic sampling, and periodically stores lines of background light with little change.

[0089] The operation of the motion contour extraction unit 6', masking extraction unit 8', and masking circuit 10 shown in Fig. 21 will be explained with reference to Fig. 22 based on the signal waveforms. The timing of the position of the customer's hand 16 is t=T 0 , t=T 1 , t=T 3 This is the case. First, t=T, which corresponds to the background image. 0At the timing of t=T, as shown in FIG. 6(a), the same waveform as the pixel signal output 5 of the background light of the linear sensor camera 4' is captured in advance in the line memory 24 with no output from the customer's hand 16 and the switch circuit turned on. This line memory output is 0 This waveform is stored in the line storage unit 24, so it appears as a background image at other times as well. In Figures 22(b) and (d), this is expressed as the line storage unit output.

[0090] Next, customer's hand 16 is t=T in Figure 3. 1 At this timing t=T, the waveform becomes as shown in FIG. 6(b), and the pixel signal output is shown in FIG. 22(b'). 1 In the differential circuit 13, this pixel signal output and t=T 0 The output of the line memory corresponding to the background image shown in FIG. 22(b) at the timing of t=T 1 This is the motion information itself, but since the pattern end portion becomes the motion contour information, it is called the motion contour information 7' ​​in relation to the first embodiment-1. t=T 0 Hereafter, t=T 1 If the customer's hand 16 is not recognized at the previous timing, the pixel signal output will have the same waveform as that shown in Figure 22(b), and the motion contour information 7' ​​that passes through the difference circuit 13 and absolute value circuit 14 will remain zero since there is no moving object and therefore no contour.

[0091] This timing t=T 1 The motion contour information 7' ​​is compared with a threshold value 19 in a masking extraction unit 8' and binarized. If the motion contour information 7' ​​exceeds the threshold value 19, 1 is assigned, and if not, 0 is assigned, and a masking signal 9 is obtained. 1 4 shows a masking signal 9 at the timing.

[0092] The period when the masking signal 9 is 1 corresponds to the timing when the movement of the customer's hand 16 occurs and the pixel output timing corresponding to the position of the customer's hand 16, and is the period when it should be masked and output from the pixel signal output 5. In other words, the pixel signal output 5 when the masking signal 9 is generated (is 1) corresponds to only observing the movement of the customer's hand 16, and is shown in FIG. 22(b"") as the masking pixel signal output 11. This is the same as the case of t=T 1 The timing is 16 signals from the customer's hand.

[0093] In a similar sequence, customer hand 16 is at t=T in Figure 3. 3 At the timing t=T, the pixel signal output is as shown in FIG. 22(d') in the same way as in FIG. 6(d). 3 In the differential circuit 13, this pixel signal output and t=T 0 22(d) is subtracted from the line memory output (same as FIG. 22(b)) corresponding to the background image of FIG. 22(d) at the timing of t=T 3 Similarly, the masking signal is shown in FIG. 22(d'''), and the masking pixel signal output 11 is shown in FIG. 12(d''"), and the masking signal output 11 is shown in FIG. 4(d) at t=T 3 The timing is 16 signals from the customer's hand.

[0094] The above is the timing t=T 0 , t=T 1 , t=T 3 We have explained the timing t=T shown in Figure 3, including other timings. 0 , t=T 1 , t=T 2 , t=T 3 , t=T 4 The left, center and right diagrams of Figures 23(a), (b), (c), (d) and (e) show what the motion contour information 7' ​​and masking signal will be like for the pixel signal output waveforms shown in Figures 6(a), (b), (c), (d) and (e) corresponding to the above.

[0095] The masking signals 9 shown in the right diagrams of Figures 23(a), (b), (c), (d), and (e) are the same as the masking signals shown in Figures 13(a), (b), (c), (d), and (e), and the masking pixel signal output 11 obtained through the masking circuit 10 in Figure 21 is the same as the right diagrams of Figures 14(a), (b), (c), (d), and (e), which correspond to the masking pixel signal output 11 obtained by the masking circuit 10 in Figure 11 at each timing. This is the same as the pixel signal output when there is no background light shown in Figures 4(a), (b), (c), (d), and (e), and the background light can be erased, resulting in an image of only the customer's hand 16, which is a moving object. The left diagrams of Figures 14(a), (b), (c), (d), and (e) also include the waveforms of the original pixel signal output 5. Timing t=T 0 , t=T 1 , t=T 2 , t=T 3 , t=T 4 corresponds to the dashed area on the customer's hand 16.

[0096] <First embodiment-4> Another configuration of the motion contour extraction section 6 and masking extraction section 8 of the imaging camera 1 in FIG. 1 will be described with reference to FIG. 24. This other configuration, which uses an area sensor 4" as the solid-state imaging device 4, is referred to as first embodiment-4. The configuration of the area sensor camera 1" is the same as that of first embodiment-2 in FIG. 16, and the pixel signal output 5 from the area sensor 4" is divided into two, one of which is input to a masking circuit 10 and the other is input to a motion contour extraction section 6'. In the motion contour extraction section 6', the pixel signal output 5 is further divided into two and taken into a frame storage section 26 via a switch circuit 23'. The pixel signal output 5 coming out of the frame storage section 26 is the frame output (frame storage section output 16, and the difference between this and the pixel signal output 5, which is the current frame output, is calculated by a difference circuit 13, and motion contour information 7' ​​is obtained via an absolute value circuit 14. Like the frame memory 22, the frame storage unit 26 stores the pixel signal output 5 of one frame, but since the signal is taken in via a switch circuit 23', it stores the pixel signal output 5 of one frame at the timing when the switch is turned ON. The frame storage unit 26 also differs from the frame memory 22 in the first embodiment-2 in Fig. 16 in that it repeatedly outputs the stored pixel signal output 5 of one frame. This is the same as in the first embodiment-3. The motion contour extraction unit 6' is distinguished as the motion contour extraction unit 6' because a switch circuit 23' has been added as a new component, and the frame memory output of the background light (pixel signal output 5) is stored in the frame memory 26 and used repeatedly. The output of the motion contour extraction unit 6' is also distinguished as motion contour information 7'.

[0097] Here, the frame storage unit output (pixel signal output 5) which is the output of the frame storage unit 26 is a still image of the background light in a state where the customer's hand 16 in FIG. 25 is not present (indicated by a dashed line in FIG. 25). 0 ~S 4 The waveforms of the scanning line pixel signal outputs of the background light of the area sensor 4″ in this case are shown in FIGS. 26(a), (b), (c), (d), and (e). When storing the pixel signal output 5 of background light where the customer's hand 16 does not appear, the switch circuit 23' is turned ON and the background light patterns shown in Figures 26(a) to (e) are captured into the frame memory unit 26. When the customer's hand 16 appears, the switch circuit 23' is turned OFF and the pixel signal output 5 is not captured into the frame memory unit 26. In this way, the difference circuit 13 can remove the background light. The frame memory output (pixel signal output 5) of the background light is stored in the frame memory 26 and is used repeatedly in the same way as the line memory 24. The difference circuit 13 outputs positive or negative depending on the magnitude relationship of the pixel signal outputs between the frames being compared. For this reason, the output of the difference circuit 13 is input to an absolute value circuit 14 to obtain motion contour information 7'. As in FIG. 21, FIG. 24 also distinguishes between the first embodiment-2 and the motion contour extraction unit 6' and motion contour information 7'.

[0098] The motion contour information 7' ​​obtained by the motion contour extraction unit 6' is input to a masking extraction unit 8'. In the masking extraction unit 8', the motion contour information 7' ​​is compared with a threshold value 19 in a masking signal generation circuit 20, and binarized for each scanning line to obtain a masking signal 9. The masking extraction unit 8' does not have a motion contour end determination unit 17 and motion contour end information 18, as in FIG. 21.

[0099] The masking signal 9, which is the output of the masking signal generating circuit 20, is a signal for extracting the timing for outputting the pixel signal output 5 corresponding to the pattern of the customer's hand 16. To achieve this, a masking circuit 10 obtains a masking pixel signal output 11 from this masking signal 9 and the pixel signal output 5, which is the output from the area sensor camera 4'', from which the background has been removed and which corresponds only to the customer's hand 16, which is a moving object.

[0100] At this time, if a problem occurs in extracting only the movement of the customer's hand 16 from the pixel signal output 5 due to the calculation processing time of the motion contour extraction unit 6' and the masking extraction unit 8', the pixel signal output 5 can be delayed by a time equivalent to this calculation time using a delay circuit 21 (not shown) present in the masking circuit 10 in Fig. 11 to obtain a masked pixel signal output 11 in the masking circuit 10. In the masking circuit 10, when the binarized masking signal 9 is 1, the pixel signal output 5 is output, and when it is 0, no pixel signal output 5 is output.

[0101] To add an explanation of the switches of the switch circuit 23, when the masking signals 9 for each scanning line are all 0, there is no movement of the customer's hand 16, which corresponds to background light, so the switch circuit stores the frame as ON, and when there is even one scanning line where the masking signal 9 is 1 (i.e., when the customer's hand 16 is moving), it is not stationary background light, so the switch circuit stores the frame as OFF. Since the background light rarely changes, there is no need to use the masking signal 9 every time to control the ON / OFF of the switch circuit 23, and it is sufficient to turn on the switch circuit 23 by periodic sampling and capture stationary background light, as in the first embodiment-3 of FIG. 21.

[0102] The masking signal 9 has a waveform with regions of 1 and 0 within one frame, and the masking circuit 10 extracts a moving object (customer's hand 16) from the pixel signal output 5. However, when the masking signal 9 that controls the switch circuit 23' is said to be 0 or 1, it is said to be 0 when the masking signal is always 0 within one frame of the masking signal, and is said to be 1 when the masking signal contains even a small amount of timing when the masking signal becomes 1. To perform this operation, a selection unit 25' is provided between the masking signal 9 and the switch circuit 23'. The selection unit 25' also has a function similar to the timer that performs the above-mentioned periodic sampling, and performs periodic frame storage of the background light with little change. In order to distinguish between line storage and frame storage, it is described as the switch circuit 23' and the selection unit 25'.

[0103] The operation of the motion contour extraction unit 6', masking extraction unit 8', and masking circuit 10 shown in Figure 24 will be described using as an example a case in which a customer's hand 16 reaching out to grab a product displayed on a display shelf 15 is monitored by an area sensor camera 1" installed at the upper front of the product display shelf 15. The background light pattern when the customer's hand 16 is not present is shown in Figures 26(a) to (e) as the pixel signal output of each scanning line of the area sensor 4" and is recorded in the frame memory unit 26. On the other hand, as shown in FIG. 17, when a customer's hand 16 reaches out to grab a product displayed on a product display shelf 15, the scanning line S of the area sensor 4" 0 ~S 4 The pixel signal output on each scanning line is shown in the left diagram of FIG. 19(a) to (e). The horizontal axis of the scanning line pixel signal output waveform in FIG. 18(a) to (e) is the scanning line (S 0 ~S 4 ) and the vertical axis corresponds to the pixel position on each scanning line, and the vertical axis corresponds to the pixel signal output on each scanning line.

[0104] FIG. 27 shows the scanning line S of the area sensor 4″. 1 , S 3 As an example, the waveforms of the frame memory output (background light), the scanning line pixel signal output, and the motion contour information 7', masking signal 9, and masking pixel signal output 11, which correspond to the motion contour extraction unit 6', masking extraction unit 8', and masking circuit 10, respectively, are explained. 1 , the right figure shows the scanning line S 3 Corresponds to. First, Fig. 27(b), (b'), (b"), (b"'), and (b"") show the scanning lines S 1 27(d), (d'), (d"), (d"'), and (d"") correspond to the frame memory output (background light), the scanning line pixel signal output, the motion contour information 7', the masking signal 9, and the masking pixel signal output 11 of FIG. 3 Corresponds to the one in.

[0105] In Fig. 27(b), (b'), (b"), (b"'), and (b""), the scanning line S 1To explain the waveform of the scanning line S 1 FIG. 27(b') shows the waveform of the frame memory output of the background light without the customer's hand 16 corresponding to the position of the scanning line S 1 27(b''). When the difference between the two is taken by the differential circuit 13 of the motion contour extraction unit 6' in FIG. 24 and the absolute value is taken by the absolute value circuit 14, motion contour information 7' ​​shown in FIG. 27(b'') is obtained, which is compared with the threshold value 19 by the masking signal generation circuit 20 of the masking extraction unit 8' in FIG. 24, and a binarized masking signal 9 is obtained as shown in FIG. 27(b'''). This and the scanning line S from the area sensor 4'' are 1 When the masking circuit 10 ANDs the pixel signal output 5 corresponding to the scan line S 1 The corresponding masking pixel signal output 11 is obtained and shown in FIG. 27(b''''). Similarly, the scan line S 3 For this, the frame memory output (background light), scanning line pixel signal output, motion contour information 7', masking signal 9, and masking pixel signal output 11 are obtained, which are shown in Figures 27(d), (d'), (d"), (d"'), and (d""), respectively.

[0106] The masking pixel signal output 11 shown in FIG. 27(b″”) and (d″”) is the scan line pixel signal output without background light shown in FIG. 18. 1 , S 3 18(b) and (d) corresponding to the above. That is, even if the subject has background light, if there is a moving object (here, the customer's hand 16), passing it through the circuit shown in the first embodiment-4 of FIG. 24 makes it possible to extract and output only the moving object for each scanning line.

[0107] The above is the scanning line S 1 , S 3 The scanning line S of the area sensor 4" of the area sensor camera 1" in FIG. 0 ~S 4Corresponding to the above, when there is background light, the pixel signal output on each scanning line is as shown in the left diagrams of Fig. 19(a), (b), (c), (d), and (e) (scanning line pixel signal output), which is shown again in the left diagrams of Fig. 28(a), (b), (c), (d), and (e). From this, the motion contour information 7' ​​extracted through the motion contour extraction unit 6 shown in Fig. 24 is shown in the center of Fig. 28(a), (b), (c), (d), and (e), and further the masking signal 9 extracted through the masking extraction unit 8' in Fig. 24 is shown in the right diagrams of Fig. 28(a), (b), (c), (d), and (e).

[0108] This masking signal 9 is the same as the masking signal obtained in the first embodiment-3, shown in Figures 23(a), (b), (c), (d), and (e), and is also the same as the masking signal shown in Figures 13(a), (b), (c), (d), and (e). That is, the masking pixel signal output 11 obtained through the masking circuit 10 in FIG. 24 is the same as the right diagrams in FIG. 14(a), (b), (c), (d), and (e), which correspond to the masking pixel signal output 11 obtained by the masking circuit 10 in FIG. 11 at each timing. This is the same as the pixel signal output when there is no background light, as shown in FIG. 4(a), (b), (c), (d), and (e). This timing is the same as the pixel signal output when there is no background light, as shown in FIG. 4(a), (b), (c), (d), and (e). In the area sensor 4'', 0 ~S 4 18, which corresponds to the scanning line pixel signal output without background light in FIG.

[0109] <First embodiment-5> Another configuration of the motion contour extraction unit 6 and masking extraction unit 8 of the imaging camera 1 in FIG. 1 will be described with reference to FIG. 29. This other configuration, which uses an area sensor 4''' with a special pixel array structure described later as the solid-state imaging device 4, is referred to as first embodiment-5. The configuration of the area sensor camera 1" is the same as that of first embodiment-2 in FIG. 16, and the pixel signal output 5 from the area sensor 4''' is divided into two, one of which is input to a masking circuit 10, while the other is input to a motion contour extraction unit 6". In the motion contour extraction unit 6", the pixel signal output 5 is further divided into two and taken into a bit memory 27. The pixel signal output 5 coming out of the bit memory 27 is the pixel signal output 5 of the previous bit, and a difference between this and the current pixel signal output 5 is taken by a difference circuit 13. This corresponds to obtaining the difference between adjacent pixels. The signal obtained by further difference passes through an absolute value circuit 14 to obtain motion contour information 7". Since the motion contour extraction unit 6" uses the bit memory 27, it is distinguished as the motion contour extraction unit 6". Also, the output of the motion contour extraction unit 6" is distinguished as the motion contour information 7".

[0110] Fig. 29 shows a case where a bit memory 27 is used as a method for obtaining pixel differentials. The difference circuit 13 calculates the difference between the current pixel signal and adjacent pixel signals of the same color, thereby making it possible to obtain pixel differentials. Since it is a difference, it takes on positive or negative values ​​(has a sign) depending on whether it is light or dark, so it goes through the absolute value circuit 14 to become absolute value information, and motion contour information 7" is obtained.

[0111] There are two methods for performing pixel differentiation between adjacent pixels of the same color: one is to use the difference between adjacent pixels of the same color in the horizontal direction on the same scan line, and the other is to read adjacent scan lines simultaneously and use the difference between adjacent pixels of the same color in the vertical direction. In the former case, the difference between pixels of the same color on the same scan line is taken, so the bit memory 27 shown in FIG. 29 is used, but in the latter case, adjacent scan lines appear simultaneously, so the bit memory 27 is not necessary. The motion end extraction unit 6" shown in FIG. 29 does not have the frame memory 22 present in the motion contour extraction unit 6 in FIG. 16. Therefore, the motion contour information 7" is an output for the most recent (not frame-based) scanning line-based image information captured by the area sensor 4"'.

[0112] A method of extracting a motion contour using a bit memory 27 without using a frame memory, as shown in FIG. 29 as the first embodiment-5, will be described. Acquiring motion contour information from the difference between adjacent pixel signals of the same color cannot be achieved with a normal pixel structure. Therefore, the pixel structure of the area sensor 4''' used in this case is special, and is composed of two types of pixels 28 with different exposure times, as shown in FIGS. 30 and 31. For this reason, it is distinguished from the area sensor 4'' in FIGS. 16 and 24 and is represented as area sensor 4''' in FIG. 29.

[0113] The configuration of the pixel 28 of the area sensor 4''' in FIG. 29 is shown in FIG. 30(a), and the pixel output characteristic is shown in FIG. 30(b). In the pixel configuration of the area sensor 4"' shown in Fig. 30(a), a short exposure pixel STS and a long exposure pixel STL are arranged adjacent to each other in the horizontal direction corresponding to two types of pixels 28 with different exposure times, and are output alternately on one scanning line. As shown in Fig. 30(b), the pixel output characteristics of both pixels 28 show the same slope (corresponding to sensitivity) of the increase in pixel output over time, and a pixel output appropriate for the exposure time is obtained. The STS and STL are read out at the same time and output as the pixel signal output 5 of the area sensor 4"'. In Fig. 30(b), the exposure times of the short exposure pixel STS and the long exposure pixel STL are respectively represented as T S , T L (T L >T S )

[0114] Of the pixel signal outputs 5 input to the motion end extraction unit 6" in Figure 29, the short-exposure pixel STS enters the bit memory 27, is delayed by a time equivalent to one pixel on a scanning line, and is then input to the difference circuit 13. This timing is the same as the input to the difference circuit 13 of the adjacent long-exposure pixel STL, and the difference between adjacent pixels on one scanning line can be obtained. At this time, since the pixel outputs of the short-exposure pixel STS and the long-exposure pixel STL differ in magnitude by the difference in exposure time as shown in Figure 30(b), gain adjustment is performed so that the outputs are aligned. Specifically, in addition to delaying by the time equivalent to one pixel on a scanning line (1 bit) in the bit memory 27, S , T L To compensate for the difference in output proportional to T L / T S This gain adjustment function is also built into the bit memory 27 (not shown). By adjusting the gain and taking the difference, the outputs are uniform in a still image and the output of the difference circuit 13 becomes zero, whereas in a moving subject, an output remains at the edge of the moving object, and after passing through the absolute value circuit 14, the output remains as motion contour information 7".

[0115] In this way, still areas have the same signal level difference and so become zero. If there is movement, and in areas where the output value of the moving object changes, the difference will not become zero when it is obtained. Also, if the light and dark are uniform inside the moving object pattern, the inside of the moving object pattern becomes zero like a still image, and the contour pattern of the moving object remains. In this way, the motion contour information 7" contains contour information of the moving object. Paragraph 26 defines "pixel difference means the difference between pixels with different exposure timings that image the same position on the subject." Therefore, the difference between adjacent pixels is strictly different from the difference between pixels in the present application, but since the number of pixels is generally several thousand for a linear sensor and one million to ten million for an area sensor, it can be considered that adjacent pixels are photographing the same position on the subject. In addition, by averaging pixels of the same color in the vertical and horizontal directions and taking the difference between the pixel outputs of adjacent short-exposure pixels STS and long-exposure pixels STL, which appears as an additional function of the second embodiment-5 in Fig. 33, it is possible to reduce noise that occurs at the edge of a stationary object and to extract the contour of a moving object with higher accuracy. In this case, it is further possible to consider that adjacent pixels are photographing the same position (area) of the subject.

[0116] In the area sensor 4''' shown in FIG. 30(a), the short exposure pixels STS and the long exposure pixels STL are arranged adjacent to each other in the horizontal direction. However, when they are adjacent to each other in the vertical direction, the two lines of the short exposure pixels STS and the long exposure pixels STL are read out simultaneously as one scanning line. In this case, there is no need for a delay of one pixel (one bit) in the bit memory 27, and the exposure time T S , T L To compensate for the difference in output proportional to the L / T S The gain-adjusted short-exposure pixel STS and long-exposure pixel STL are input to a differential circuit 13, and a difference is calculated.

[0117] Another configuration of the pixel 28 of the area sensor 4''' in FIG. 29 is shown in FIG. 31(a), and its pixel output characteristics are shown in FIG. 31(b). In the configuration of the pixels 28', 28" of the area sensor 4''' shown in FIG. 31(a), a short exposure pixel 28' (indicated as STS in the figure) and a long exposure pixel 28" (indicated as STL in the figure) are arranged, but the two pixels have different sizes and therefore different sensitivities. If the sensitivity of the short exposure pixel STS is SS and the sensitivity of the long exposure pixel STL is SL, then in the case where the sensitivity ratio between the two; SS / SL is set to the reciprocal of the exposure time ratio between the two; TS / TL, the pixel output characteristics will be as shown in FIG. 31(b), and the pixel outputs of the short exposure pixel STS and the long exposure pixel STL will be the same.

[0118] The short-exposure pixel STS and the long-exposure pixel STL are adjacent to each other in the vertical direction in the pixel arrangement shown in FIG. 31(a). These two lines are read out simultaneously as one scanning line. As shown in FIG. 31(b), the exposure time T S :TL The sensitivity ratio between the two is T L :T S As a result, the pixel outputs of the short exposure pixel STS and the long exposure pixel STL are aligned. The aligned pixel outputs of the short exposure pixel STS and the long exposure pixel STL are input to the difference circuit 13, which takes the difference. In a still image, the outputs are aligned and the output of the difference circuit 13 becomes zero, whereas in a moving subject, an output remains at the edge of the moving object, and after passing through the absolute value circuit 14, the output remains as motion contour information 7".

[0119] 31(b), the pixel outputs of the short exposure pixel STS and the long exposure pixel STL are consistent, eliminating the need for gain adjustment in the bit memory 27. However, since only the exposure time ratio (TS / TL) determined by the sensitivity ratio SS / SL can be selected, it is better to have gain adjustment in order to increase the degree of freedom in the exposure time. In this way, the bit memory 27 is used as a general term for a circuit that has the function of bit delaying or adjusting the gain if necessary so that the outputs of pairs of adjacent pixels with different exposure times are aligned in the case of a still image and become zero when the difference is taken.

[0120] By adjusting the gain for a pixel pair having different exposure times as shown in FIG. 30 to make the outputs uniform, or by adjusting the pixel sensitivity to match the exposure time for a pixel pair having different exposure times as shown in FIG. 31 to make the outputs uniform, motion contour information 7" is prevented from being generated, and the still image in the background can be eliminated.

[0121] In the first embodiment-5 of FIG. 29, an area sensor 4''' is used and the pixel structure of the area sensor as shown in FIG. 30 and FIG. 31 is described, but this pixel structure also applies to a linear sensor. The greatest feature of the present invention is that the pixel signal output 5 from the solid-state imaging device 4 is processed for each readout line, and a masking pixel signal output 11 is generated in real time. Therefore, it is the same whether it is an area sensor or a linear sensor. As described above, the final masking pixel signal output 11 is the same in the first embodiment-1 (linear sensor), the first embodiment-2 (area sensor), the first embodiment-3 (linear sensor), the first embodiment-4 (area sensor), and the first embodiment-5 (area sensor).

[0122] <Second embodiment> The imaging camera 1 has been described as an imaging device, but next, a second embodiment of the imaging device will be described as an imaging device, and a solid-state imaging device 4 shown in Fig. 32 will be described as an imaging device. In Fig. 32, the block configuration of the solid-state imaging device 4 of the present invention and signals of each block will be described.

[0123] A lens 2 forms an image of a subject on a pixel section 28 of a solid-state imaging device 4. A pixel signal output 5 output from the pixel section is split into two, one of which is input to a motion contour extraction section 6 and the other to a masking circuit 10. The motion contour extraction section 6 extracts motion contour information 7 from the pixel signal output 5. The extracted motion contour information 7 is input to a masking extraction section 8. The masking extraction section 8 creates and outputs a masking signal 9 from the motion contour information 7, which is used to output only moving objects. The masking signal 9 is input to the masking circuit 10, and together with the pixel signal output 5 also input, a masking pixel signal output 11 is created and output by the masking circuit. This masking pixel signal output is an output of only pixels corresponding to an image of a moving object, making this an imaging device that outputs only moving object information, which is the objective of the present invention. The pixel signal output 5 and subsequent components have the same block configuration and signals as the imaging camera 1 of the first embodiment shown in FIG. 1, and the block configuration is formed (on-chip) on the solid-state imaging device 4 having the pixel section.

[0124] That is, in the second embodiment, the circuit block of the imaging camera 1 of the first embodiment shown in FIG. 1, which is located after the pixel signal output 5 of the first embodiment-1 (linear sensor), first embodiment-2 (area sensor), first embodiment-3 (linear sensor), first embodiment-4 (area sensor), and first embodiment-5 (area sensor), is formed on a solid-state imaging device 4, and the second embodiment is respectively called second embodiment-1 (linear sensor), second embodiment-2 (area sensor), second embodiment-3 (linear sensor), second embodiment-4 (area sensor), and second embodiment-5 (area sensor). Its functions and operations are as described above, but here, taking the second embodiment-5 (area sensor) having a special pixel structure as an example, the circuit configuration of the solid-state imaging device 4''' will be described with reference to FIG.

[0125] <Second embodiment-5> In the imaging camera 1 of FIG. 1, the pixel signal output 5 output from the area sensor 4"' in FIG. 29 passes through a bit memory 27 having a gain adjustment function, a difference circuit 13, and an absolute value circuit 14 in the motion contour extraction unit 6" to obtain motion contour information 7". There are several types of pixel arrangements in the area sensor 4"', as shown in FIG. 30(a) and FIG. 31(a), so in FIG. 33, the pixel signal output 5 passes through a separation circuit 29 and a gain adjustment circuit 29', and the difference is taken from the short-exposure pixel STS and the long-exposure pixel STL in the difference circuit 13 to obtain absolute value. The method is to obtain motion contour information 7" through a value circuit 14. The processing method is the same as in the first embodiment-5, and since the bit memory 27 includes the functions of a separation circuit 29 and a gain adjustment circuit 29', they are denoted by the same numbers. In Figure 33, the method of obtaining a masking pixel signal output 11 from the motion contour information 7" through a masking extraction unit 8" and a masking circuit 10 is the same as in the first embodiment-5 of Figure 29. The masking circuit 10 is an AND circuit 30 (which has not been given a number until now).

[0126] In the circuit configuration of the solid-state imaging device 4''' shown in FIG. 33, more specific circuit blocks are added. In the pixel arrays shown in Figures 30(a) and 31(a), processing using the short-exposure pixels STS and the long-exposure pixels STL was performed using the difference between adjacent pixel signals of the same color. To obtain a color image, color video information 32 is obtained through a further color processing unit 31. The pixel unit 28 is composed of two types of pixels with different exposure times as shown in Figures 30(a) and 31(a).

[0127] Regarding the pixel section 28 of the solid-state imaging device 4''' in FIG. 33, color filters in a Bayer arrangement are formed in each pixel for color imaging. These are shown as green (Gr, Gb), red (R), and blue (B) in the drawing. Exposure control, vertical scanning, and horizontal scanning are performed at the pixels based on a master clock generated by a timing generation circuit 33. As an exposure control method, the exposure times TL and TS of the long exposure pixel STL and short exposure pixel STS explained in Fig. 30 and Fig. 31 are generated by an exposure time generation circuit 34, and the pixel unit 28 is controlled via a vertical scanning circuit 35.

[0128] In the vertical parallel control circuit 36, by simultaneously turning on multiple pixel drive lines arranged in the vertical direction, it is also possible to vertically combine (average) pixel signals of the same color arranged vertically. At the top of the pixel section 28, the circuit that simultaneously reads out pixel signals of multiple vertical signal lines output from the pixel section 28 is composed of a horizontal scanning circuit 37 that scans the horizontal readout, and a horizontal combining circuit 38 that receives the vertical signal lines and combines the signals in the horizontal direction. It is also possible to horizontally combine (average) pixel signals of the same color in the horizontal combining circuit 38. This pixel combination of the same color is performed between pixels with the same exposure time, which can reduce noise generated at the edges of stationary objects and enable more accurate extraction of the contours of moving objects.

[0129] As shown in Figures 30 and 31, the short exposure pixels STS and the long exposure pixels STL are alternately repeated for each scanning line, but the synthesis (averaging) of pixel signals of the same color and with the same exposure time is performed over a plurality of scanning lines, so that long exposure information and short exposure pixel information of approximately the same area can be obtained, and the motion contour information 7" obtained by subtracting the information in the difference circuit 13 is a comparison of approximately the same area, which has the advantage of improving the accuracy of the motion information. When taking the difference between the short-exposure pixel STS and the long-exposure pixel STL on a pixel-by-pixel basis, a defect (edge ​​noise) occurs where the difference does not become zero even in the case of a stationary object, but by averaging, the edge noise in the contour area can be significantly suppressed, which is the advantage. This is the reason why the contours of moving objects can be extracted with higher accuracy as mentioned above.

[0130] Combining (averaging) pixel signals of the same color with the same exposure time has the advantage of significantly reducing the amount of information, although this depends on the number of pixels to be combined in the vertical and horizontal directions. For example, averaging 10 pixels in each direction can reduce the amount of information by two orders of magnitude (1 / 100). In conventional motion analysis, processing the massive amount of image information output by solid-state imaging devices in real time on a pixel-by-pixel basis required a large image processing load on the downstream processing circuitry, making battery power difficult. This method has the great advantage that the amount of information corresponding to the number of pixels is greatly reduced (by two digits in the above example), and the amount of information per pixel is also greatly reduced (for example, from 8 bits to 1 bit) by processing the binarized motion contour information 9. This, along with outputting only moving objects, which is a feature of the imaging device of the present invention, contributes to reducing the load by reducing the amount of information.

[0131] In the solid-state imaging device 4"' in Figure 33, a column-type noise cancellation circuit (CDS) 39 to which pixel signals of multiple vertical signal lines output from the pixel section 28 are input, a column-type analog-to-digital converter circuit (AD conversion) 40, a line memory 41, and a horizontal pixel scanning circuit 42 are added below the pixel section 28. Based on multiple lines of pixels in a Bayer array (2 x 2 pixels) (which is why the line memory 41 is necessary), color processing is performed in the color processing unit 31 to generate color video information 32. The line memory 41 has the same name as the line memory 12 in Figure 2, but it is a line memory for performing color processing by an area sensor, and so although the name is the same, different numbers are used. In the explanation of the symbols, it is written as line memory (for color processing).

[0132] 33, the parts related to color processing may be omitted, and only the masking pixel signal output 11 may be output. Omitting the parts related to color processing means omitting the CDS circuit 39, ADC circuit 40, line memory 41, horizontal pixel scanning circuit 42, and color processing unit 31.

[0133] In Fig. 33, a method was explained in which the vertical parallel control circuit 36 ​​and horizontal synthesis circuit 38 are used to vertically synthesize (average) pixel signals of the same color arranged vertically, and horizontally synthesize (average) pixel signals of the same color in the horizontal direction, respectively, to reduce the amount of information and edge noise and obtain motion vector information. The horizontal synthesis circuit 38 and horizontal scanning circuit 37 are added to the parts related to color processing of a normal solid-state imaging device. The present invention can also be applied to the output of normal color processing, as will be explained in the next Fig. 34.

[0134] In Fig. 34, the horizontal scanning circuit 37 and horizontal synthesis circuit 38 are omitted. Color filters (green (Gr, Gb), red (R), and blue (B) in the figure) in a Bayer arrangement are formed in each pixel for color imaging, and from the output of a line memory 41 for color processing of these, motion contour extraction unit 6'' extracts motion contour information 7'' from STS and STL of the same color. The rest is the same as in Fig. 33.

[0135] <Relationship between claims, embodiments, and corresponding drawings> Claim 1: The first and second embodiments of the present invention, which correspond to Figs. 1 and 32, respectively. Claim 2: The first embodiment of the present invention, which corresponds to Figs. Claim 3: The first embodiment-3 and -4 of the present invention, which correspond to Figs. 21 to 24 and Figs. 24 to 28, respectively. Claim 4: The first embodiment-3 and -4 of the present invention, which correspond to Figs. 21 and 24, respectively. Claim 5: The first embodiment-1 and -2 of the present invention, which correspond to Figs. 2 to 15 and Figs. 16 to 20, respectively. Claim 6: The first embodiment-5 and the second embodiment-5 of the present invention, which correspond to Figs. 29 to 31 and Figs. 33 to 34, respectively. Claim 7: This is the fifth embodiment of the second invention and corresponds to FIG. Claim 8: The first embodiment of the present invention, which corresponds to the first and third embodiments of the present invention, respectively to Fig. 2 and Fig. 21. It also corresponds to the second embodiment of the present invention, which corresponds to the first and third embodiments of the present invention, although there are no figures. Claim 9: The first embodiment of the present invention, which corresponds to the second and fourth embodiments of the present invention, corresponds to Fig. 16 and Fig. 24. Although not shown in the figures, it also corresponds to the second embodiment of the present invention, which corresponds to the second and fourth embodiments of the present invention. Claim 10: A first embodiment of the present invention, which corresponds to Figs. Claim 11: A second embodiment of the present invention, which corresponds to Figs. [Explanation of symbols]

[0136] 1 Imaging camera 1' Linear sensor camera 1” area sensor camera 2 Lenses 3 Camera housing 4. Solid-state imaging device 4' Linear Sensor 4", 4"' Area Sensor 5 pixel signal output 6, 6', 6” Motion Contour Extraction Unit 7, 7', 7” Motion Contour Information 8, 8', 8” Masking Extraction 9 Masking Signals 10 Masking Circuit 11 Masking pixel signal output 12 Line Memory 13 Differential circuit 14 Absolute Value Circuit 15 Product display shelves 16 Customer's Hands 17, 17” Motion contour edge determination section 18, 18” Motion Contour End Information 19 Threshold 20 Masking signal generating circuit 21 Delay Circuit 22 Frame Memory 23, 23' Switch circuit 24, 24' Line Memory 25, 25' Selection section 26 Frame Memory Unit 27-bit memory 28, 28', 28” pixels 29 Separation circuit 29' Gain adjustment circuit 30 AND Circuit 31 Color Processing Section 32. Color image information 33 Timing generation circuit 34 Exposure time generation circuit 35 Vertical scanning circuit 36 Vertical parallel control circuit 37 Horizontal scanning circuit 38 Horizontal synthesis circuit 39 CDS circuit 40 ADC circuits 41 Line memory (for color processing) 42 Horizontal pixel scanning circuit

Claims

1. a pixel unit that captures an image of a subject formed by an optical imaging unit and has a plurality of pixels that are photoelectric conversion element units that convert an optical signal of the subject into an electric signal, and at least one of two image capture signals captured at different times is a pixel signal output that is a current image capture signal, and a differential signal between the two image capture signals is generated by calculating a difference between pixels corresponding to the same location of the subject; an adjustment means for adjusting the difference signal so that the difference signal is not generated from a photographic signal from a stationary subject; a means for obtaining, for each read line, motion contour information related to contour information of a moving subject by taking an absolute value of the difference signal; a means for obtaining a binarized masking signal for each readout line, the binarized masking signal being obtained by comparing the motion contour information with a threshold value and setting a region of a moving subject to 1 and other regions to 0; a means for performing a logical AND operation on the binarized masking signal and a pixel signal output, which is a current photographing signal, from a pixel unit, and outputting only the pixel signal output where the masking signal is 1 as an image of a moving subject for each readout line; An imaging device comprising:

2. The imaging device according to claim 1, characterized in that a memory is used as a means for generating a differential signal between two imaging signals captured by the pixel section at different imaging times, one of which is the imaging signal at the current time, and the time interval between the different imaging times is set to an integer multiple of the cycle time required to read out all pixels in the pixel section, thereby performing image difference.

3. The imaging device according to claim 1, characterized in that a means for generating a difference signal between two imaging signals captured by the pixel section at different imaging times, one of which is an imaging signal at a current time, is realized by repeatedly using a still imaging signal with no movement as the imaging signal for performing pixel difference, thereby performing image difference.

4. 4. The imaging device according to claim 3, further comprising: a binarized masking signal that is used to determine whether an image signal is in a stationary state and no motion and that should be stored in memory; an image signal in which the masking signal is all 0 is determined to be an image signal in a stationary state and no motion; and this image signal is periodically temporarily stored in memory.

5. The imaging device described in claim 2, characterized in that the time interval between different shooting times is the cycle time required to read out all pixels of the pixel section, and image difference is performed, and a binarized masking signal is obtained by comparing with a threshold value through motion contour end information corresponding to information on the outer contour of the moving object.

6. As a means for generating a difference signal between two image pickup signals captured by the pixel unit at different image pickup times, an exposure time control means for dividing a pixel arranged in the pixel section into at least two parts and controlling the length of exposure time to be different; a means for generating a difference signal between the image pickup signals having different exposure times, the difference signal being adjusted so that no difference signal is generated from an image pickup signal of a stationary subject; a means for generating a difference signal between the two photographing signals, the difference being between adjacent pixels that are considered to correspond to the same location on a subject, the two photographing signals being pixel signal outputs that are current photographing signals; 2. The imaging device according to claim 1, wherein the imaging device is realized by

7. 7. The imaging device according to claim 6, wherein in the imaging device that outputs only images of a moving subject, the means for generating a difference signal between two shooting signals captured by the pixel section at different shooting times is a means for averaging shooting signals of multiple adjacent pixels under the same conditions and generating a difference signal between the averaged shooting signals.

8. 2. The imaging device according to claim 1, wherein the pixel section is a linear sensor in which pixels are linearly arranged, the memory used is a line memory, and the cycle time is a line period.

9. 2. The image pickup apparatus according to claim 1, wherein the pixel section is an area sensor in which pixels are arranged in an area, the memory used is a frame memory, and the cycle time is a frame period.

10. 2. The imaging device according to claim 1, wherein the imaging device is an imaging camera.

11. 2. The imaging device according to claim 1, wherein the imaging device is a solid-state imaging device.

Images