Signal processing apparatus and signal processing method

Abstract

A signal processing apparatus includes: a position detection unit that detects, as an object position, a position of a moving object in a captured image, based on the captured image of an imaging unit that captures the moving object; a position estimation unit that estimates, as a predicted position, the position of the object in the captured image after the next frame on the basis of the moving speed of the object and the detected object position; and a frame setting unit that sets a cut frame in the captured image after the next frame on the basis of the predicted position.

Description

Technical Field

[0001] The present technology relates to the technical field of a signal processing device and a signal processing method that tracks a moving object as a detection target object and generates and outputs a cutout image in a manner including the moving object. Background Art

[0002] There is a region of interest (ROI) function that cuts out a partial area from a captured image captured over the entire viewing angle. In some ROI functions, only partial image data of an area including a target object is output by setting the ROI in a manner that tracks the detected subject when the subject moves (for example, the following patent document 1). Citation List Patent Literature

[0003] Patent Document 1: Japanese Patent Application Publication No. 2021-166317 Summary of the invention Problems to be solved by the present invention

[0004] Since the subject to be tracked in each frame image is detected by image processing, the ROI function for tracking the subject has a problem of a large processing load.

[0005] The present technology has been proposed in view of such a problem, and an object of the present technology is to reduce the processing load of the ROI function that tracks a subject. Solution to the problem

[0006] According to the present technology, the signal processing device includes: a position detection unit, which detects the position of the target object in the captured image as the object position based on the captured image of the imaging unit that captures the moving target object; a position estimation unit, which estimates the position of the target object in the captured image after the next frame as the predicted position based on the moving speed of the target object and the detected object position; and a frame setting unit, which sets a cutout frame in the captured image after the next frame according to the predicted position. As a result, the cutout frame can be set in a manner including the position of the target object predicted based on the object position and the moving speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a block diagram of a camera device. Figure 2 It is a diagram for explaining a cropping frame and a detection frame set in a partial area of ​​a captured image. Figure 3 is a diagram showing an example in which a detection frame wider than a cutout frame is set. Figure 4 This is a diagram for explaining the overview of barcode monitoring. Figure 5 is a diagram for explaining a state in which a cutout frame is set to follow a barcode. Figure 6 is a block diagram showing the functional configuration of the image sensor in the first embodiment. Figure 7 is an example of output data compliant with the MIPI standard. Figure 8 is a flowchart of an example of a process performed by a control unit of an image sensor. Fig. 9 is a block diagram showing the functional configuration of an image sensor in the second embodiment. Fig.10 is a block diagram showing the functional configuration of an image sensor in the third embodiment. DETAILED DESCRIPTION

[0008] Hereinafter, embodiments according to the present technology will be described in the following order with reference to the drawings. <1. Configuration of signal processing device> <2. Overview of Monitoring of Signal Processing Device> <3. Functional Configuration of Signal Processing Device> <4. Processing Flow> <5. Second Implementation Method> <6. Third Implementation Plan> <7. Conclusion> <8. This technology>

[0009] <1. Configuration of signal processing device> The signal processing device according to the first embodiment of the present embodiment is a device that performs various types of processing on a captured image captured by an imaging unit. Such a signal processing device can be considered in various ways, and in the following description, the image sensor IS will be described.

[0010] The image sensor IS is provided in the camera device 1. Figure 1 An example of the internal configuration of the camera device 1 will be described.

[0011] The camera apparatus 1 includes an image sensor IS, an imaging optical system 31 , an optical system driving unit 32 , a control unit 33 , a storage unit 34 , and a communication unit 35 .

[0012] The image sensor IS, the control unit 33, the storage unit 34, and the communication unit 35 are connected to each other through a bus 36 so as to perform data communication with each other.

[0013] The imaging optical system 31 includes lenses such as a cover lens, a zoom lens, and a focus lens, and an aperture (iris) mechanism. Light (incident light) from a subject is guided by the imaging optical system 31, and the light is condensed on a light receiving surface of the image sensor IS.

[0014] The optical system driving unit 32 collectively represents a driving unit of the zoom lens, focus lens, and aperture mechanism included in the imaging optical system 31. Specifically, the optical system driving unit 32 includes an actuator for driving each of the zoom lens, focus lens, and aperture mechanism, and a driving circuit of the actuator.

[0015] For example, the control unit 33 includes a microcomputer including a central processing unit (CPU), a read only memory (ROM), and a random access memory (RAM), and the CPU performs various types of processing according to a program stored in the ROM or a program loaded in the RAM, so that the control unit 33 performs overall control of the camera device 1.

[0016] Furthermore, the control unit 33 instructs the optical system driving unit 32 to drive the zoom lens, focus lens, diaphragm mechanism, etc. The optical system driving unit 32 moves the focus lens and zoom lens, opens or closes diaphragm blades of the diaphragm mechanism, etc. in response to such driving instructions.

[0017] Furthermore, the control unit 33 controls writing and reading of various types of data to and from the storage unit 34 . The storage unit 34 is a nonvolatile storage device such as a hard disk drive (HDD) or a flash memory device, and serves as a storage destination (recording destination) of image data output from the image sensor IS.

[0018] Furthermore, the control unit 33 performs various data communications with external devices via the communication unit 35 .

[0019] For example, the camera device 1 in this example is used to monitor (or inspect) an object to be monitored that is placed on an object transport mechanism such as a belt conveyor and moves. That is, the camera device 1 performs various inspections on the monitored object OB by performing predetermined image processing based on captured image data obtained by imaging the monitored object OB. The communication unit 35 is configured to be able to communicate data with the belt conveyor control device or the like, and can output its monitoring results (inspection results) to the outside. As a result, the malfunctioning monitoring object OB can be removed from the belt conveyor, or an alarm can be generated.

[0020] For example, the image sensor IS is configured as an image sensor of a CCD type, a CMOS type, or the like.

[0021] The image sensor IS includes an imaging unit 41 , an image signal processing unit 42 , an in-sensor control unit 43 , a storage unit 45 , and a communication I / F 46 , which can perform data communication with each other via a bus 47 .

[0022] The imaging unit 41 includes a pixel array unit and a readout circuit. In the pixel array unit, pixels each including a photoelectric conversion element such as a photodiode are arranged in two dimensions. The readout circuit reads an electrical signal obtained by photoelectric conversion from each pixel included in the pixel array unit and can output the electrical signal as a captured image signal.

[0023] For example, the readout circuit performs correlated double sampling (CDS) processing, automatic gain control (AGC) processing, etc. on the electrical signal obtained by photoelectric conversion, and further performs analog / digital (A / D) conversion processing on the electrical signal.

[0024] The image signal processing unit 42 performs preprocessing, synchronization processing, YC generation processing, resolution conversion processing, codec processing, and the like on a captured image signal of digital data obtained as a result of the A / D conversion processing. In the pre-processing, the captured image signal is subjected to: a clamping process for clamping the black levels of R, G, and B to a predetermined level; a correction process between the color channels of R, G, and B, etc. In the synchronization process, a color separation process is performed so that the image data of each pixel has all R, G, and B color components. For example, in the case of an imaging element using a color filter of a Bayer array, a demosaicing process is performed as a color separation process. In the YC generation process, a brightness (Y) signal and a color (C) signal are generated (separated) from the image data of R, G, and B. In the resolution conversion process, a resolution conversion process is performed on the image data that has undergone various types of signal processing. In the codec processing, for example, encoding processing and file generation for recording or communication are performed on the image data subjected to the above-mentioned various types of processing. In the codec processing, it is possible to generate a file in a format such as MPEG (moving picture experts group)-2 or H.264 as a moving image file format. It is also conceivable to generate a file as a still image file in a format such as JPEG (joint photographic experts group), TIFF (tagged image file format) or GIF (graphics interchange format). Note that, for example, in the case where the image sensor IS is a distance measuring sensor, the image signal processing unit 42 calculates distance information about the subject based on two signals output from the image sensor IS as indirect time of flight (iToF) and outputs a distance image.

[0025] The in-sensor control unit 43 issues an instruction to the imaging unit 41 to perform execution control of the imaging operation. Similarly, the execution of the processing of the image signal processing unit 42 is also controlled.

[0026] The image signal processing unit 42 and the in-sensor control unit 43 according to the present embodiment perform various types of processing in order to monitor (or inspect) the monitoring object OB placed on an object transport mechanism such as a belt conveyor while moving.

[0027] Specifically, as described later, the image signal processing unit 42 and the in-sensor control unit 43 perform binarization processing, mask processing, moment processing, center of gravity calculation processing, frame position calculation processing, movement amount calculation processing, movement speed acquisition processing, position estimation processing, image clipping processing, output data shaping processing, etc.

[0028] Each process can be executed by the image signal processing unit 42 or the in-sensor control unit 43. That is, each process can be realized by hardware such as a logic circuit, or can be realized by software such as a program.

[0029] The storage unit 45 can be used as a so-called frame memory that stores the captured image data (RAW image data) and the synchronously processed image data obtained by the image signal processing unit 42. In addition, the storage unit 45 can be used to store temporary data generated when the image signal processing unit 42 or the in-sensor control unit 43 performs each of the above-mentioned processes.

[0030] The communication I / F 46 is an interface for communicating with the control unit 33, the storage unit 34, etc. located outside the image sensor IS. The communication I / F 46 performs communication for acquiring a program or the like executed by the image signal processing unit 42 from the outside, and stores the program or the like in the storage unit 45 included in the image sensor IS.

[0031] <2. Overview of Monitoring of Signal Processing Device> As described above, the image sensor IS as a signal processing device captures an image of the monitoring object OB conveyed by an object conveying mechanism such as a belt conveyor, and outputs a partial image of a region including a target object which is a part of the monitoring object OB as an ROI image.

[0032] For example, the monitoring object OB is a product placed in a packaging material such as cardboard, and a barcode BC such as a one-dimensional code or a two-dimensional code is attached to the surface of the packaging material. The image sensor IS monitors these barcodes BC as monitoring targets.

[0033] In the following description, the barcode BC is described as an example of a target object (which is a part of the monitored object OB), but the target object may be other objects. For example, the target object may be a part of a product (e.g., a handle, etc.), and the image sensor IS may determine whether the part is normally attached to the product.

[0034] In order to detect the barcode BC on the monitored object OB, the image sensor IS outputs a partial area on the captured image 2 corresponding to the effective area of ​​the sensor surface as the ROI area. To this end, the image sensor IS sets the frame for cutting the ROI area as the cutting frame 3 (see Figure 2 ).

[0035] The partial image cut out by the cutout frame 3 is the ROI image 4 . exist Figure 2 In FIG. 1 , the cut frame 3 is represented by a single-dot chain line.

[0036] The partial area in the cutout frame 3 is set as the target area for image processing for performing object recognition. The image sensor IS sets the detection frame 5 as a frame for specifying the target area. Figure 2 Indicated by dotted line.

[0037] The detection frame 5 is set so that when the monitored object OB is conveyed to a predetermined position in the object conveying mechanism, the barcode BC is located within the frame. In the case where the position of the barcode BC may be greatly offset, the detection frame 5 is also set to be larger. In this case, the detection frame 5 may be set to be larger than the cutout frame 3 (see Figure 3 ).

[0038] When the barcode BC is detected, the cutout frame 3 and the detection frame 5 are set to move in the captured image 2 to follow the movement of the barcode BC. In other words, the image sensor IS sets the cutout frame 3 and the detection frame 5 so that the moving barcode BC falls within the frame.

[0039] Figure 4 An overview of the positional relationship among the cutout frame 3 , the detection frame 5 , and the barcode BC is shown. Figure 4 There are shown five captured images 2 and a cutout frame 3 and a detection frame 5 set on each captured image 2. The five captured images 2 are arranged in time series from the top, and are captured images 2A, 2B, 2C, 2D, and 2E.

[0040] also, Figure 4 ROI images 4A, 4B, 4C, 4D, and 4E corresponding to the captured images 2A, 2B, 2C, 2D, and 2E are shown.

[0041] A cutout frame 3A and a detection frame 5A are set on the captured image 2A. In addition, the barcode BC is not detected in the detection frame 5A of the captured image 2A.

[0042] On the captured image 2B, a cutout frame 3B and a detection frame 5B are set. A portion of the barcode BC is detected in the detection frame 5B of the captured image 2B. In this way, in the case where it is determined that a portion of the barcode BC is detected in the detection frame 5B, the image sensor IS can capture the barcode BC in the detection frame 5 as early as possible by setting the cutout frame 3 and the detection frame 5 to be offset to the right in the figure in the captured image 2 as the next frame image (not shown).

[0043] On the captured image 2C, a cutout frame 3C and a detection frame 5C are set. In addition, the entire barcode BC is included in the detection frame 5C of the captured image 2C, and the barcode BC is detected.

[0044] In the captured image 2D, a cutout frame 3D and a detection frame 5D are set. In addition, the cutout frame 3D and the detection frame 5D are set to be offset to the left in the figure as the barcode BC moves. The detection frame 5D of the captured image 2D includes the entire barcode BC and detects the barcode BC.

[0045] In the captured image 2E, a cutout frame 3E and a detection frame 5E are set. The cutout frame 3E and the detection frame 5E are set to be further shifted to the left in the figure as the barcode BC moves. The detection frame 5E of the captured image 2E includes the entire barcode BC and detects the barcode BC.

[0046] Note that the image sensor IS acquires the moving speed of the barcode BC, and after acquiring the moving speed, performs processing for predicting the position of the barcode BC based on the moving speed. At this time, image processing for detecting the barcode BC in the detection frame 5 is not performed. Therefore, the image sensor IS may not set the detection frame 5 after calculating the moving speed.

[0047] Specifically, reference will be made to Figure 5 The flow of processing executed by the image sensor IS will be described. Figure 5 The relationship between a captured image 2 captured by the image sensor IS, an ROI image 4 outputted according to a cutout frame 3 set on the captured image 2, and processing performed by the image signal processing unit 42 or the in-sensor control unit 43 at this time is shown.

[0048] The captured images 2 are captured images 2a, 2b, 2c, 2d and 2e frame by frame in time sequence, and captured images 2 (not shown) are followed by captured images 2y and 2z. The captured images 2y and 2z are the images of the last two frames when the tracking of the barcode BC is stopped.

[0049] Note that Figure 5 , the captured images 2 captured before the captured image 2a, that is, the captured images 2 until the entire barcode BC is captured in the detection frame 5 are omitted. That is, the captured image 2a is the first captured image 2 including the entire barcode BC in the detection frame 5.

[0050] The image signal processing unit 42 and the in-sensor control unit 43 (hereinafter, simply referred to as the "control unit") of the image sensor IS perform image processing in the detection frame 5a on the captured image 2a to perform detection processing for detecting the barcode BC. As a result, in response to the detection of the barcode BC in the detection frame 5a, the control unit of the image sensor IS calculates the center of gravity position of the barcode BC. As a result, the position of the barcode BC is determined. This will be described in detail later.

[0051] In the captured image 2b captured immediately after the captured image 2a, a cutout frame 3b and a detection frame 5b are set in the image. The positions of the cutout frame 3b and the detection frame 5b in the image are the same as those of the cutout frame 3a and the detection frame 5a in the captured image 2a. That is, the cutout frames 3a and 3b are set at predetermined positions in the image, and the detection frames 5a and 5b are set at predetermined positions in the image different from the cutout frames 3a and 3b.

[0052] The control unit of the image sensor IS performs a detection process of detecting the barcode BC by performing image processing in the detection frame 5 b on the captured image 2 b. Furthermore, the control unit of the image sensor IS calculates the position of the center of gravity of the barcode BC, and specifies the position of the barcode BC. Furthermore, the control unit of the image sensor IS calculates the moving speed of the barcode BC from the difference between the center of gravity position of the barcode BC of the captured image 2a and the center of gravity position of the barcode BC of the captured image 2b and the length of the frame period.

[0053] For the captured image 2c captured after the captured image 2b, the control unit of the image sensor IS performs processing for estimating the predicted position of the barcode BC on the captured image 2c, and sets the cutout frame 3c so as to include the predicted position.

[0054] Similarly, for the captured images 2d and 2e, the control unit of the image sensor IS estimates the predicted positions of the barcode BC on the captured images 2d and 2e, and sets the cutout frames 3d and 3e.

[0055] The control unit of the image sensor IS does not perform image processing for detecting the barcode BC on the captured images 2c, 2d, and 2e. As a result, the processing load of the control unit of the image sensor IS can be reduced, and a reduction in power consumption can be achieved. Note that the period of setting the cutout frame 3 based on the predicted position of the barcode BC is referred to as a "tracking period".

[0056] When the tracking period of barcode BC ends, that is, when the estimation of the predicted position of barcode BC ends, as Figure 5 As shown in the captured images 2y and 2z, the control unit of the image sensor IS sets the detection frames 5y and 5z again.

[0057] That is, the control unit of the image sensor IS specifies the position of the barcode BC in the detection frames 5y and 5z. At this time, the control unit of the image sensor IS performs the detection processing and the center of gravity position calculation processing of the barcode BC. In addition, the control unit of the image sensor IS calculates the moving speed of the barcode BC based on the difference between the center of gravity positions of the barcode BC in the detection frames 5y and 5z.

[0058] The calculated moving speed is used for comparison with the moving speed calculated based on the captured images 2a and 2b. As a result, the correctness of the moving speed calculated for the barcode BC can be determined. Note that the moving speed calculated based on the captured images 2a and 2b is the moving speed at the beginning of the tracking period, and is therefore referred to as the "moving speed at the beginning". Similarly, the moving speed calculated based on the captured images 2y and 2z is referred to as the "moving speed at the end".

[0059] Note that, when there is no need to determine the correctness of the moving speed, the cutout frames 3y and 3z may be set based on the predicted positions in the captured images 2y and 2z without specifying the position of the barcode BC through image processing.

[0060] <3. Functional Configuration of Signal Processing Device> Will refer to Figure 6 The functional configuration of the image sensor IS as a signal processing device is described. Note that each function is realized by hardware processing of the image signal processing unit 42 of the image sensor IS or software processing of the in-sensor control unit 43 .

[0061] The captured image data supplied from the imaging unit 41 of the image sensor IS is transferred to the binarization processing unit 10 .

[0062] Since the target object to be detected is the barcode BC, the binarization processing unit 10 performs a process of comparing the pixel value of the captured image 2 with the pixel value used as a threshold value and binarizing the pixel value. Specifically, in the case where the pixel value is less than or equal to the threshold value, the pixel value of the pixel is set to "1", and in the case where the pixel value is higher than the threshold value, the pixel value of the pixel is set to "0". As a result, a binary image in which the pixel value of the black line portion of the barcode BC is "1" is generated.

[0063] The binarized image obtained as a result of the binarization process is supplied to the mask processing unit 11 .

[0064] The mask processing unit 11 performs mask processing to replace the pixel values ​​of the area outside the cutout frame 3 with “0”.

[0065] The mask image obtained as a result of the mask processing is supplied to the detection processing unit 12 in the subsequent stage.

[0066] The detection processing unit 12 sets the detection frame 5 in a predetermined area in the cutout frame 3 of the mask processing unit 11, and detects the barcode BC to be detected in the detection frame 5. In the case where the barcode BC is not detected, the subsequent processing is not performed.

[0067] In case the barcode BC is detected, the mask image is supplied to the moment processing unit 13 .

[0068] Note that the detection processing unit 12 is located at a subsequent stage of the mask processing unit 11, but may be configured to detect the barcode BC with respect to the image before binarization. That is, various positions of the detection processing unit 12 may be considered.

[0069] The moment processing unit 13 calculates the mn-order moment M(m, n). The mn-order moment M(m, n) is calculated by the following equation (1).

[0070] [Mathematical formula 1] M(m,n)=∑ 区域A (x coordinate m ×y coordinate n )…Formula (1)

[0071] Here, region A is a region where the pixel value in the mask image is "1", and is a black line region of the barcode BC. The moment processing unit 13 calculates M(0,0), M(1,0), and M(0,1).

[0072] Each calculated value is output to the center-of-gravity calculation unit 14 .

[0073] The center of gravity calculation unit 14 calculates the center of gravity position of the barcode BC using M(0,0), M(1,0) and M(0,1). Specifically, the x coordinate of the center of gravity position of the barcode BC is calculated by the following equation (2), and the y coordinate is calculated by the following equation (3).

[0074] [Mathematical formula 2]

[0075] [Mathematical formula 3]

[0076] Note that the center-of-gravity calculation unit 14 can be regarded as performing a process of detecting the position of the barcode BC. That is, the center-of-gravity calculation unit 14 functions as a position detection unit of the barcode BC.

[0077] The calculated center-of-gravity position of the barcode BC is supplied to the frame position calculation unit 15 .

[0078] The frame position calculation unit 15 calculates the position of the cutout frame 3 based on the supplied center-of-gravity position. As a result, the cutout frame 3 includes the detected barcode BC.

[0079] The calculated frame position is supplied to the frame setting unit 16. The processing of the frame setting unit 16 will be described later.

[0080] Note that the center-of-gravity position calculated by the center-of-gravity calculation unit 14 is also supplied to the movement amount calculation unit 17. Then, the output of the center-of-gravity position to the movement amount calculation unit 17 is performed for a plurality of frame images (captured images 2).

[0081] The movement amount calculation unit 17 calculates the movement amount of the barcode BC based on the center of gravity position of each of the plurality of captured images 2 supplied from the center of gravity calculation unit 14. Specifically, the difference between the center of gravity positions is calculated. The calculated movement amount is supplied to the movement speed acquisition unit 18 .

[0082] Furthermore, the difference between the imaging times of the plurality of frame images for which the center-of-gravity positions have been calculated is output from the difference time calculation unit 19 to the movement speed acquisition unit 18 .

[0083] The differential time calculation unit 19 uses the frequency (clock number) information to calculate the differential time between the frame start times of each of the plurality of captured images 2 for which the center of gravity position has been calculated. For example, in the case of calculating the center of gravity position of the captured images 2 as two consecutive frame images, the inverse of the frame rate is calculated as the differential time. The calculated differential time is output to the moving speed acquisition unit 18.

[0084] The moving speed acquisition unit 18 calculates the moving speed of the barcode BC based on the moving amount of the barcode BC calculated in the captured image 2 as a plurality of frame images and the differential time. The calculated moving speed is output to the position estimating unit 20 .

[0085] The position estimation unit 20 obtains the moving speed of the barcode BC from the moving speed acquisition unit 18 , and obtains the current center-of-gravity position of the barcode BC from the center-of-gravity calculation unit 14 .

[0086] The position estimation unit 20 estimates the predicted position of the barcode BC in the captured image 2 of the next frame based on the center of gravity position and the moving speed of the barcode BC. The estimated predicted position is output to the frame setting unit 16.

[0087] The frame setting unit 16 sets the cutout frame 3 in such a manner as to include the barcode BC. Note that the frame setting unit 16 functions as a selector. Specifically, the frame setting unit 16 selects one of the frame position obtained by performing image processing on the captured image 2 as the current frame image acquired from the frame position calculation unit 15 (output from the frame position calculation unit 15) and the frame position corresponding to the predicted position of the barcode BC based on the captured image 2 captured before the current frame image (output from the position estimation unit 20), sets the position of the cutout frame 3, and outputs the position to the image cutout unit 21.

[0088] for Figure 5 For the captured images 2a and 2b shown, the frame setting unit 16 sets the cutout frame 3 (first setting process) based on the frame position information obtained from the frame position calculation unit 15. That is, for the captured images 2a and 2b, the barcode BC is detected by image processing, and the position of the cutout frame 3 is set according to the center of gravity position.

[0089] Then, for Figure 5In the captured images 2c, 2d, and 2e shown, the frame setting unit 16 sets the cutout frame 3 based on the predicted position of the barcode BC obtained from the position estimation unit 20 (second setting process). That is, the cutout frame 3 is set using the predicted position based on the moving speed of the barcode BC calculated based on the difference between the center of gravity positions of the barcode BC obtained from the captured images 2a and 2b.

[0090] The frame position of the cutout frame 3 set by the frame setting unit 16 is output to the image cutout unit 21 .

[0091] Note that, in other words, the frame setting unit 16 starts the first setting processing in response to the change from the non-detection state in which the barcode BC is not detected to the detection state in which the barcode BC is detected, and performs the processing of switching to the second setting processing in response to the calculation of the moving speed of the barcode BC by the position estimation unit 20.

[0092] Not only the frame position of the cutout frame 3 but also the captured image data are input to the image cutout unit 21. The image cutout unit 21 cuts out the ROI image 4 as a partial image based on the cutout frame 3. The cutout ROI image 4 is supplied to the output data shaping processing unit 22.

[0093] The information on the moving speed of the barcode BC obtained by the moving speed acquisition unit 18 is output to the comparison unit 23 .

[0094] The comparison unit 23 compares the moving speed of the barcode BC when the calculation of the predicted position of the barcode BC starts and the moving speed of the barcode BC when the calculation of the predicted position stops. That is, a comparison is made between the moving speed at the start (i.e., the moving speed at the start point of the tracking period) and the moving speed at the end (i.e., the moving speed at the end point of the tracking period). Specifically, it will be based on Figure 5 The moving speed of the barcode BC calculated based on the captured images 2a and 2b is shown to be compared with the moving speed of the barcode BC calculated based on the captured images 2y and 2z.

[0095] In the case where the difference between the two moving speeds is less than the threshold value, the comparison unit 23 determines that the tracking of the position of the barcode BC has been performed normally. In addition, in the case where the difference between the two moving speeds is greater than or equal to the threshold value, it is determined that the tracking of the position of the barcode BC has not been performed normally. For example, the determination result is provided to the output data shaping processing unit 22 as flag information.

[0096] The output data shaping processing unit 22 is supplied with not only the ROI image 4 but also the information of the moving speed of the barcode BC obtained by the moving speed obtaining unit 18 and the comparison result of the moving speed.

[0097] For example, the output data shaping processing unit 22 stores each of these information in each area of ​​a data structure that complies with the mobile industry processor interface (MIPI) standard, and sends the information to a processing unit in a subsequent stage. For example, the processing unit in the subsequent stage is Figure 1 The control unit 33 shown, a control unit included in a device provided outside the camera apparatus 1, or the like.

[0098] Here, we will refer to Figure 7 An example illustrating a data structure compliant with the MIPI standard.

[0099] Packet data that complies with the MIPI standard and is output within a frame period (hereinafter referred to as "MIPI data") is sent as a series of packet data (row data), which starts with a frame start ("FS" in the figure) and ends with a frame end ("FE" in the figure).

[0100] MIPI data is sent as frame data, where multiple packets are connected between the start and end of the frame. Each packet data includes a packet header ("PH" in the figure), a payload area, and a packet tail ("PF" in the figure), and various types of transmission data are stored in the payload area.

[0101] exist Figure 6 In the example shown, first, after the start of a frame, packets in which embedded data ("embedded data" in the figure) are stored in the payload area are sent (row data LD1, LD2, LD3, and LD4), and next, packets in which data of an ROI image 4 cut out using a cut frame 3 is stored in the payload area are sent (row data LD5, LD6, ..., LD11, L12), and further, packets in which embedded data are stored in the payload area are sent (row data LD13, LD14, LD15, and LD16).

[0102] Figure 6 The output data shaping processing unit 22 shown stores the ROI image 4 cut out by the image cutout unit 21 as cutout image data in the payload area of ​​the line data LD5 , LD6 , . . . , LD11 , and LD12 . Furthermore, the output data shaping processing unit 22 stores the information of the moving speed supplied from the moving speed acquisition unit 18 and the information of the comparison result supplied from the comparison unit 23 as embedded data in the payload areas of the line data LD1 , LD2 , LD3 , and LD4 .

[0103] Note that the moving speed information and the comparison result information may be stored in the payload area of ​​the line data LD13, LD14, LD15, and LD16. However, by sending the moving speed information and the comparison result information before sending the ROI image 4, it is easy to end the setting change of the object conveying device, the imaging setting change of the camera device 1, and the like before outputting the ROI image 4 of the next frame. Therefore, it is possible to quickly perform processing to cope with a situation where the barcode BC cannot be followed. Note that in the case where the barcode BC cannot be followed normally, in addition to the speed change of the object conveying device, the process of returning from the above-mentioned second setting process to the first setting process can also be performed. That is, the state can be returned to the state of specifying the barcode BC by image processing without using the predicted position.

[0104] Note that the information on the moving speed and the information on the comparison result are output as metadata of the cut-out image data. In addition, the information on the center-of-gravity position of the barcode BC may also be output as metadata.

[0105] <4. Processing Flow> Will refer to Figure 8 The flow of processing performed by the control unit (image signal processing unit 42 or in-sensor control unit 43) of the image sensor IS will be described. Note that Figure 8 The illustrated order of the respective processes is an example, and the order of some processes may be performed one after another.

[0106] In step S101, the control unit of the image sensor IS determines whether it is a tracking cycle. The tracking cycle is a cycle for setting the cutout frame 3 based on the predicted position of the barcode BC as described above.

[0107] For example, in a state where the barcode BC is not detected, it is determined that it is not a tracking period. In this case, in step S102 , the control unit of the image sensor IS acquires the captured image 2 from the imaging unit 41 .

[0108] Subsequently, the control unit of the image sensor IS performs binarization processing in step S103 and then performs mask processing in step S104. Note that since each processing has been described above, its detailed description will be omitted. The details of each processing described below may be omitted.

[0109] Subsequently, in step S105 , the control unit of the image sensor IS executes a detection process of the barcode BC. The target area of ​​the detection process is an area within the detection frame 5 set on the captured image 2 .

[0110] In the case where no barcode BC is detected, e.g. Figure 4In the case of the captured image 2A shown, the control unit of the image sensor IS returns to the process of step S101 again.

[0111] On the other hand, in the case where it is determined that the detection has been successfully performed, e.g. Figure 4 In the case of the captured image 2C shown, the control unit of the image sensor IS performs moment processing in step S107.

[0112] The control unit of the image sensor IS calculates the center of gravity position of the barcode BC in the subsequent step S108 , and calculates the position of the cutout frame 3 in step S109 .

[0113] The control unit of the image sensor IS calculates the position of the cutout frame 3 based on the image processing by executing a series of processing shown in steps S102 to S109. As a result, the cutout frame 3 is set based on the image processing. This is the above-mentioned first setting processing.

[0114] In step S110, the control unit of the image sensor IS determines whether the movement amount can be calculated. Figure 5 When the center of gravity position of the barcode BC is detected in both the captured image 2a and the captured image 2b shown, it is determined that the movement amount can be calculated.

[0115] In this case, the control unit of the image sensor IS calculates the movement amount in step S111 , calculates the difference time in step S112 , and calculates the movement speed in step S113 .

[0116] That is, the control unit of the image sensor IS calculates the moving speed of the barcode BC required to estimate the predicted position of the barcode BC by performing a series of processes shown in steps S111 to S113 .

[0117] In addition, in obtaining Figure 5 At the time point of the captured image 2 a shown, since there is only one piece of information about the center of gravity position of the barcode BC, it is determined in step S110 that the movement amount cannot be calculated. In this case, the control unit of the image sensor IS proceeds to step S114 without performing each process in steps S111 , S112 , and S113 .

[0118] In step S114 , the control unit of the image sensor IS determines whether the moving speeds can be compared. For example, in Figure 5 At the time after the process of step S113 is executed for the captured image 2z shown, the starting movement speed based on the captured images 2a and 2b and the ending movement speed based on the captured images 2y and 2z are obtained. Therefore, the two movement speeds can be compared.

[0119] In this case, the control unit of the image sensor IS performs a moving speed comparison process in step S115 , and supplies the comparison result to the output data shaping processing unit 22 .

[0120] On the other hand, in obtaining Figure 5 Since only one moving speed is obtained at the time point of the captured image 2 b shown, it is determined in step S114 that the moving speeds cannot be compared. In this case, the processing in step S115 is avoided.

[0121] The control unit of the image sensor IS performs a process of cutting out the ROI image 4 based on the set cutout frame 3 in step S116 , and performs a process of outputting the ROI image 4 and metadata in step S117 . Note that in the output process of step S117, in the case where the moving speed is calculated, the moving speed is output as metadata. Also, in the case where the comparison result of the moving speed is calculated, the comparison result is output as metadata.

[0122] After the process of step S117 , the control unit of the image sensor IS returns to the process of step S101 .

[0123] In the case where it is determined in the determination process of step S101 that it is the tracking cycle, the control unit of the image sensor IS proceeds to the process of step S118 . For example, in getting Figure 5 At the time when the images 2c, 2d and 2e are captured as shown, it is determined that it is a tracking period. In this case, each process in steps S102 to S109, that is, the image processing for specifying the position of the barcode BC is not performed.

[0124] In step S118, the control unit of the image sensor IS performs a predicted position estimation process. Furthermore, in step S119, the control unit of the image sensor IS performs a process of setting the cutout frame 3 based on the predicted position. This is the second setting process described above.

[0125] Thereafter, the control unit of the image sensor IS performs each process in steps S116 and S117 to perform image cutout processing and data output processing.

[0126] <5. Second Implementation Method> The image sensor IS as the signal processing device in the second embodiment has a configuration capable of providing information on the moving speed of the barcode BC from the outside. Will refer to Fig. 9 This will be described in detail.

[0127] Fig. 9is the same as in the first embodiment Figure 6 The corresponding figures show the functional structure of the image sensor IS.

[0128] Note that references will be omitted as appropriate. Figure 6 The components described are similar to the components described above.

[0129] The movement speed acquisition unit 18 can calculate the movement speed of the barcode BC based on the movement amount of the barcode BC calculated by the movement amount calculation unit 17 and the differential time calculated by the differential time calculation unit 19 .

[0130] Furthermore, the moving speed acquisition unit 18 in the present embodiment is configured to be capable of acquiring and employing the moving speed input from the outside of the image sensor IS or the outside of the camera apparatus 1 , rather than calculating the moving speed.

[0131] For example, the moving speed input from the outside is a moving speed input when the moving speed of the object flowing on the line is known by the setting of the object conveying device or the like, and is information on the speed setting.

[0132] In the case where information on the moving speed is input from the outside, by further specifying the position of the barcode BC of one photographed image 2, the movement of the barcode BC thereafter can be specified, and the predicted position of the barcode BC can be estimated.

[0133] Therefore, for the captured image 2 which is a subsequent frame image, each processing in the binarization processing unit 10, the mask processing unit 11, the detection processing unit 12, the moment processing unit 13, the center of gravity calculation unit 14, the frame position calculation unit 15, the movement amount calculation unit 17, the differential time calculation unit 19 and the comparison unit 23 can be omitted, and the processing load can be reduced.

[0134] <6. Third Implementation Plan> The image sensor IS as the signal processing device in the third embodiment calculates the moving speed of the barcode BC outside the image sensor IS or outside the camera device 1 . Will refer to Fig.10 This will be described in detail.

[0135] Fig.10 is the same as in the first embodiment Figure 6 and the second embodiment Fig. 9 The corresponding figures show the functional structure of the image sensor IS.

[0136] Note that references will be omitted as appropriate. Figure 6 The components described are similar to the components described above.

[0137] The center-of-gravity calculation unit 14 supplies the calculated center-of-gravity position to the output data shaping processing unit 22. As a result, the center-of-gravity position of the barcode BC is output to the control unit 33 outside the image sensor IS or to a device outside the camera apparatus 1. Here, these devices are referred to as "external equipment".

[0138] The external device calculates the moving speed of the barcode BC based on the received plurality of gravity center positions. Then, the external device supplies the calculated moving speed and the information of the received gravity center positions to the moving speed acquisition unit 18 of the image sensor IS.

[0139] The moving speed acquisition unit 18 receives information on the position of the center of gravity and information on the moving speed from an external device, and supplies these pieces of information to the position estimation unit 20 .

[0140] The position estimation unit 20 estimates the predicted position of the barcode BC based on the information of the center-of-gravity position and the information of the moving speed, and supplies the predicted position to the frame setting unit 16 .

[0141] In this way, the image sensor IS sets the cutout frame 3 using the information of the moving speed of the barcode BC calculated in the external device.

[0142] In addition, in the case where the moving speed acquisition unit 18 obtains a plurality of moving speeds, by outputting the information of the plurality of moving speeds to the comparison unit 23, the comparison unit 23 performs a process of comparing the two moving speeds and determines whether the tracking of the position of the bar code BC has been performed normally. For example, the determination result is provided to the output data shaping processing unit 22 as flag information.

[0143] By calculating the moving speed of the barcode BC outside the image sensor IS, the amount of calculation in the image sensor IS is reduced, and the processing load is reduced.

[0144] Note that the function of the comparison unit 23 may be provided by a device external to the image sensor IS or a device external to the camera apparatus 1 .

[0145] <7. Conclusion> As described in each of the above examples, the image sensor IS as a signal processing device includes: a position detection unit (center of gravity calculation unit 14), which detects the position of a target object in a captured image 2 as an object position based on a captured image 2 captured by an imaging unit 41 that captures a moving target object (e.g., a barcode BC); a position estimation unit 20, which estimates the position of the target object in the captured image 2 after the next frame as a predicted position based on the detected object position and the moving speed of the target object; and a frame setting unit 16, which sets a cutout frame 3 in the captured image 2 after the next frame according to the predicted position. As a result, the cutout frame 3 can be set to include the position of the target object predicted based on the object position and the moving speed. Therefore, by performing image processing on the captured image 2 of each frame, the frequency of image processing is reduced compared to the case where the position of the object is detected each time, so that the processing load can be reduced. Note that, for example, the target object moves in one direction at a constant speed. For example, the term "constant speed" used here includes that even if the moving speed of the target object slightly changes due to vibration or the like during the operation of the driving mechanism of the belt conveyor, the target object can be considered to move at a substantially constant speed as a whole. Furthermore, as for the cutout frame 3 , a rectangular frame has been exemplified in the above example, but the cutout frame 3 may have any shape such as a circular shape, an elliptical shape, or a star shape.

[0146] As reference Figure 6 As described above, the image sensor IS as a signal processing device may include a moving speed acquisition unit 18 that acquires the moving speed of the target object (for example, the barcode BC). By acquiring the moving speed of the target object, the predicted position can be estimated with high accuracy based on the object position of the target object.

[0147] As reference Figure 6 As described above, the moving speed acquisition unit 18 of the image sensor IS as a signal processing device can calculate the moving speed based on the difference between the positions of the object in the captured images 2 of a plurality of frames. Specifically, the moving speed acquisition unit 18 can calculate the moving speed of the target object (eg, barcode BC) based on the length of the frame period and the difference (distance) between the object positions. That is, the moving speed of the target object can be calculated by image processing. For example, when the moving speed of the target object cannot be obtained, the moving speed may be specified based on the captured image 2 .

[0148] As reference Fig. 9 As described above, the moving speed acquisition unit 18 of the image sensor IS as a signal processing device can acquire information input about the moving speed. For example, the moving speed acquisition unit 18 can acquire the moving speed input by the user, or can acquire the moving speed of the belt conveyor input by processing performed in other devices. That is, the predicted position of the target object (e.g., barcode BC) can be calculated based on the moving speed input from the outside. For example, when the moving speed of the target object cannot be obtained by image processing of the captured image 2 , information on the moving speed can be obtained.

[0149] As reference Figure 6As described above, the image sensor IS as a signal processing device may include a center of gravity calculation unit 14 that calculates the center of gravity position of a target object (e.g., a barcode BC) on a captured image 2, and a position detection unit (center of gravity calculation unit 14) may detect the object position based on the center of gravity position. That is, the object position is calculated by calculating the center of gravity position of the object in each of the two captured images 2. Then, the movement amount of the object is specified by the difference between the object positions. As a result, the movement speed of the object can be calculated by image processing. Therefore, even when the specific position of the target object cannot be detected, if the center of gravity position can be calculated, the moving speed of the target object can be calculated.

[0150] As reference Figure 6 As described above, the image sensor IS as a signal processing device may include an output unit (output data shaping processing unit 22) that outputs the center of gravity position. By outputting the center of gravity position, the moving speed of the target object (eg, barcode BC) can be calculated in a device outside the signal processing device. That is, the processing load on the signal processing device can be reduced.

[0151] As reference Fig.10 As described above, the image sensor IS as the signal processing device may include a moving speed acquisition unit 18 that acquires the moving speed based on the center-of-gravity position output from the output unit (output data shaping processing unit 22) calculated externally. The predicted position can be estimated using the moving speed of the target object (eg, barcode BC) calculated externally. Therefore, the processing load of the signal processing device can be reduced.

[0152] As reference Figure 2 As described above, the position detection unit (center of gravity calculation unit 14 ) of the image sensor IS as a signal processing device can detect the position of an object in the detection frame 5 set as a partial area of ​​the captured image 2 . As a result, the area that is the detection target of the target object (for example, the barcode BC) can be limited to the area in the detection frame 5, and the processing load of the process of detecting the target object can be reduced.

[0153] As reference Figure 2 As described above, the frame setting unit 16 of the image sensor IS as a signal processing device can set the cutout frame 3 so as to include the target object (for example, the barcode BC) detected in the detection frame 5 . As a result, the cutout image including the target object can be output to other devices with a small processing load. In addition, the burden of image processing in the device at the subsequent stage can be reduced compared to outputting the entire captured image 2.

[0154] As reference Figure 7 As described above, the image sensor IS as the signal processing device may include an output unit (output data shaping processing unit 22 ) that outputs the moving speed acquired by the moving speed acquisition unit 18 . As a result, the moving speed and the like calculated in the signal processing device can be presented to the user in other devices. Therefore, the user can determine whether the moving speed is normal, and the convenience is improved.

[0155] As reference Figure 6 As described above, the frame setting unit 16 of the image sensor IS as a signal processing device can switch between a first setting process of setting the cutout frame 3 based on the object position detected in the captured image 2 and a second setting process of setting the cutout frame 3 based on the predicted position. As a result, the setting of the cutout frame 3 including the target object (e.g., barcode BC) in the first setting process based on the image processing and the setting of the cutout frame 3 in the second setting process based on the predicted position without performing the image processing can be selectively used. Therefore, it is possible to selectively use the cutout frame 3 including the object position specified with high accuracy and the cutout frame 3 including the specified predicted position while reducing the processing load according to the situation. Note that in the case where the target object moves at a substantially constant speed in one direction, the predicted position can be calculated with high accuracy, and therefore, a reduction in processing load and high-accuracy tracking of the target object can be achieved.

[0156] As reference Figure 8 As described above, the position detection unit (the center of gravity calculation unit 14 ) of the image sensor IS as the signal processing device may not perform the process of detecting the position of the object in the captured image 2 in the second setting process. As a result, the processing load during execution of the second setting process is reduced.

[0157] As reference Figure 8 As described above, the frame setting unit 16 of the image sensor IS as a signal processing device can start the first setting processing in response to the change from the non-detection state in which the target object (for example, the barcode BC) is not detected to the detection state in which the target object is detected, and switch from the first setting processing to the second setting processing in response to the calculation of the moving speed, and the image sensor IS may include a moving speed acquisition unit 18 and an output unit (output data shaping processing unit 22), wherein the moving speed acquisition unit 18 calculates and acquires the starting moving speed and the ending moving speed, wherein the starting moving speed is the moving speed at the beginning of the tracking period for executing the second setting processing, and the ending moving speed is the moving speed at the end of the tracking period, and the output unit outputs the comparison result between the starting moving speed and the ending moving speed. By comparing the moving speed at the start and the moving speed at the end, the correctness of the calculated moving speed can be determined. Therefore, by presenting the comparison result in another information processing device that outputs the comparison result, the user can understand the correctness of the calculated moving speed. Note that the amount of movement per frame period can be calculated instead of the movement speed.

[0158] As reference Figure 7 As described above, the image sensor IS as a signal processing device may include an output unit (output data shaping processing unit 22), which outputs the cropped image data (data of ROI image 4) cropped by the cropping frame 3 and metadata of the cropped image data, and the metadata may include at least one of the moving speed and the center of gravity position of the target object (for example, the barcode BC) on the captured image 2, and is stored in an embedded data area in a data structure that complies with the mobile industry processor interface (MIPI: mobile industry processor interface) standard. As a result, both the cropped image data and the metadata may be transmitted in a data format conforming to the MIPI standard.

[0159] As reference Figure 7 As described above, in an image sensor IS serving as a signal processing device, metadata can be stored in a payload area of ​​row data sent before row data in which cut image data (data of ROI image 4) is stored among multiple row data included in the same frame data in a data structure compliant with the MIPI standard. As a result, in an information processing device that has received data in accordance with the MIPI standard, the time from receiving metadata to receiving the next frame of data can be extended. Therefore, the setting of the belt conveyor and the change of the imaging setting based on the metadata can be completed before receiving the next frame of data. In other words, the setting can be quickly changed based on the metadata.

[0160] As reference Figure 2 As described above, in the image sensor IS as a signal processing device, the target object may be a barcode (a one-dimensional code, a two-dimensional code, etc.) provided on the object. For example, the cutting frame 3 is set for each frame according to the moving speed of the belt conveyor so that the one-dimensional code or two-dimensional code attached to the surface of the packaging material of the product enters the frame. Therefore, it is easy to check whether the barcode is forgotten to be attached to the packaging material.

[0161] As reference Figure 6As described above, the image sensor IS as a signal processing device may include a binarization processing unit 10 and a position detection unit (center of gravity calculation unit 14), wherein the binarization processing unit 10 generates a binarized image obtained by binarizing the captured image 2, and the position detection unit may perform object position detection on the binarized image. For a barcode formed in black and white, image processing is simplified by binarizing the captured image 2, and the processing load can be reduced.

[0162] As reference Figure 1 As described above, the signal processing device may include an image sensor IS having an imaging unit 41 . Setting of the cutout frame 3 following the target object (eg, barcode BC), cutting of the image using the cutout frame 3, and the like are performed in the image sensor, whereby the amount of calculation of the processing performed outside the image sensor can be reduced.

[0163] The signal processing method according to the present technology enables a computer device to perform the following processing: based on a captured image of an imaging unit that captures a moving target object, detecting the position of the target object in the captured image as the object position; based on the detected object position and the moving speed of the target object, estimating the position of the target object in the captured image after the next frame as the predicted position; and setting a cutout frame in the captured image after the next frame according to the predicted position.

[0164] For example, a program according to the present technology is a program for causing an image signal processing unit 42 and an in-sensor control unit 43 included in an image sensor IS to perform the following functions: a function of detecting the position of a target object in a captured image as an object position based on a captured image of an imaging unit that captures a moving target object; a function of estimating the position of the target object in a captured image after the next frame as a predicted position based on the detected object position and the moving speed of the target object; and a function of setting a cropping frame in the captured image after the next frame according to the predicted position. By using such a procedure, in the image sensor IS as the signal processing device, it is possible to reduce the processing load of the ROI function for tracking the subject.

[0165] Such a program may be recorded in advance in a hard disk drive (HDD) as a recording medium built into a device such as a computer device, a ROM in a microcomputer having a CPU, or the like. Alternatively, the program may be temporarily or permanently stored (recorded) in a removable recording medium such as a floppy disk, a compact disc read only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disc (DVD), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such a removable recording medium may be provided as so-called packaged software. Furthermore, such a program may be installed from a removable recording medium into a personal computer or the like, or may be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0166] Note that the effects described in this specification are merely examples and not limitations, and other effects may be provided.

[0167] Furthermore, the above-described examples may be combined in any manner, and even in the case of using various combinations, the various functions and effects described above can be obtained.

[0168] <8. This technology> Note that the present technology can also adopt the following configurations. (1) A signal processing device, comprising: a position detection unit that detects, based on an image captured by an imaging unit that captures a moving target object, a position of the target object in the captured image as an object position; a position estimating unit that estimates the position of the target object in the captured image after the next frame as a predicted position based on the moving speed of the target object and the detected object position; and A frame setting unit sets a cutout frame in the captured image in the next frame and thereafter according to the predicted position. (2) The signal processing device according to (1) above further includes: A moving speed acquisition unit is used to acquire the moving speed of the target object. (3) According to the signal processing device described in (2) above, The moving speed acquisition unit calculates the moving speed based on a difference between positions of the object in the captured images of a plurality of frames. (4) The signal processing device according to any one of (2) to (3) above, The moving speed acquisition unit acquires information input about the moving speed. (5) The signal processing device according to any one of (1) to (4) above, further comprising: A center of gravity calculation unit calculates a center of gravity position of the target object on the captured image, wherein the position detection unit detects the object position based on the center of gravity position. (6) The signal processing device according to (5) above further includes: An output unit outputs the center of gravity position. (7) The signal processing device according to (6) above further includes: A moving speed acquisition unit acquires the moving speed calculated externally based on the center-of-gravity position output from the output unit. (8) The signal processing device according to any one of (1) to (7) above, The position detection unit detects the position of the object in a detection frame set as a partial area of ​​the captured image. (9) According to the signal processing device described in (8) above, The frame setting unit sets the cutout frame so that the target object detected in the detection frame is included in the cutout frame. (10) The signal processing device according to any one of (2) to (4) above, further comprising: An output unit outputs the moving speed acquired by the moving speed acquisition unit. (11) The signal processing device according to any one of (1) to (10) above, The frame setting unit is switchable between a first setting process for setting the cutout frame based on a position of the object detected in the captured image and a second setting process for setting the cutout frame based on the predicted position. (12) According to the signal processing device described in (11) above, Here, in the second setting process, the position detection unit does not perform a process of detecting the position of the object in the captured image. (13) The signal processing device according to any one of (11) to (12) above, wherein the frame setting unit starts the first setting process in response to a change from a non-detection state in which the target object is not detected to a detection state in which the target object is detected, and switches from the first setting process to the second setting process in response to calculation of the movement speed, and The signal processing device comprises: a moving speed acquisition unit that calculates and acquires a start moving speed that is the moving speed at the start of a tracking period in which the second setting process is executed, and an end moving speed that is the moving speed at the end of the tracking period; and An output unit outputs a comparison result between the moving speed at the start and the moving speed at the end. (14) The signal processing device according to any one of (1) to (13) above, further comprising: an output unit that outputs cut image data cut by the cut frame and metadata of the cut image data, The metadata includes at least one of the moving speed and the center of gravity position of the target object on the captured image, and The metadata is stored in an embedded data area in a data structure that complies with the mobile industry processor interface (MIPI) standard. (15) According to the signal processing device described in (14) above, The metadata is stored in a payload area of ​​line data sent before the line data storing the cut image data among a plurality of line data included in the same frame data in a data structure conforming to the MIPI standard. (16) The signal processing device according to any one of (1) to (15) above, Wherein, the target object is a barcode set on an object. (17) The signal processing device according to (16) above further includes: a binarization processing unit that generates a binarized image obtained by binarizing the captured image, Wherein, with respect to the binary image, the position detection unit detects the position of the object. (18) The signal processing device according to any one of (1) to (17) above, further comprising: An image sensor comprises the imaging unit. (19) A signal processing method for causing a computer device to perform the following processing: Based on a captured image of an imaging unit capturing a moving target object, detecting a position of the target object in the captured image as an object position; estimating the position of the target object in the captured image after the next frame as a predicted position based on the moving speed of the target object and the detected position of the object; and A cutout frame is set in the captured image in the next frame and thereafter according to the predicted position. Reference numerals list

[0169] 2, 2A, 2B, 2C, 2D, 2E Capture images 2a, 2b, 2c, 2d, 2e, 2y, 2z Capture images 3, 3A, 3B, 3C, 3D, 3E Cut box 3a, 3b, 3c, 3d, 3e, 3y, 3z Cut box 4, 4A, 4B, 4C, 4D, 4E ROI images (cropped images) 4a, 4b, 4c, 4d, 4e, 4y, 4z ROI images (cropped images) 5, 5A, 5B, 5C, 5D, 5E detection frame 5a, 5b, 5y, 5z detection frame 10 Binarization processing unit 14 Center of gravity calculation unit (position detection unit) 16 Frame setting unit 18 Movement speed acquisition unit 20 Position estimation unit 22 Output data shaping processing unit (output unit) 41 Imaging Unit IS Image Sensor BC Barcode (target)

Claims

1. A signal processing device, comprising: a position detection unit that detects, based on an image captured by an imaging unit that captures a moving target object, a position of the target object in the captured image as an object position; a position estimating unit that estimates the position of the target object in the captured image after the next frame as a predicted position based on the moving speed of the target object and the detected position of the object; and A frame setting unit sets a cutout frame in the captured image in the next frame and thereafter according to the predicted position.

2. The signal processing device according to claim 1, further comprising: A moving speed acquisition unit is used to acquire the moving speed of the target object.

3. The signal processing device according to claim 2, in, The moving speed acquisition unit calculates the moving speed based on a difference between positions of the object in the captured images of a plurality of frames.

4. The signal processing device according to claim 2, in, The moving speed acquisition unit acquires information input regarding the moving speed.

5. The signal processing device according to claim 1, further comprising: a center of gravity calculation unit, which calculates the center of gravity position of the target object on the captured image, Wherein, the position detection unit detects the position of the object based on the center of gravity position.

6. The signal processing device according to claim 5, further comprising: An output unit outputs the center of gravity position.

7. The signal processing device according to claim 6, further comprising: A moving speed acquisition unit acquires the moving speed calculated externally based on the center-of-gravity position output from the output unit.

8. The signal processing device according to claim 1, in, The position detection unit detects the position of the object in a detection frame set as a partial area of ​​the captured image.

9. The signal processing device according to claim 8, in, The frame setting unit sets the cutout frame so that the target object detected in the detection frame is included in the cutout frame.

10. The signal processing device according to claim 2, further comprising: An output unit outputs the moving speed acquired by the moving speed acquisition unit.

11. The signal processing device according to claim 1, in, The frame setting unit is switchable between a first setting process of setting the cutout frame based on the position of the object detected in the captured image and a second setting process of setting the cutout frame based on the predicted position.

12. The signal processing device according to claim 11, in, In the second setting process, the position detection unit does not perform a process of detecting the position of the object in the captured image.

13. The signal processing device according to claim 11, in, The frame setting unit starts the first setting process in response to a change from a non-detection state in which the target object is not detected to a detection state in which the target object is detected, and switches from the first setting process to the second setting process in response to calculation of the movement speed, and The signal processing device comprises: a moving speed acquisition unit that calculates and acquires a start moving speed that is the moving speed at the start of a tracking period in which the second setting process is executed, and an end moving speed that is the moving speed at the end of the tracking period; and An output unit outputs a comparison result between the moving speed at the start and the moving speed at the end.

14. The signal processing device according to claim 1, further comprising: an output unit that outputs cut image data cut by the cut frame and metadata of the cut image data, The metadata includes at least one of the moving speed and the center of gravity position of the target object on the captured image, and The metadata is stored in an embedded data area in a data structure compliant with the Mobile Industry Processor Interface (MIPI: mobile industry processor interface) standard.

15. The signal processing device according to claim 14, in, The metadata is stored in a payload area of ​​line data transmitted before the line data storing the cut image data among a plurality of line data included in the same frame data in a data structure conforming to the MIPI standard.

16. The signal processing device according to claim 1, in, The target is a barcode set on an object.

17. The signal processing device according to claim 16, further comprising: a binarization processing unit that generates a binarized image obtained by binarizing the captured image, Wherein, with respect to the binary image, the position detection unit detects the position of the object.

18. The signal processing device according to claim 1, further comprising: An image sensor comprises the imaging unit.

19. A signal processing method, for causing a computer device to perform the following processing: Based on a captured image of an imaging unit capturing a moving target object, detecting a position of the target object in the captured image as an object position; estimating the position of the target object in the captured image after the next frame as a predicted position based on the moving speed of the target object and the detected position of the object; and A cutout frame is set in the captured image in the next frame and thereafter according to the predicted position.

Images