Image processing method, image processing device, and program

Abstract

The present disclosure relates to an image processing method, an image processing device, and a program that make it possible to appropriately correct rolling shutter distortion. The present invention corrects rolling shutter distortion in an image captured by a rolling shutter method by combining culling correction for correcting the rolling shutter distortion by means of a culling method and frame interpolation correction for correcting the rolling shutter distortion by means of a frame interpolation method. The present invention can be applied to a rolling shutter-type imaging device.

Description

Image processing method, image processing device, and program

[0001] The present disclosure relates to an image processing method, an image processing device, and a program, and more particularly to an image processing method, an image processing device, and a program that are capable of appropriately suppressing rolling shutter distortion.

[0002] In an imaging device that captures images using the rolling shutter method, exposure and readout are performed sequentially on a line-by-line basis in an imaging element that has multiple pixels arranged in a two-dimensional array. In other words, the exposure timing differs for each line of pixels.

[0003] Therefore, if the positional relationship between the imaging device and the subject changes during imaging, differences in exposure timing on a line-by-line basis can cause image distortion (so-called rolling shutter distortion) depending on the degree of shift in the imaging position.

[0004] Therefore, a technology has been proposed to suppress rolling shutter distortion by thinning out the pixels that are exposed and read out on a line-by-line basis when capturing an image using the rolling shutter method, and interpolating the pixels of the thinned-out lines (see Patent Document 1).

[0005] Furthermore, a technique has been proposed for suppressing rolling shutter distortion by calculating the amount of deviation on a line-by-line basis based on the positional relationship between the imaging device and the subject, and then performing frame interpolation through resampling (see Patent Document 2).

[0006] International Publication No. 2020 / 035043 Japanese Patent Application Laid-Open No. 2012-253778

[0007] However, the technology of Patent Document 1 reduces rolling shutter distortion by thinning out the pixels that are exposed and read out on a line-by-line basis, and therefore, theoretically, it is not possible to completely suppress rolling shutter distortion.

[0008] Furthermore, in Patent Document 2, it is known that when a subject that is moving at a short distance and at high speed is corrected, a phenomenon called Jello occurs.

[0009] Furthermore, a simple combination of Patent Documents 1 and 2 does not allow for appropriate weighting. In particular, Patent Document 1 can only infer subjects moving at a constant speed in a straight line, so even if weighting is set, it does not follow changes in the image. For this reason, it has not been possible to accurately represent subjects moving at a constant speed in rotation, pendulum motion, or pendulum motion around a fulcrum with measurable precision.

[0010] In either case, rolling shutter distortion cannot be sufficiently suppressed, and in particular cannot be suppressed with sufficient precision for application to visual effects techniques such as VFX (Visual Effects) used in TV dramas and movies. For example, when attempting to key a subject waving their hands at high speed, conventional techniques result in the subject blending into the background or collapsing like Jello, resulting in excessively processed edges, resulting in an insufficient visual effect.

[0011] The present disclosure has been made in light of such circumstances, and in particular, aims to appropriately suppress rolling shutter distortion.

[0012] An image processing method according to one aspect of the present disclosure is an image processing method that includes correcting rolling shutter distortion in an image captured using a rolling shutter method by combining thinning correction, which corrects the distortion using a thinning method, and frame interpolation correction, which corrects the distortion using a frame interpolation method.

[0013] An image processing device and a program according to one aspect of the present disclosure include a thinning correction unit that corrects rolling shutter distortion in an image captured using a rolling shutter method using a thinning method, and a frame interpolation correction unit that corrects the rolling shutter distortion using a frame interpolation method, and corrects the rolling shutter distortion by combining the correction by the thinning correction unit and the correction by the frame interpolation correction unit.

[0014] In one aspect of the present disclosure, rolling shutter distortion of an image captured by the rolling shutter method is corrected by a combination of thinning correction, which corrects by the thinning method, and frame interpolation correction, which corrects by the frame interpolation method.

[0015] 5 is a diagram illustrating rolling shutter distortion. FIG. 6 is a diagram illustrating thinning correction. FIG. 7 is a diagram illustrating frame interpolation correction. FIG. 8 is a diagram illustrating an overview of the rolling shutter distortion correction method of the present disclosure. FIG. 9 is a diagram illustrating a configuration example of a preferred embodiment of an image processing system of the present disclosure. FIG. 10 is a diagram illustrating weighting of thinning correction and frame interpolation correction. FIG. 11 is a diagram illustrating the distortion correction amount per unit time of one line estimated from a target motion path for each subject. FIG. 12 is a diagram illustrating the distortion correction amount per unit time of one line estimated from the estimated motion path of the imaging device. FIG. 13 is a diagram illustrating the distortion correction amount per unit time of one line, which is the sum of the distortion amount per unit time of one line estimated from the estimated motion path of the imaging device and the distortion amount per unit time of one line estimated from the estimated motion path of the imaging device. FIG. 14 is a flowchart illustrating image processing by the image processing system of FIG. 5. FIG. 15 is a flowchart illustrating imaging processing (including thinning correction processing). FIG. 16 is a diagram illustrating examples of a velocity map, a distance map, and a distance confidence map corresponding to an RGB image. FIG. 17 is a flowchart illustrating weighting processing. FIG. 18 is a flowchart illustrating frame interpolation correction processing. FIG. 19 is a diagram illustrating an overview of a modified example of the rolling shutter distortion correction method of the present disclosure. Fig. 10 is a flowchart illustrating imaging processing (including thinning-out correction processing) in a modified example. Fig. 11 is a diagram illustrating an example of a velocity map, a distance map, a distance confidence map, and a weighting map corresponding to an RGB image. Fig. 12 is a flowchart illustrating weighting processing in a modified example. Fig. 13 is a flowchart illustrating frame interpolation correction processing. Fig. 14 is a diagram illustrating an example configuration of an application example of an image processing system according to the present disclosure. Fig. 15 is a flowchart illustrating clonic convulsion monitoring processing. Fig. 16 is a diagram illustrating an example configuration of a general-purpose computer.

[0016] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0017] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Overview of the present disclosure 2. Preferred embodiments 3. Modifications 4. Application examples 5. Examples of implementation by software

[0018] <<1. Overview of the Present Disclosure>> The present disclosure is directed to, in particular, appropriately suppressing rolling shutter distortion. Therefore, first, an overview of the present disclosure will be described.

[0019] <Rolling Shutter Distortion> For example, consider a state in which a rectangular subject Mx is moving at a predetermined speed in the right direction indicated by the arrow in the real space Pr on the left side of Fig. 1. In this state, when the subject Mx is imaged by an imaging device made up of a rolling shutter type imaging element, rolling shutter distortion occurs.

[0020] Each of the pixels constituting the imaging element, which is made up of a plurality of pixels arranged in a two-dimensional array, generates a pixel signal according to the amount of incident light incident from the real space Pr.

[0021] In a rolling shutter type image sensor, as shown in the center of FIG. 1, pixel signals of the pixels constituting the image sensor are read out sequentially in time series in units of lines on which the pixels are arranged.

[0022] In the center of Fig. 1, the vertical axis represents lines, the horizontal axis represents time, and the exposure period and readout timing of each pixel in units of a line are shown. Note that in the center of Fig. 1, the exposure period of the pixels constituting each line is represented by period tp, and an example is shown in which the readout line order is from top to bottom.

[0023] That is, as shown in the center of Figure 1, exposure of each pixel in a line unit starts from the top row first. Then, after the exposure period tp has elapsed, each pixel in the top row unit is read out. After that, exposure of each pixel in the line below starts, and after the exposure period tp has elapsed, reading is performed. Next, exposure of each pixel in the line below starts, and after the exposure period tp has elapsed, reading is performed. The above process is repeated line by line.

[0024] As a result, the exposure period and readout timing are delayed by a predetermined time for each line from top to bottom. Therefore, when the pixel signals output from each pixel of the image sensor are combined, an image Ps as shown in the right part of Figure 1 is generated as the captured image.

[0025] Consider the relationship between the upper left end Pt and lower left end Pb of the subject Mx in the real space Pr and the upper left end Pt' and lower left end Pb' of the image Mx' in the image Ps, which is the corresponding imaging result.

[0026] As described above, the exposure period and readout timing are gradually delayed from top to bottom in units of lines. As a result, the readout timing of pixels capturing the image of the lower left end Pb of the subject Mx in real space Pr is delayed by a time tda relative to the readout timing of pixels capturing the image of the upper left end Pt of the subject Mx in real space Pr, as shown in the center of Figure 1.

[0027] At this time, the subject Mx is moving to the right in the figure at a predetermined speed, and therefore moves to the right in the figure by the delay time tda even during the period in which the imaging element is capturing the image of the real space Pr.

[0028] Therefore, at the lower left end Pb of the subject Mx, exposure begins at a position moved to the right from the upper left end Pt at a predetermined speed by the delay time tda, and reading is performed after the exposure period tp.

[0029] As a result, on the image Ps that is the imaging result, the lower left end Pb' of the image Mx' corresponding to the subject Mx is shifted from the upper left end Pt' by a width da, which corresponds to the distance moved to the right in the figure by the delay time tda at a predetermined speed.

[0030] All pixel signals constituting the image Mx' corresponding to the moving subject Mx are treated as an image that has been moved to the right by the delay time in units of lines. Therefore, the rectangular subject Mx in the real space Pr appears as an image Mx' that is distorted as a whole into a parallelogram on the image Ps that is the captured image.

[0031] The image distortion that occurs when capturing an image using the rolling shutter method is called rolling shutter distortion.

[0032] 1 illustrates an example of rolling shutter distortion that occurs when a subject moves relative to an imaging device. However, a similar phenomenon occurs when the positional relationship between the imaging device and the subject changes over time. Therefore, for example, a similar phenomenon occurs when the imaging device pans its imaging direction even when the subject is not moving.

[0033] <Distortion Correction by Thinning-Out Method> Generally, two methods are known to address this rolling shutter distortion: rolling shutter distortion correction by thinning out method and rolling shutter distortion correction by frame interpolation method. Hereinafter, rolling shutter distortion correction by thinning out method will be referred to as thinning-out correction, and rolling shutter distortion correction by frame interpolation method will be referred to as frame interpolation correction.

[0034] As explained with reference to the central portion of FIG. 1, rolling shutter distortion is a phenomenon that occurs when pixels are read out line by line.

[0035] The thinning correction is a correction that suppresses rolling shutter distortion by thinning out the number of lines to be read out and shortening the overall readout time, as shown in the lower center of FIG.

[0036] The upper left, upper center, and upper right parts of Fig. 2 are the same as those in Fig. 1. The lower left part of Fig. 2 is the same as the real space Pr in Fig. 1. The lower center part of Fig. 2 is a diagram explaining thinning readout. The lower right part of Fig. 2 is an image Ps'' that is the imaging result corrected by the thinning method.

[0037] The subject Mx is moving to the right in the figure at a predetermined speed. However, as shown in the lower center of Figure 2, the lines of pixels to be read are thinned out, resulting in a decrease in the overall number of lines to be read. As a result, exposure begins at the lower left edge Pb of the subject Mx at a position moved to the right from the upper left edge Pt at a predetermined speed by a delay time tdb (<tda), and reading is performed after an exposure period tp.

[0038] As a result, on the image Ps'', which is the captured image, the lower left end Pb'' of the image Mx'' is positioned at a position shifted by a width db (<da) corresponding to the distance moved to the right in the figure by the delay time tdb (<tda) at a predetermined speed relative to the upper left end Pt''.

[0039] As a result, the horizontal movement distance in units of lines of all pixel signals constituting the image Mx'' corresponding to the moving subject Mx becomes smaller than that of all pixel signals constituting the image Mx'. Therefore, the rolling shutter distortion of the image Mx'' is suppressed more than that of the image Mx'.

[0040] <Distortion correction using frame interpolation> Distortion correction using frame interpolation is a correction method that suppresses rolling shutter distortion by using pixel signals from multiple consecutive frames and angular velocity and position information of the imaging device to determine the amount of correction per line, and then performing frame interpolation by resampling the pixel signals so as to correct them based on the determined amount of correction.

[0041] More specifically, the correction amount per line is a combination of an estimated motion correction amount, which is a correction amount per line based on the actual motion of the imaging device when capturing an image estimated by a gyro sensor or the like, and a correction amount per line based on the target motion, which is the movement path of the subject. Although a gyro sensor or the like was mentioned above, this may also be an acceleration sensor, SLAM (Simultaneous Localization and Mapping), or a combination thereof.

[0042] First, multiple frames and motion data measured by a gyro sensor or the like of the imaging device are acquired, and the motion data and frames are associated to determine an estimated motion correction amount, which is the correction amount per line. The estimated motion correction amount per line is, for example, the detection results of a gyro sensor or the like actually used by the imaging device when capturing an image, as shown in the left part of Figure 3, smoothed as shown in the right part of Figure 3. Although a gyro sensor or the like has been mentioned above, this may also be an acceleration sensor, SLAM, or a combination thereof.

[0043] Furthermore, the correction amount for each line in the target motion, which is the motion of the subject, is the target motion correction amount based on the target motion.

[0044] In the frame interpolation method, a correction amount for each line is determined based on an estimated motion correction amount and a target motion correction amount, and then resampling is performed according to the correction amount, thereby suppressing rolling shutter distortion.

[0045] <Rolling Shutter Distortion Correction According to the Present Disclosure> For example, as shown in FIG. 4, consider a case where a scene in real space Po is captured by a rolling shutter imaging device, the scene being composed of a subject Mo that is rectangular like the subject Mx in FIG. 1 and moves to the right at a predetermined speed, and flag-like subjects F1o and F2o that are provided in fixed positions in the background and flutter at high speed.

[0046] When such a real space Po is captured by a rolling shutter imaging device, the image M corresponding to the subject Mo is captured with rolling shutter distortion such as a parallelogram as it moves, as shown in image PI in the upper center of Figure 4. In contrast, the images F1 and F2 corresponding to the subjects F1o and F2o are not moving, and are therefore captured without rolling shutter distortion.

[0047] It is conceivable to correct the rolling shutter distortion by applying distortion correction by a conventional thinning method to this image PI and then applying distortion correction by a frame interpolation method.

[0048] That is, image Pt is generated by applying thinning correction to image PI. In image Pt, image M corresponding to a moving subject Mo is converted to image Mt. This suppresses distortion in the horizontal direction. Images F1t and F2t corresponding to images F1 and F2 are not subject to rolling shutter distortion and remain unchanged.

[0049] However, as described above, thinning correction suppresses distortion by adjusting the number of lines to be read out depending on the number of lines to be thinned out. Therefore, the amount of distortion that can be suppressed is determined by the number of lines that can be thinned out. Therefore, when a certain degree of rolling shutter distortion occurs, it is necessary to thin out all of the lines between the first and last lines in order to completely suppress the distortion, which is not realistic. As a result, thinning correction cannot completely remove rolling shutter distortion from an image Mt corresponding to an image M of a moving subject Mo.

[0050] Furthermore, by applying frame interpolation correction to this image Pt, for example, an image Pa is generated. In the image Pa, an image Mt corresponding to the object Mo is made into an appropriately corrected image Ma.

[0051] In contrast, the flag-shaped subjects F1o and F2o flap at high speed. For this reason, an error known as Jello may occur in the images F1a and F2a corresponding to the images F1t and F2t of the subjects F1o and F2o during frame interpolation correction. As a result, rolling shutter distortion may not be properly corrected.

[0052] In image Pa, the images F1a and F2a are depicted as if a square flag has been distorted into a parallelogram. However, this does not depict the actual distortion that occurs, such as rolling shutter distortion, but merely indicates that an error such as Jello has occurred.

[0053] Therefore, the present disclosure sets zones for each subject on a line-by-line basis, sets the weights (degrees) of thinning correction and frame interpolation correction for each zone according to the speed, distance, and distance reliability of the subject within the zone, and combines the respective correction methods to correct rolling shutter distortion.

[0054] That is, for example, as shown in the real space Po in Figure 4, in a zone Za that can be set in line units where there is no image M corresponding to a moving subject Mo, the weight of the thinning correction is increased and the weight of the frame interpolation correction is decreased.

[0055] Furthermore, in zone Zb below line L, where image M and image F1 corresponding to moving subject Mo, which can be set on a line-by-line basis, exist, image M corresponding to subject Mo, which is large in size and moving at a predetermined speed, and image F1, which is small in size, distant, and not moving, exist. Therefore, for example, the weight of thinning correction and the weight of frame interpolation correction are set in a balanced manner, using large-sized image Ma as a reference.

[0056] By such processing, a large weight is set for the thinning correction and a small weight is set for the frame interpolation correction for the image F2 present in the zone Za, thereby suppressing the occurrence of errors such as Jello.

[0057] Furthermore, for the image Ma present in the zone Zb, the weight of the thinning correction and the weight of the frame interpolation correction are set in a balanced manner, so that it is possible to suppress rolling shutter distortion of the moving image Ma.

[0058] This makes it possible to appropriately suppress rolling shutter distortion in the image as a whole.

[0059] <<2. Preferred Embodiment>> Next, with reference to the block diagram of FIG. 5, a configuration example of an image processing system according to the present disclosure that is adapted to appropriately suppress rolling shutter distortion will be described.

[0060] The image processing system 101 in FIG. 3 is composed of an imaging device 111, a display processing unit 112, and a cloud storage 113.

[0061] The image capturing device 111 is a camera that captures images using a so-called rolling shutter method, and captures RGB images while applying thinning-out rolling shutter correction according to the subject.

[0062] In this case, the imaging device 111 identifies the amount of rolling shutter distortion, sets weights for thinning correction and frame interpolation correction, and controls the imaging of the RGB image so as to perform line thinning according to the weight of the thinning correction, thereby achieving thinning correction.

[0063] The imaging device 111 sets a frame interpolation correction amount for the amount of rolling shutter distortion according to the weight of the frame interpolation correction, and stores the set amount in the memory unit 191 of the cloud storage 113 together with the RGB image captured in a thinning-corrected state.

[0064] The display processing unit 112 acquires the RGB image and the frame interpolation correction amount stored in the storage unit 191 of the cloud storage 113 .

[0065] The imaging device 111 may directly supply information about the RGB image and the frame interpolation correction amount to the display processing unit 112. In this case, the display processing unit 112 acquires the RGB image and the frame interpolation correction amount supplied from the imaging device 111.

[0066] When the display processing unit 112 acquires the RGB image captured with thinning correction applied and the frame interpolation correction amount, it applies frame interpolation correction to the RGB image captured with thinning correction applied based on the frame interpolation correction amount to generate and display a stabilized frame RGB image, and stores the generated stabilized frame RGB image in the memory unit 191 of the cloud storage 113.

[0067] With the above series of configurations, thinning correction and frame interpolation correction are applied to an image captured using the rolling shutter method in an appropriate ratio depending on the subject, and a stabilized frame in which rolling shutter distortion has been corrected is generated.

[0068] The imaging device 111, the display processing unit 112, and the cloud storage 113 will be described in detail below.

[0069] The imaging device 111 generates a distance map, a distance reliability map, a velocity map, a subject map (including subject information and a zone map described later), thinning settings, imaging device information, motion data which is information on the position and orientation of the imaging device, weights for each of the thinning correction and frame interpolation correction, and a frame interpolation correction amount, which correspond to the captured RGB image, and outputs and stores them in a memory unit 191 in the cloud storage 113.

[0070] When capturing RGB images as a video, the imaging device 111 generates a distance map, a distance reliability map, a velocity map, a subject map (including subject information and a zone map, which will be described later), thinning settings, imaging device information, motion data which is information on the position and orientation of the imaging device, weights for each of the thinning correction and frame interpolation correction, and a frame interpolation correction amount, based on the timestamp assigned to each frame.

[0071] More specifically, the imaging device 111 includes a distance sensor 121 , a distance map generation unit 122 , an RGB sensor 123 , an RGB image signal processing unit 124 , a six-axis sensor 125 , a motion detection unit 126 , and a thinning correction unit 127 .

[0072] The distance sensor 121 is, for example, a LiDAR, a depth camera, or a stereo camera, and measures the distance to the subject in pixel units and outputs the pixel-by-pixel distance measurement results to the distance map generation unit 122.

[0073] The distance map generating unit 122 generates a distance map by mapping the distances that are the distance measurement results in pixel units supplied from the distance sensor 121 , and outputs the distance map to the thinning correction unit 127 .

[0074] The RGB sensor 123 is an imaging element formed, for example, of a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge-Coupled Device) image sensor, and generates pixel signals according to the amount of incident light based on a predetermined RGB array, and outputs the signals to the RGB image signal processing unit 124.

[0075] The RGB sensor 123 generates an RGB image signal by thinning out pixel signals in units of a predetermined number of lines based on the thinning setting supplied from the thinning correction unit 127 .

[0076] The RGB image signal processing unit 124 performs predetermined signal processing on the RGB pixel signals supplied from the RGB sensor 123 to generate an RGB image, and outputs the image to a thinning correction unit 127 .

[0077] When the RGB sensor 123 is performing thinning imaging based on the thinning setting, the RGB image signal processing unit 124 generates an RGB image by interpolating pixel signals of the thinned lines based on the thinning setting, and outputs the generated RGB image to the thinning correction unit 127.

[0078] The six-axis sensor 125 detects the acceleration and angular velocity in three axial directions (for example, X, Y, and Z directions) of the imaging device 111 and outputs them to the motion detection unit 126 .

[0079] The motion detection unit 126 detects information about the position and orientation of the imaging device 111 in three-dimensional space based on the acceleration and angular velocity of the imaging device 111 in three-axis directions supplied from the six-axis sensor 125, and outputs the information to the thinning correction unit 127 as motion data of the imaging device 111.

[0080] The thinning correction unit 127 acquires the distance map from the distance map generation unit 122 , acquires the RGB image supplied from the RGB image signal processing unit 124 , and acquires motion data of the imaging device 111 from the motion detection unit 126 .

[0081] The thinning correction unit 127 determines the thinning setting of the RGB sensor 123 based on the distance map, the RGB image, and the motion data of the image capturing device 111 , and controls the image capturing by the image capturing device 111 .

[0082] Based on the distance map, RGB image, and motion data of the imaging device 111, the thinning correction unit 127 generates a distance reliability map, a velocity map, a subject map (including subject information), thinning settings, imaging device information, motion data which is information on the position and orientation of the imaging device, weights for each of the thinning correction and frame interpolation correction, and a frame interpolation correction amount, and outputs and stores them in the memory unit 191 in the cloud storage 113.

[0083] More specifically, the thinning correction unit 127 includes a semantic segmentation processing unit 141 , a distance confidence map generation unit 142 , a velocity map generation unit 143 , a weight setting unit 144 , a correction amount determination unit 145 , and a thinning setting unit 146 .

[0084] The semantic segmentation processing unit 141 generates an object map by identifying objects in an RGB image individually and in units of pixels through semantic segmentation. Note that as long as it is possible to identify objects in an RGB image and generate an object map, the process does not have to be semantic segmentation; for example, the semantic segmentation processing unit 141 may be configured with a DNN (Deep Neural Network), a recognizer, or AI (Artificial Intelligence) that has undergone machine learning to recognize objects, or may use instance segmentation or panoptic segmentation.

[0085] The distance reliability map generating unit 142 generates a distance reliability map in units of pixels from the magnitude of change in distance between frames in a time series of pixels belonging to the same subject, based on the subject map and the distance map. That is, if the distance information of pixels belonging to the same subject changes suddenly in the distance reliability map, the reliability of the distance information in the distance map is set low, and if the change in the distance information is gradual, the reliability of the distance information is set high.

[0086] The velocity map generating unit 143 generates a velocity map by mapping the apparent velocity of each object in units of pixels from the positional change of each object in the RGB image between frames in a time series.

[0087] Based on the RGB image, the object map, the distance map, the distance reliability map, and the velocity map, the weighting setting unit 144 sets the range in which each object exists in the RGB image as a zone with lines as units, generates a zone map, and sets weights for thinning correction and frame interpolation correction for each zone, and sets the amount of correction for each.

[0088] More specifically, the weight setting unit 144 sets zones line by line based on the subject in the RGB image, as described with reference to FIG. 4, based on the subject map.

[0089] Since zones are set in line units, if multiple subjects exist on the same line, there is a possibility that multiple subjects will be included in the same zone.

[0090] In such a case, the weight setting unit 144 sets a reference object from among multiple objects present in the same zone based on a predetermined condition, and sets the zone based on the set reference object. The predetermined condition may be the largest object, a specific type of object, the fastest object, the object with the highest distance reliability, or the closest object in the same zone. Alternatively, the reference object may be set by comparing these multiple conditions.

[0091] That is, the weighting setting unit 144 sets a reference subject under any of the conditions based on the RGB image, the subject map, the distance map, the distance reliability map, and the velocity map, and sets the thinning correction amount for the zone based on the reference subject.

[0092] The weight setting unit 144 sets weights based on the speed map, distance map, and distance reliability map so that the weight of the thinning correction amount increases as the speed of the reference subject increases, the distance decreases, and the distance reliability increases, as shown in FIG. 6, for example.

[0093] Since there is a limit to the amount of correction that can be performed with thinning correction, FIG. 6 shows an example in which the thinning correction weight is set to a maximum of about 3 / 4, and conversely, the frame interpolation correction weight is set to a minimum of about 1 / 4.

[0094] Note that Figure 6 shows an example in which the maximum value of the weight related to the thinning correction amount is approximately 3 / 4 and the minimum value of the weight related to the frame interpolation correction amount is approximately 1 / 4, but the maximum and minimum values ​​of the weight may be set in various ways based on the maximum value of the thinning correction amount to be used.

[0095] Furthermore, in the following explanation, the weights related to the thinning correction amount and the weights related to the frame interpolation correction amount will simply be referred to as thinning correction weights and frame interpolation correction weights, and for simplicity of explanation, we will proceed assuming that both are set to values ​​between 0 and 1.

[0096] More specifically, the weight setting unit 144 normalizes the velocity histogram based on the velocity map and sets the thinning velocity weight, which is set to a value closer to 1 as the velocity increases and to 0 as the velocity decreases.

[0097] The weight setting unit 144 normalizes the distance histogram based on the distance map and sets a thinning distance weight, which is set to a value closer to 1 for shorter distances and closer to 0 for longer distances.

[0098] The weight setting unit 144 normalizes the histogram of the distance confidence based on the distance confidence map and sets the thinned distance confidence weight, which is set to a value closer to 1 as the distance confidence is higher and to a value closer to 0 as the distance confidence is lower.

[0099] The weight setting unit 144 then multiplies the thinning speed weight, the thinning distance weight, and the thinning distance reliability weight to set a thinning correction weight, and also subtracts the thinning correction weight from 1 to set a frame interpolation correction weight.

[0100] The correction amount determination unit 145 calculates the amount of rolling shutter distortion for each subject for each zone, and determines the amount of thinning correction and the amount of frame interpolation correction by multiplying the amount of rolling shutter distortion for each subject by the weight set for each zone by the weight setting unit 144.

[0101] That is, the correction amount determination unit 145 calculates the thinning correction amount for each zone as the rolling shutter distortion amount x the thinning correction weight, and calculates the frame interpolation correction amount as the rolling shutter distortion amount x the frame interpolation correction weight.

[0102] The amount of rolling shutter distortion (angle) for each subject is expressed as (the line direction angular velocity in the world coordinate system relative to the image capture device + the line direction angular velocity for each subject relative to the world coordinate system) x the readout delay time per line before thinning out. The amount of rolling shutter distortion (angle) for each subject can be calculated from the motion data, RGB image, distance map, and velocity map of the image capture device 111.

[0103] However, the amount of rolling shutter distortion after thinning correction is applied is affected by the apparent angular velocity due to the thinning. Therefore, after thinning correction is applied, the line direction angular velocity for each subject relative to the world coordinate system is the apparent angular velocity per line for the angle of view of each subject / thinning correction amount. That is, for example, if the thinning setting is set so that 1 / 4 of all lines are thinned, the apparent angular velocity is multiplied by four.

[0104] The thinning setting unit 146 determines the specific number of lines to be thinned as a thinning setting based on the thinning correction amount determined by the correction amount determination unit 145, supplies the setting to the RGB sensor 123, and controls thinning imaging. The thinning setting unit 146 also supplies the thinning setting to the RGB image signal processing unit 124. That is, the RGB image signal processing unit 124 generates an RGB image by interpolating pixels of the thinned lines using pixel signals from the RGB sensor 123 that have been thinned out based on the thinning setting.

[0105] The display processing unit 112 includes a frame interpolation correction unit 161 and a display unit 162 .

[0106] The frame interpolation correction unit 161 reads out the RGB image, distance confidence map, velocity map, subject map (including subject information and zone map), motion data, and frame interpolation correction amount from the memory unit 191 of the cloud storage 113, and applies frame interpolation correction to the thinning-corrected RGB image to generate a stabilized frame RGB image, which is output to the display unit 162 for display.

[0107] At this time, the frame interpolation correction unit 161 stores stabilized motion data, which is motion data of the imaging device 111 corrected when generating the stabilized frame RGB image, a stabilized subject map, which is a subject map in the stabilized frame RGB image, a stabilization distance map corresponding to the stabilized frame RGB image, a stabilization distance reliability map, and a stabilization speed map in the memory unit 191 of the cloud storage 113.

[0108] More specifically, the frame interpolation correction unit 161 includes a motion trajectory estimation unit 181 , a target motion trajectory estimation unit 182 , and a motion correction unit 183 .

[0109] The motion path estimation unit 181 smooths the motion of the image capture device 111 based on the motion data, which is time-series position and orientation data of the image capture device 111, and corrects the motion data.

[0110] The motion trajectory estimation unit 181 maps each pixel of the RGB image into a 3D space and matches the motion data for each time-series frame based on the timestamp to estimate the motion trajectory of the image capture device 111. Hereinafter, the estimated motion trajectory of the image capture device 111 will also be referred to as an estimated motion trajectory.

[0111] The target motion path estimation unit 182 estimates the movement of a reference subject set for each zone as a target motion path based on the subject map, the distance map, and information on each pixel of the RGB image mapped in 3D space.

[0112] The motion correction unit 183 estimates the amount of correction per unit time for one line from the motion path of the image capture device 111 to a target motion path, which is the reference motion of the subject set for each zone. Then, based on the amount of correction per unit time for one line, the motion correction unit 183 resamples the RGB image so as to correct it by the frame interpolation correction amount, thereby generating a stabilized frame RGB image.

[0113] More specifically, for example, if the movement path of the target subject can be estimated to be uniform linear motion, the apparent angular velocity (amount of distortion) per line relative to the subject's angle of view will be the angular velocity when the subject, that is, the target, moves uniformly in a straight line with the imaging device 111 fixed.

[0114] Here, as shown by the solid arrows in Figure 7, consider a case where the angular velocity of the second line relative to the first line is expressed by angular velocity ω12, the angular velocity of the third line relative to the second line is expressed by angular velocity ω23, the angular velocity of the fourth line relative to the third line is expressed by angular velocity ω34, and the angular velocity of the fifth line relative to the fourth line is expressed by angular velocity ω45.

[0115] In this case, the apparent angular velocity per line relative to the angle of view of the subject can be considered to be the average value ω of the angular velocities between the lines (=(ω12+ω23+ω34+ω45) / 4).

[0116] 7 shows the difference in the imaging position of the subject on lines 1 to 5 (lines 1 to 5) when the same subject moving to the right is imaged using the global shutter method and the rolling shutter method with the imaging device 111 fixed, with the left part being an example of the global shutter method and the right part being an example of the rolling shutter method. In other words, the global shutter method is a method in which no delay occurs on a line-by-line basis, and so in order to compare the case in which a delay occurs with the case in which no delay occurs, an example of the global shutter method is shown on the left part of each of FIGS. 7 to 9.

[0117] Furthermore, if the movement path of the imaging device 111 can be estimated to be uniform linear movement based on the movement data, the angular velocity in the line direction of the world coordinate system relative to the imaging device 111 will be the angular velocity when the imaging device 111 is panned to the right at uniform linear movement to the right and an image of a fixed subject, that is, a target, is captured.

[0118] Here, as shown by the dotted arrows in Figure 8, consider a case where the angular velocity of the second line relative to the first line is expressed by angular velocity ω'12, the angular velocity of the third line relative to the second line is expressed by angular velocity ω'23, the angular velocity of the fourth line relative to the third line is expressed by angular velocity ω'34, and the angular velocity of the fifth line relative to the fourth line is expressed by angular velocity ω'45.

[0119] In this case, the angular velocity in the line direction of the world coordinate system for the imaging device 111 can be considered to be the average value ω' (=(ω'12+ω'23+ω'34+ω'45) / 4) of the angular velocities between the lines.

[0120] Note that Figure 8 shows the difference in the position of the subject on lines 1 to 5 (lines 1 to 5) when the same fixed subject is imaged using the global shutter method and the rolling shutter method with the imaging device 111 panned to the right, with the left part being an example of the global shutter method and the right part being an example of the rolling shutter method.

[0121] If the thick solid rectangle shown in Figure 9 is taken as the unit block, the amount of correction per unit time from the movement path of the imaging device 111 to the target movement path, which is the reference movement of the subject set for each zone, can be estimated to be the sum of the amount of correction per unit time when the subject moves to the right with the imaging device 111 fixed, as indicated by the solid arrow, and the amount of correction per unit time when the imaging device 111 pans to the right, as indicated by the dotted arrow.

[0122] In other words, if the thick solid rectangle shown in Figure 9 is taken as a unit block, the correction amount per unit time for one line of this block can be estimated to be the sum ω+ω' of the angular velocity ω, which is the correction amount per unit time for one line in the target movement, and the angular velocity ω', which is the correction amount per unit time for one line in the movement path of the imaging device 111.

[0123] Note that Figure 9 shows the difference in the position of the subject on lines 1 to 5 when the imaging device 111 is panned to the right and the same subject moving to the right is imaged using the global shutter method and the rolling shutter method, with the left part being an example of the global shutter method and the right part being an example of the rolling shutter method.

[0124] Furthermore, in Figures 7 to 9, we have explained cases where each path is uniform linear motion, but this is not limited to this, and it may be estimated as uniform rotational motion, or as a combination of uniform linear motion and uniform rotational motion, or in particular, the uniform rotational motion may be estimated as pendulum motion seen from the side, or in particular as pendulum motion at the same distance from the fulcrum, or in particular, as arm clonic convulsions, which are pendulum motion of the arm with the elbow as the fulcrum, or as leg clonic convulsions, which are pendulum motion of the leg with the knee as the fulcrum.

[0125] In this way, the motion correction unit 183 estimates the amount of correction per unit time from the motion path of the image capture device 111 to the target motion path, which is the reference motion of the subject set for each zone. Furthermore, the motion correction unit 183 resamples the RGB image so as to correct it by the frame interpolation correction amount based on the amount of correction per unit time, to generate a stabilized frame RGB image, and outputs it to the display unit 162 for display.

[0126] When generating the stabilized frame RGB image, the motion corrector 183 generates a stabilized object map, which is a subject map in the stabilized frame RGB image, a stabilization distance map, a stabilization distance reliability map, and a stabilization speed map corresponding to the stabilized frame RGB image, based on the positional relationship of each pixel in the resampled RGB image.The motion corrector 183 then stores the generated maps, together with stabilized motion data, which is motion data of the imaging device 111 corrected when generating the stabilized frame RGB image, in the storage unit 191 of the cloud storage 113.

[0127] <Image Processing> Next, with reference to the flowchart of FIG. 10, image processing for correcting rolling shutter distortion and displaying an image by the image processing system 101 of FIG. 5 will be described.

[0128] In step S11, the imaging device 111 executes imaging processing including thinning-out rolling shutter correction processing, applies thinning-out correction while capturing RGB images in successive frame units, and outputs and stores the imaging results in the memory unit 191 in the cloud storage 113.

[0129] In this case, the imaging device 111 generates weights for thinning correction and frame interpolation correction, as well as a frame interpolation correction amount, for each zone in which a subject exists in the RGB image, and corrects rolling shutter distortion by imaging the RGB image while applying thinning correction by the distortion correction amount corresponding to the thinning correction weight.

[0130] In addition, the imaging device 111 generates a distance map, a distance reliability map, a velocity map, a subject map (including subject information and a zone map), thinning settings, imaging device information, motion data which is information on the position and attitude of the imaging device, weights for thinning correction and frame interpolation correction for each zone in which a subject exists in the RGB image, and a frame interpolation correction amount, which correspond to the captured RGB image, and outputs and stores them in a memory unit 191 in the cloud storage 113.

[0131] The imaging process will be described in detail later with reference to the flowchart of FIG.

[0132] In step S12, the display processing unit 112 executes frame interpolation correction processing, acquires an RGB image from the storage unit 191 in the cloud storage 113, performs frame interpolation correction, and generates a stabilized frame RGB image.

[0133] At this time, the display processing unit 112 generates a stabilized frame RGB image by resampling and correcting the RGB image that has been subjected to thinning correction using frame interpolation correction by the frame interpolation correction amount that is set based on the respective weights of the thinning correction and frame interpolation correction supplied from the imaging device 111.

[0134] In addition, the display processing unit 112 generates stabilized motion data, which is motion data of the imaging device 111 corrected when resampling the stabilized frame RGB image, a stabilized subject map, which is a subject map in the stabilized frame RGB image, a stabilization distance map corresponding to the stabilized frame RGB image, a stabilization distance reliability map, and a stabilization speed map, and stores these in the memory unit 191 of the cloud storage 113.

[0135] The frame interpolation correction process will be described in detail later with reference to the flowchart of FIG.

[0136] In step S13, the display processing unit 112 outputs the generated stabilized frame RGB image to the display unit 162 for display.

[0137] Through the above processing, weights for thinning correction and frame interpolation correction are set for each zone in which a subject exists in the RGB image, and an image is captured with thinning settings made according to the set weights, resulting in an RGB image being captured with thinning correction applied.

[0138] Furthermore, the RGB image that has been subjected to thinning correction according to the weight is subjected to frame interpolation correction corresponding to the weight set for the frame interpolation correction, thereby generating and displaying a stabilized frame RGB image.

[0139] As a result, by applying thinning correction and frame interpolation correction, each of which is combined with an appropriate weight, to an RGB image captured using the rolling shutter method, it is possible to appropriately suppress rolling shutter distortion.

[0140] <Image Capture Processing (Including Thinning-Out Correction Processing)> Next, the image capture processing (including thinning-out correction processing) will be described with reference to the flowchart of FIG.

[0141] In step S31, the RGB sensor 123 generates RGB pixel signals according to the amount of incident light and outputs them to the RGB image signal processing unit 124. The RGB image signal processing unit 124 performs predetermined signal processing on the RGB pixel signals supplied from the RGB sensor 123 to generate an RGB image and outputs the generated RGB image to the thinning correction unit 127.

[0142] Since the process is performed using a sequence of multiple frames, the process of step S31 may be repeated initially until the multiple frames required for the process are captured.

[0143] Furthermore, in the initial processing, since no thinning setting is made, the RGB image signal processing unit 124 generates an RGB image that has not been thinned out and outputs it to the thinning correction unit 127 .

[0144] In step S32, distance sensor 121 measures the distance to the subject and outputs the pixel-by-pixel distance measurement result to distance map generation unit 122. Distance map generation unit 122 generates a distance map based on the pixel-by-pixel distance measurement result supplied from distance sensor 121, and outputs the distance map to thinning correction unit 127.

[0145] In step S33, the six-axis sensor 125 detects the acceleration and angular velocity of the image capture device 111 in three axial directions and outputs the results to the motion detection unit 126. The motion detection unit 126 detects information about the position and orientation of the image capture device 111 in three-dimensional space based on the acceleration and angular velocity of the image capture device 111 in the three axial directions supplied from the six-axis sensor 125, and outputs the information as motion data of the image capture device 111 to the thinning correction unit 127. The motion detection unit 126 converts the detection results in the local coordinate system supplied from the six-axis sensor 125 into a world coordinate system to generate motion data.

[0146] In step S34, the semantic segmentation processing unit 141 performs semantic segmentation based on the RGB image, identifies individual objects in the image, and generates an object map.

[0147] In step S35, the distance confidence map generation unit 142 generates a distance confidence map by mapping distance confidence from the magnitude of change in distance over time between pixels belonging to the same object based on the object map and the distance map, so that the smaller the change in pixel unit, the higher the confidence. Also, the velocity map generation unit 143 generates a velocity map by mapping velocity for each object in pixel unit from the position change of each object in the RGB image between frames in the time series.

[0148] In step S36, the weight setting unit 144 sets zones in line units based on the subject in the RGB image, based on the subject map, and generates a zone map that groups together the zones.

[0149] For example, let us consider a case where an RGB image such as the image P11 shown in Fig. 12 is captured, and two subjects are recognized: an airplane in the upper right part of the image P11, and its background. In reality, more detailed subjects such as the airplane's propeller, wings, windows, wheels, people and their limbs, etc. are recognized, but for simplicity's sake, we will consider only the airplane and the background as subjects.

[0150] In the case of image P11, the airplane recognized as the subject is closer to the imaging position than the background, so the airplane is set as the reference subject. Therefore, in the case of image P11, the weighting setting unit 144 sets the range above the line L on which the airplane is located as zone Z1, and sets the range below line L as zone Z2.

[0151] Furthermore, when the RGB image is image P11, the velocity map, distance map, and distance confidence map are expressed, for example, as velocity map Mv, distance map Md, and distance confidence map Mc in FIG.

[0152] In the example of Fig. 12, in the velocity map Mv, the faster the velocity of the subject, the whiter the corresponding region is represented by a distribution, and the slower the velocity of the subject, the darker the corresponding region is represented by a distribution. For this reason, in the velocity map Mv of Fig. 12, for example, in image P11, which is an RGB image, the main body of the airplane, which is close to the imaging position and traveling at almost the same speed as the imaging position, is represented in black because its apparent speed from the imaging position is zero, and the region where the rotating propeller is present is represented in white, indicating the fastest speed, and the background is represented in gray.

[0153] In the distance map Md, the closer the distance to the subject, the whiter the corresponding area is represented, and the farther the distance to the subject, the blacker the corresponding area is represented. For example, in the distance map Md of Figure 12, in image P11, which is an RGB image, the main body of the airplane is closer to the imaging position, so the main body of the airplane is represented in white and the distant background is represented in gray.

[0154] Furthermore, in the distance reliability map Mc, the higher the distance reliability, the whiter the corresponding area is represented, and the lower the distance reliability, the blacker the corresponding area is represented. For this reason, in the distance reliability map Mc of Figure 12, for example, in image P11, which is an RGB image, the background has little change in position, so it is represented in white, the airplane body, which has next little change, is represented in gray, and the areas near the propellers, wheels, and windows, which tend to blend into the background, have large changes and are therefore represented in black.

[0155] In step S37, the weight setting unit 144 sets one of the unprocessed zones as the processing target zone.

[0156] In step S38, the correction amount determination unit 145 calculates the amount of rolling shutter distortion (angle) based on the reference object present in the processing target zone. As described above, the amount of rolling shutter distortion (angle) for each object is expressed as (the line direction angular velocity of the world coordinate system relative to the image capture device 111 + the line direction angular velocity of the object relative to the world coordinate system) × the readout delay time per line.

[0157] The correction amount determination unit 145 calculates the line direction angular velocity of the subject in the world coordinate system relative to the image capture device 111 based on the motion data of the image capture device 111, and calculates the line direction angular velocity of the subject in the RGB image world coordinate system based on the time series velocity map. The correction amount determination unit 145 also calculates the readout delay time per line by multiplying the exposure time of the RGB sensor 123 by a specific coefficient that is stored in advance as information about the image capture device 111.

[0158] As described above, the amount of rolling shutter distortion in frames after thinning correction is applied is affected by the apparent angular velocity due to the thinning. Therefore, in frames after thinning correction, the line-direction angular velocity of each subject relative to the world coordinate system must be the apparent angular velocity per line relative to the angle of view of the subject / thinning amount. That is, for example, if the thinning amount is set to 1 / 4 of all lines, the line-direction angular velocity of each subject relative to the world coordinate system is multiplied by four.

[0159] In step S39, the weight setting unit 144 executes weighting processing to set weights for each of the thinning-out correction and the frame interpolation correction.

[0160] The weighting process will be described in detail later with reference to the flowchart of FIG.

[0161] In step S40, the correction amount determination unit 145 determines the amount of thinning correction and the amount of frame interpolation correction by multiplying the amount of rolling shutter distortion in the processing target zone by the weight set for each zone by the weight setting unit 144.

[0162] That is, the correction amount determination unit 145 calculates the amount of thinning correction for the processing target zone as the amount of rolling shutter distortion multiplied by the weight of the amount of thinning correction, and calculates the amount of frame interpolation correction as the amount of rolling shutter distortion multiplied by the weight of the amount of frame interpolation correction.

[0163] In step S41, the weight setting unit 144 determines whether or not there are any unprocessed zones, and if there are any unprocessed zones, the process returns to step S37. That is, the processes from steps S37 to S41 are repeated until weights for thinning-out correction and frame interpolation correction are set for all zones and the amounts of thinning-out correction and frame interpolation correction are determined.

[0164] Then, in step S41, weights for thinning correction and frame interpolation correction are set for all zones, the amount of thinning correction and the amount of frame interpolation correction are determined, and if it is determined that there are no unprocessed zones, processing proceeds to step S42.

[0165] In step S42, the thinning setting unit 146 determines the number of lines to be thinned in imaging by the RGB sensor as a thinning setting based on the thinning correction amount, and controls the RGB sensor 123 to capture images based on the thinning setting. In other words, if the thinning setting is set here, thinned imaging will be performed from then on. At this time, the thinning setting unit 146 also outputs the thinning setting, which is information on the number of lines to be thinned, to the RGB image signal processing unit 124.

[0166] In step S 43 , the RGB image signal processing unit 124 generates an RGB image by interpolating pixel signals of the lines thinned out by the RGB sensor 123 based on the thinning setting, and outputs the RGB image to the thinning correction unit 127 .

[0167] In step S44, the thinning correction unit 127 stores the thinning-corrected RGB image, the motion data of the imaging device 111, the distance map, the distance reliability map, the velocity map, the subject map (including subject information and a zone map), the imaging device information (the exposure time during imaging and the delay time per line), and the frame interpolation correction amount (including information on the weight of the frame interpolation correction) in the memory unit 191 of the cloud storage 113.

[0168] If necessary, the thinning correction unit 127 may directly output the thinning-corrected RGB image, movement data of the imaging device 111, a distance map, a distance reliability map, a velocity map, a subject map (including subject information and a zone map), imaging device information (exposure time during imaging and delay time per line), and a frame interpolation correction amount for each zone set based on the subject (including information on the weight of the frame interpolation correction) to the frame interpolation correction unit 161 of the display processing unit 112.

[0169] In step S45, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S32, and the subsequent steps are repeated.

[0170] Then, in step S45, if an instruction to end the process is given, the process ends.

[0171] Through the above processing, weights for thinning correction and frame interpolation correction are set for each zone based on the RGB image, distance map, and motion data, the amount of thinning correction and the amount of frame interpolation correction are determined according to the set weights, and an RGB image that has been thinned and corrected according to the amount of thinning correction is captured.The thinning-corrected RGB image, motion data of the image capture device 111, distance map, distance reliability map, velocity map, subject map (including subject information and zone map), image capture device information (exposure time during image capture and delay time per line), and the amount of frame interpolation correction for each zone are stored in the storage unit 191 of the cloud storage 113.

[0172] <Weighting Processing> Next, the weighting processing will be described with reference to the flowchart of FIG.

[0173] In step S61, the weight setting unit 144 initializes the thinning correction weight, which is the weight for the thinning correction of the processing target zone, to zero.

[0174] In step S62, the weighting setting unit 144 sets the thinning speed weight for the processing target zone according to the apparent speed, with the higher the speed being a value closer to 1 and the lower the speed being a value closer to 0. In other words, by setting the weight of the thinning correction larger according to the speed, it becomes possible to suppress the rolling shutter distortion that becomes larger as the speed becomes higher.

[0175] For example, in the velocity map Mv of Figure 12, if the processing target zone in which an object consisting of an airplane exists is zone Z1, the weight setting unit 144 sets a weight so that the thinning speed weight is minimized based on the velocity at which the area of ​​the nearest airplane is represented in black with the largest area.

[0176] For example, in the velocity map Mv of FIG. 12, when the processing target zone is zone Z2, the weight setting unit 144 sets the thinning velocity weight based on the velocity represented in gray.

[0177] In step S63, the weight setting unit 144 determines whether the movement of the subject that is the reference for the processing target zone is in accordance with the movement of the imaging device 111, based on the movement data of the imaging device 111.

[0178] In step S63, if the movement of the subject that is the reference for the processing target zone is a movement that follows the movement of the image capture device 111, the process proceeds to step S64. While it has been described above that the movement of the subject that is the reference for the processing target zone is a movement that follows the movement of the image capture device 111, this may also be the case when recording at a low resolution such as HD (High Definition), when image processing is performed for monitoring, or a combination of these.

[0179] In step S64, the weight setting unit 144 sets the thinning speed weight of the processing target zone to a value close to one.

[0180] In step S64, if the movement of the subject that is the reference for the processing target zone does not follow the movement of the image capturing device 111, the processing of step S64 is skipped. That is, in this case, a thinning speed weight is set according to the apparent speed.

[0181] In step S65, the weight setting unit 144 sets the thinning distance weight for the processing target zone based on the distance map so that the farther the distance, the closer it is to 0, and the closer it is to 1. In other words, when the distance is short, it is possible to suppress the occurrence of jello by reducing the weight of the frame interpolation correction.

[0182] For example, in the distance map Md of Figure 12, if the processing zone in which the subject consisting of an airplane exists is zone Z1, the weight setting unit 144 sets a weight so that the thinning distance weight is maximized based on the distance to the nearest position represented in white on the body of the airplane.

[0183] For example, in the distance map Md of FIG. 12, when the processing target zone is zone Z2, the weight setting unit 144 sets the thinning distance weight based on the distance represented in gray.

[0184] In step S66, the weight setting unit 144 sets the thinning distance confidence weight for the processing target zone to a value closer to 0 the lower the distance confidence, and to a value closer to 1 the higher the distance confidence, based on the distance confidence map. In other words, when the distance confidence is low and it is necessary to consider the possibility that the distance is close, it is possible to suppress the occurrence of jello by reducing the weight of the frame interpolation correction.

[0185] For example, in the distance confidence map Mc of Figure 12, if the processing target zone in which the subject consisting of an airplane exists is zone Z1, the weight setting unit 144 sets the thinning distance confidence weight based on the distance confidence that is expressed in the widest range of gray within the main body of the airplane.

[0186] For example, in the distance confidence map Mc of FIG. 12, when the processing target zone is zone Z2, the weight setting unit 144 sets the thinned distance confidence weight based on the distance confidence expressed in light gray.

[0187] In step S67, the weight setting unit 144 calculates a thinning weight by multiplying the thinning speed weight, the thinning distance weight, and the thinning distance reliability weight.

[0188] In step S68, the weight setting unit 144 subtracts the thinning weight from 1 to calculate a frame interpolation weight, which is a weight for frame interpolation correction.

[0189] Through the above processing, for each zone set for each subject, a thinning-out correction weight and a frame interpolation correction weight are set based on the velocity map, distance map, and distance reliability map in the area where the reference subject that constitutes the zone exists.

[0190] For each zone, the faster the speed of the reference subject, the closer the distance value of the reference subject, and the higher the reliability of the distance reliability map of the reference subject, the larger the thinning weight that is the weight for thinning correction is set, and the smaller the frame interpolation weight that is the weight for frame interpolation correction is set.

[0191] Furthermore, for each zone, the slower the speed of the reference subject, the farther the distance value of the reference subject, and the lower the distance reliability of the distance reliability map of the reference subject, the smaller the thinning weight that is the weight for thinning correction is set, and the larger the frame interpolation weight that is the weight for frame interpolation correction is set.

[0192] As explained with reference to FIG. 6, in reality, there is a limit to the amount of correction that can be made with thinning correction, so the weight associated with the amount of thinning correction is set to, for example, at most about 3 / 4, and conversely, the weight associated with the amount of frame interpolation correction is set to at most about 1 / 4.

[0193] Also, Figure 6 shows an example in which the maximum value of the weight related to the thinning correction amount is approximately 3 / 4 and the minimum value of the weight related to the frame interpolation correction amount is approximately 1 / 4, but the maximum and minimum values ​​of the weight may be set in various ways based on the maximum value of the thinning correction amount to be used.

[0194] <Frame Interpolation Correction Processing> Next, the frame interpolation correction processing will be described with reference to the flowchart of FIG.

[0195] In step S81, the frame interpolation correction unit 161 of the display processing unit 112 reads and acquires the thinning-corrected RGB image stored in the memory unit 191 of the cloud storage 113, the motion data of the imaging device 111, the distance map, the distance reliability map, the velocity map, the subject map, imaging device information (exposure time during imaging and delay time per line), and the frame interpolation correction amount (including information on the weight of the frame interpolation correction).

[0196] In step S82, the motion path estimation unit 181 maps each pixel of the RGB image into a 3D space.

[0197] In step S83, the motion path estimation unit 181 smooths the motion of the image capture device 111 based on the motion data, which is time-series position and orientation data of the image capture device 111, to correct the motion data.

[0198] In step S84, the motion path estimation unit 181 matches the motion data for each frame in time series based on the time stamps, and estimates the motion path of the image capture device 111.

[0199] In step S85, the target motion trajectory estimating unit 182 sets the unprocessed zone as the processing target zone.

[0200] In step S86, the target motion path estimation unit 182 estimates the movement of the reference subject set in the processing target zone as a target motion path based on the subject map, the distance map, and information on each pixel of the RGB image mapped in 3D space.

[0201] In step S87, the motion correction unit 183 estimates the amount of correction per unit time for one line from the estimated motion path of the imaging device 111 to the target motion path, which is the movement of the subject that serves as the reference and is set in the processing target zone.

[0202] In step S88, the motion correction unit 183 resamples the RGB image so as to correct it by the frame interpolation correction amount based on the correction amount per unit time of one line calculated in the processing of step S87, and generates a stabilized frame RGB image of the processing target zone.

[0203] In step S89, the motion correction unit 183 determines whether or not there is an unprocessed zone.

[0204] If it is determined in step S89 that there is an unprocessed zone, the process returns to step S85.

[0205] That is, the processes from steps S85 to S89 are repeated until stabilized frame RGB images are generated for all zones.

[0206] Then, in step S89, the motion correction unit 183 generates a complete stabilized frame RGB image by combining the stabilized frame RGB images of all the zones.

[0207] In step S91, when generating a stabilized frame RGB image, the motion correction unit 183 generates a stabilized object map, which is a subject map in the stabilized frame RGB image, a stabilization distance map, a stabilization distance reliability map, and a stabilization speed map corresponding to the stabilized frame RGB image, based on the positional relationship of each pixel in the RGB image generated by resampling, and stores these in the memory unit 191 of the cloud storage 113 together with stabilized motion data, which is the motion data of the imaging device 111 corrected when generating the stabilized frame RGB image.

[0208] By performing the above processing, frame interpolation correction is performed on the thinning-corrected RGB image, and it becomes possible to perform rolling shutter correction that combines both thinning correction and frame interpolation correction based on the thinning weight and frame interpolation weight that are set based on the subject's speed, distance, and distance reliability.

[0209] This makes it possible to combine thinning correction and frame interpolation correction in an appropriate balance based on weights set on the basis of the subject's speed, distance, and distance reliability, making it possible to appropriately suppress distortions that cannot be corrected by thinning correction alone and distortions that cannot be corrected by frame interpolation correction alone.

[0210] As a result, images captured using the rolling shutter method can be applied to high-definition visual effects technology such as VFX (Visual Effects). While we mentioned VFX (Visual Effects) above, this can also be used for clonic convulsion monitoring, which will be discussed later, or for photogrammetry, or for any video that requires accurate, measurable precision without the need to disguise fast-moving subjects, particularly subjects that move in uniform rotational motion, pendulum motion, or pendulum motion around a fulcrum, which conventional technology struggles to achieve.

[0211] In the above, an example has been described in which the image processing system 101 is configured in a state in which the imaging device 111, the display processing unit 112, and the cloud storage 113 are capable of communicating with each other over a network.

[0212] However, a configuration having the functions realized by the imaging device 111, the display processing unit 112, and the cloud storage 113 may be configured within hardware consisting of a single housing, such as a personal computer, a smartphone, or a new imaging device that incorporates the imaging device 111.

[0213] <<3. Modified Examples>> In the above, an example has been described in which, for each subject, zones are set in units of lines in an RGB image based on a subject map, weights for thinning correction and frame interpolation correction are set for each zone, and thinning correction and frame interpolation correction are performed in combination according to the set weights, thereby appropriately suppressing rolling shutter distortion.

[0214] However, by setting weights for thinning correction and frame interpolation correction for each subject based on the subject map, thinning correction and frame interpolation correction can be performed in combination according to the weights, thereby appropriately suppressing rolling shutter distortion.

[0215] That is, for example, as shown by the images Ma', F1t', and F2t' surrounded by dotted rectangular frames in the image Pe of Figure 15, the weights for the thinning correction and the weights for the frame interpolation correction may be set based on the speed, distance, and distance reliability of each of the subjects.

[0216] This makes it possible to set a well-balanced weight for thinning correction and a weight for frame interpolation correction for an image Ma' corresponding to a moving subject Mo at a short distance, for example.

[0217] Furthermore, for images F1t' and F2t' corresponding to flag-like subjects F1o and F2o fluttering at high speed, the occurrence of Jello can be suppressed by increasing the weight of thinning correction and decreasing the weight of frame interpolation correction.

[0218] By this processing, the weights of the thinning-type correction and the inter-frame correction are appropriately set according to the speed, distance, and distance reliability of the subject, making it possible to appropriately combine thinning-type correction and frame interpolation correction for each subject, thereby making it possible to appropriately suppress rolling shutter distortion.

[0219] <Imaging Process (Including Thinning Correction Process) in Modification> Next, the imaging process (including thinning correction process) in the modification will be described with reference to the flowchart in FIG.

[0220] The image processing in the modified example is the same as the processing explained with reference to the flowchart in FIG. 10, and therefore will not be explained further.

[0221] 16 are similar to the processes from steps S31 to S35 and steps S42 to S45 in the flowchart of FIG. 11, and therefore will not be described here.

[0222] That is, when the RGB image, motion data, distance map, distance confidence map, and velocity map are generated by the processes from steps S111 to S115, the process proceeds to step S116.

[0223] Regarding the area of ​​the subject in the RGB image based on the subject map, for example, an RGB image such as that shown in image P11' in Figure 17 is captured, and the five subjects recognized as the subjects in image P11 are subject Sb1 corresponding to the body of the airplane, subject Sb2 corresponding to the propeller, subject Sb3 corresponding to the window, subject Sb4 corresponding to the wheels, and subject Sb5 corresponding to the background.

[0224] Furthermore, when the RGB image is image P11', the velocity map, distance map, and distance confidence map are expressed as, for example, velocity map Mv', distance map Md', and distance confidence map Mc' in FIG.

[0225] 17, in the velocity map Mv', the faster the velocity of the subject, the whiter the corresponding region is represented as a distribution, and the slower the velocity of the subject, the darker the corresponding region is represented as a distribution. For example, in the velocity map Mv' in image P11', which is an RGB image, velocity distributions Sv1 to Sv5 corresponding to subjects Sb1 to Sb5 are set.

[0226] In the distance map Md', the closer the distance to the subject, the whiter the corresponding area is, and the farther the distance to the subject, the blacker the corresponding area is. For example, in the distance map Md' in image P11', which is an RGB image, distance distributions Sd1 to Sd5 corresponding to subjects Sb1 to Sb5 are set.

[0227] Furthermore, in the distance confidence map Mc', the higher the distance confidence, the whiter the corresponding area is represented, and the lower the distance confidence, the blacker the corresponding area is represented. For example, in the distance confidence map Mc' in image P11', which is an RGB image, distance confidence distributions Sc1 to Sc5 corresponding to subjects Sb1 to Sb5 are set.

[0228] In step S116, the weight setting unit 144 sets one of the unprocessed subjects as a processing target subject.

[0229] In step S117, the correction amount determination unit 145 calculates the amount of rolling shutter distortion (angle) based on the subject to be processed. Note that the specific method of calculating the amount of rolling shutter distortion (angle) is the same as when the zone is used as a unit, and therefore a description thereof will be omitted.

[0230] In step S118, the weight setting unit 144 executes weighting processing to set weights for thinning correction and frame interpolation correction of the processing target object.

[0231] The weighting process will be described in detail later with reference to the flowchart of FIG.

[0232] In step S119, the correction amount determination unit 145 determines the amount of thinning correction and the amount of frame interpolation correction by multiplying the amount of rolling shutter distortion of the processing target object by the weight set for the processing target object by the weight setting unit 144.

[0233] In step S120, the weight setting unit 144 determines whether or not there are any unprocessed subjects, and if there are any unprocessed subjects, the process returns to step S 116. That is, weights for thinning-out correction and frame interpolation correction are set for all subjects, and the processes from step S116 to S120 are repeated until the amounts of thinning-out correction and frame interpolation correction are determined.

[0234] Then, in step S120, weights for thinning correction and frame interpolation correction are set for all subjects, the amount of thinning correction and the amount of frame interpolation correction are determined, and if it is determined that there are no unprocessed subjects, processing proceeds to step S121.

[0235] Furthermore, by the processing of steps S121 and S122, the RGB sensor 123 is controlled so that the set number of lines are thinned based on the thinning correction amount set for each subject, thinned image capture is performed, the pixels of the thinned lines are generated by interpolation, and an RGB image with corrected thinning is captured.

[0236] Furthermore, in step S123, the thinning correction unit 127 stores the thinning-corrected RGB image, movement data of the imaging device 111, the distance map, the distance reliability map, the velocity map, the subject map, imaging device information (exposure time during imaging and delay time per line), and the frame interpolation correction amount for each subject (including information on the weight of the frame interpolation correction) in the memory unit 191 of the cloud storage 113.

[0237] If necessary, the thinning correction unit 127 may directly output the thinning-corrected RGB image, movement data of the imaging device 111, a distance map, a distance reliability map, a velocity map, a subject map, imaging device information (exposure time during imaging and delay time per line), and frame interpolation correction amount for each subject (including information on the weight of frame interpolation correction) to the frame interpolation correction unit 161 of the display processing unit 112.

[0238] Through the above processing, weights for thinning correction and frame interpolation correction are set for each subject based on the RGB image, distance map, and motion data, the amount of thinning correction and the amount of frame interpolation correction according to the set weights are determined for each subject, and an RGB image that has been thinned and corrected according to the amount of thinning correction for each subject is captured.The thinning-corrected RGB image, motion data of the image capture device 111, distance map, distance reliability map, velocity map, subject map, image capture device information (exposure time during image capture and delay time per line), and frame interpolation correction amount (including information on the weight of frame interpolation correction) are then stored in the storage unit 191 of the cloud storage 113.

[0239] <Weighting Processing in Modified Example> Next, the weighting processing in the modified example will be described with reference to the flowchart in Fig. 18. Basically, the weighting processing in the modified example is performed in units of subjects, whereas the weighting processing described with reference to Fig. 13 was performed in units of zones.

[0240] That is, in step S141, the weight setting unit 144 initializes the thinning weight, which is the weight for thinning correction of the processing target subject, to zero.

[0241] In step S142, the weight setting unit 144 sets the thinning speed weight of the subject to be processed based on the speed map, so that the higher the speed, the closer to 1 the weight, and the lower the speed, the closer to 0 the weight.

[0242] For example, if the subject is one of subjects Sb1 to Sb5 in the velocity map Mv' in Figure 17, the weight setting unit 144 sets a thinning speed weight based on the speed represented by color in the velocity distribution Sv1 to Sv5 of the corresponding velocity map Mv'.

[0243] In step S143 , the weight setting unit 144 determines whether the movement of the processing target subject is in accordance with the movement of the image capturing device 111 , based on the movement data of the image capturing device 111 .

[0244] In step S143, if the movement of the subject to be processed is in accordance with the movement of the image capturing device 111, the process proceeds to step S144.

[0245] In step S144, the weight setting unit 144 sets the thinning speed weight of the processing target subject to a value close to one.

[0246] In step S144, if the movement of the processing target subject does not follow the movement of the image capturing device 111, the processing of step S144 is skipped. That is, in this case, a thinning speed weight is set according to the apparent speed.

[0247] In step S145, the weight setting unit 144 sets the thinning distance weight for the processing target subject to a value closer to 0 the farther the distance, and to 1 the closer the distance, based on the distance map.

[0248] For example, if the subject is one of subjects Sb1 to Sb5 in the distance map Md' of Figure 17, the weight setting unit 144 sets a thinning distance weight based on the distance represented by color in the distance distribution Sd1 to Sd5 of the corresponding distance map Md'.

[0249] In step S146, the weight setting unit 144 sets the thinning distance reliability weight for the processing target subject to a value closer to 0 as the reliability decreases and to a value closer to 1 as the reliability increases, based on the distance reliability map.

[0250] For example, in the distance confidence map Mc' of Figure 17, if the subject is one of subjects Sb1 to Sb5, the weight setting unit 144 sets the thinning distance confidence weight based on the distance confidence represented by color in the distance confidence distributions Sc1 to Sc5 of the corresponding distance confidence map Mc'.

[0251] In step S147, the weight setting unit 144 calculates a thinning weight for the processing target subject by multiplying the thinning speed weight, the thinning distance weight, and the thinning reliability weight.

[0252] In step S148, the weight setting unit 144 subtracts the thinning weight from 1 to calculate a frame interpolation weight, which is a weight for frame interpolation correction of the processing target subject.

[0253] Through the above processing, a thinning weight that is a weight for thinning correction and a frame interpolation weight that is a weight for frame interpolation correction are set for each object based on the velocity map, distance map, and distance reliability map.

[0254] For example, based on the velocity map Mv', distance map Md', and distance confidence map Mc' in Figure 17, if the thinning weights of all subjects are mapped in correspondence with the RGB image, they will become, for example, as shown in the thinning weight map Mw in Figure 17.

[0255] In the thinning weight map Mw of FIG. 17, the closer the magnitude of the thinning weights Sw1 to Sw5 for each of the subjects Sb1 to Sb5 is to 0, the more black the corresponding area is, and the closer it is to 1, the more white the corresponding area is.

[0256] That is, the subject Sb1 corresponding to the main body of the airplane is moving but at a low speed and is also close, so the thinning weight Sw1 is set to be small for the corresponding area and it is colored dark gray.

[0257] For the object Sb2 corresponding to the propeller, the distance reliability Sv2 is extremely low, so it is colored black, the thinning weight Sw2 is minimized, and frame interpolation correction is maximized. Ideally, the object Sb2 corresponding to the propeller would be white and thinning correction would be maximized, but a frame with extremely low distance reliability is shown as an example to demonstrate that jello does not occur even in frames with extremely low distance reliability. Without the processing of step S64 or step S144, the background weight Sw2 would also be minimized and frame interpolation correction would be maximized, resulting in jello, where the background fluctuates due to the movement of the propeller. In the present disclosure, the processing of steps S64 and S144 makes it possible to always suppress jello even in implementations where distance reliability fluctuates.

[0258] The subject Sb3 corresponding to the window and the subject Sb4 corresponding to the wheel are moving at a low speed and have low distance reliability, so the thinning weights Sw3 and Sw4 are set small and are displayed in light gray.

[0259] The subject Sb5 corresponding to the background is relatively far away and moves at a slow speed, but has a high distance reliability, so the thinning weight Sw5 is set large and the subject is colored white.

[0260] Through the above processing, a thinning weight, which is a weight for thinning correction, and a frame interpolation weight, which is a weight for frame interpolation correction, are appropriately set for each subject.

[0261] <Frame interpolation correction processing in a modified example> Next, the frame interpolation correction processing in a modified example will be described with reference to the flowchart of Fig. 19. Note that the frame interpolation correction processing in the modified example in the flowchart of Fig. 19 is processing in units of subjects, whereas the frame interpolation correction processing in the flowchart of Fig. 14 is processing in units of zones.

[0262] That is, in step S161, the frame interpolation correction unit 161 of the display processing unit 112 reads and acquires the thinning-corrected RGB image stored in the memory unit 191 of the cloud storage 113, the motion data of the imaging device 111, the distance map, the distance reliability map, the speed map, the subject map, imaging device information (exposure time during imaging and delay time per line), and the frame interpolation correction amount for each subject (including information on the weight of the frame interpolation correction).

[0263] In step S162, the motion path estimation unit 181 maps each pixel of the RGB image into a 3D space.

[0264] In step S163, the motion path estimation unit 181 smooths the motion of the image capture device 111 based on the motion data, which is time-series position and orientation data of the image capture device 111, to correct the motion data.

[0265] In step S164, the motion path estimation unit 181 matches the motion data for each frame in time series based on the time stamps, and estimates the motion path of the image capture device 111.

[0266] In step S165, the target motion path estimating unit 182 sets the unprocessed subject as the processing target subject.

[0267] In step S166, the target motion path estimation unit 182 estimates the movement of the processing target subject as a target motion path based on the subject map, the distance map, and information on each pixel of the RGB image mapped in 3D space.

[0268] In step S167, the motion correction unit 183 estimates the amount of correction per unit time for one line from the estimated motion path of the image capture device 111 to the target motion path, which is the motion of the subject to be processed.

[0269] In step S168, the motion correction unit 183 resamples the RGB image so as to correct it by the frame interpolation correction amount based on the correction amount per unit time of one line, and generates a stabilized frame RGB image of the subject to be processed.

[0270] In step S169, the motion correction unit 183 determines whether or not there is an unprocessed subject.

[0271] In step S169, if there is an unprocessed subject, the process returns to step S165.

[0272] That is, the processes from steps S165 to S169 are repeated until stabilized frame RGB images are generated for all subjects.

[0273] Then, in step S170, the motion correction unit 183 generates a complete stabilized frame RGB image by combining the stabilized frame RGB images of all the subjects.

[0274] In step S171, when generating a stabilized frame RGB image, the motion correction unit 183 generates a stabilized object map, which is a subject map in the stabilized frame RGB image, a stabilization distance map, a stabilization distance reliability map, and a stabilization speed map corresponding to the stabilized frame RGB image, based on the positional relationship of each pixel in the RGB image generated by resampling, and stores these in the memory unit 191 of the cloud storage 113 together with stabilized motion data, which is the motion data of the imaging device 111 corrected when generating the stabilized frame RGB image.

[0275] By performing the above processing, frame interpolation correction is performed on the thinning-corrected RGB image, and it becomes possible to perform rolling shutter correction that combines thinning correction and frame interpolation correction for each subject based on the thinning weight and frame interpolation weight that are set based on the speed, distance, and distance reliability.

[0276] This makes it possible to combine thinning correction and frame interpolation correction in an appropriate balance based on weights set for each subject on the basis of speed, distance, and distance reliability, making it possible to appropriately suppress distortions that cannot be corrected by thinning correction alone and distortions that cannot be corrected by frame interpolation correction alone.

[0277] Furthermore, in the modified example of the preferred embodiment, it is possible to appropriately combine thinning correction and frame interpolation correction for each subject, thereby enabling more detailed and appropriate suppression of rolling shutter distortion.

[0278] As a result, images captured using the rolling shutter method can be applied to high-definition visual effects technology such as VFX (Visual Effects). While we mentioned VFX (Visual Effects) above, this can also be used for convulsion monitoring, which will be described later as an application example, or photogrammetry, or any other video that requires accurate, measurable precision without the need to disguise fast-moving subjects, particularly subjects that move in uniform rotational motion, pendulum motion, or pendulum motion around a fulcrum, which conventional technology struggles to capture, with measurable precision without the need to disguise them with blur.

[0279] <<4. Application Example>> In the above, image processing is realized to correct rolling shutter distortion and generate and display a stabilized frame RGB image, and as by-products, the stabilized frame RGB image, stabilized motion data, stabilized subject map, stabilization distance map, stabilization distance reliability map, and stabilization speed map are sequentially stored as metadata in the storage unit 191 of the cloud storage 113.

[0280] The metadata thus stored may be used to realize other processes.

[0281] For example, metadata can be used to identify and individually identify people among the subjects in the stabilized frame RGB image, and estimate their bone structure and height, allowing for monitoring of convulsions and determining whether they are clonic convulsions due to cardiac disorders, shivering due to cold that occurs in healthy people, or essential tremor that occurs in healthy elderly people.

[0282] FIG. 20 shows an example configuration of an image processing system that uses metadata generated by image processing to correct rolling shutter distortion to monitor clonic convulsions caused by cardiac disorders.

[0283] In the image processing system 101' of FIG. 20, components having the same functions as those in the image processing system 101 of FIG. 5 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0284] That is, the image processing system 101' in FIG. 20 differs from the image processing system 101 in FIG. 5 in that a cardiogenic malfunction monitoring processor 211 is newly provided.

[0285] The cardiogenic disorder monitoring processing unit 211 monitors the presence or absence of clonic convulsions caused by cardiogenic disorder based on the metadata generated when the above-mentioned image processing is performed to correct rolling shutter distortion and display an image.

[0286] More specifically, the cardiogenic malfunction monitoring processing unit 211 first acquires the stabilized frame RGB image, stabilized object map, stabilized velocity map, and stabilized distance map stored in the memory unit 191 of the cloud storage 113.

[0287] Next, the cardiogenic disorder monitoring processing unit 211 extracts and individually identifies people from the subjects in the stabilized subject map, estimates the skeleton and height of each person based on the stabilized distance map, and calculates the threshold value of the convulsion speed (= predetermined coefficient / height) at which it is considered that clonic convulsions are occurring based on the height.

[0288] The cardiogenic disorder monitoring processing unit 211 then estimates the limb skeleton for each person based on the subject map and velocity map, calculates the convulsion velocity of the limb skeleton, and compares it with a convulsion velocity threshold to determine whether clonic convulsions caused by cardiogenic disorder are occurring.

[0289] Until now, the occurrence of clonic convulsions could be determined by simply measuring the speed of the limb skeleton convulsions, but this required an evaluation based on the person's height, making it impossible to determine whether the tremors were those caused by cold, which can occur in healthy people, or essential tremors, which can occur in healthy elderly people, or clonic convulsions.

[0290] This is because, even if the person's height had been measured in advance, there was no way to accurately measure the speed of the convulsion from an image in which rolling shutter distortion had not been removed, or in which incorrect distortion correction processing had been performed without estimating that the convulsion was a clonic convulsion, which is a pendulum movement with the elbows and knees as fulcrums, as disclosed in this disclosure.

[0291] Therefore, Patent No. 6372388 proposes a technology to distinguish between tremors (essential tremor) and convulsions (clonic convulsions) by detecting both the posture and direction of the fall. However, there have been cases of false positives of convulsions that do not involve a fall. In particular, it has been nearly impossible to detect convulsions when the person is lying down in bed, not while driving.

[0292] Therefore, Patent No. 6538055 proposes a technology for distinguishing between essential tremor and clonic convulsions even while in bed by also detecting EMG (surface myoelectric potential) signal data. However, Patent No. 6538055 requires the wearer to wear a contact-type device in order to make the diagnosis while in bed.

[0293] In contrast, in the image processing system 101' of Fig. 20 of the present disclosure, it is possible to set a threshold value for clonic convulsions according to the height of each person based on metadata generated when the above-described image processing is performed to correct rolling shutter distortion and display an image. This makes it possible to appropriately monitor the presence or absence of clonic convulsions even when the person is lying down in bed and without wearing a contact-type device.

[0294] <Clonic Convulsion Monitoring Process> Next, the clonic convulsion monitoring process will be described with reference to the flowchart of FIG.

[0295] This processing is premised on the fact that image processing for correcting the above-mentioned rolling shutter distortion has been performed, and that processing that combines the thinning correction and frame interpolation correction for each subject, as described in the modified example, has been performed.

[0296] In step S201, the cardiogenic malfunction monitoring processing unit 211 reads and acquires the stabilized frame RGB image, stabilized motion data, stabilized distance map, stabilized velocity map, and stabilized object map stored in the memory unit 191 of the cloud storage 113.

[0297] In step S202, the cardiogenic malfunction monitoring processing unit 211 maps each pixel of the RGB image into a normalized 3D space.

[0298] In step S203, the cardiogenic disorder monitoring processing unit 211 individually identifies and extracts people from the subjects based on the stabilized subject map, and estimates and stores the bone structure and height of each person based on the stabilized distance map.

[0299] In step S204, the cardiogenic malfunction monitoring processing unit 211 sets the unprocessed person as the processing target person.

[0300] In step S205, the cardiac disorder monitoring processing unit 211 calculates, from the height of the person being processed, a threshold range of the convulsion speed of the person being processed (= predetermined coefficient / height ± error) at which the person being processed can be considered to be experiencing clonic convulsions.

[0301] For example, in the prior art, if rolling shutter distortion is not removed, when a person is 160 cm tall, the predetermined coefficient / height ratio is 5.00 Hz ± 30.0 Hz, or 0.166 to 0.233 times / second. This predetermined coefficient / height ratio is equivalent to a height ranging from 192 cm to 137 cm. Therefore, in the prior art, even if a threshold range calculated by processing dependent on the height of the person being processed, which corresponds to the processing of step S205 of the present disclosure, is used, it is not possible to appropriately determine whether the person being processed is experiencing clonic convulsions.

[0302] Furthermore, while the prior art has been described using a case in which rolling shutter distortion has not been removed, in a captured image in which erroneous distortion correction processing has been performed without estimating clonic convulsions, which are pendulum movements with the elbows and knees as fulcrums, as in the present disclosure, correction is applied to both the positive and negative sides of the frequency of the pendulum movements compared to a captured image in which rolling shutter distortion has not been removed. Therefore, even if a threshold range calculated by processing dependent on the height of the person being processed, which corresponds to the processing in step S205 of the present disclosure, is used, it is not possible to appropriately determine whether the person being processed is experiencing clonic convulsions.

[0303] In contrast, when using the present disclosure to estimate clonic convulsions as pendulum movements with the elbows and knees as fulcrums, the ratio of the predetermined coefficient to height is 5.00 Hz ± 300 Hz, or 0.196 to 0.203 beats per second. This predetermined coefficient / height ratio is equivalent to a height ranging from 163 cm to 157 cm, which is within the error range of a separately estimated height. Therefore, the threshold range calculated by processing dependent on the height of the person to be processed, such as step S205 of the present disclosure, makes it possible to appropriately determine whether the person to be processed is experiencing clonic convulsions.

[0304] In step S206, the cardiac illness monitoring processing unit 211 estimates the limb skeleton of the processing target person based on the stabilized subject map and the stabilized frame RGB image.

[0305] In step S207, the cardiogenic disorder monitoring processing unit 211 extracts the velocity at the position of the limb skeleton of the processing target person as the seizure velocity of the processing target person based on the stabilization velocity map. As shown with reference to Figures 7 to 9 of the present disclosure, by correcting rolling shutter distortion based on the assumption that the seizure is a clonic convulsion, which is a pendulum movement with the elbows and knees as fulcrums, it is possible to stably and accurately extract the seizure velocity of the processing target person. Therefore, the cardiogenic disorder monitoring processing unit 211 determines whether this stably and accurately extracted seizure velocity is within a threshold range of the seizure velocity that can be considered to be a clonic convulsion of the processing target person.

[0306] If it is determined in step S207 that the convulsion speed of the limb skeleton of the processing target person is within the threshold range of the convulsion speed that can be considered as clonic convulsion of the processing target person, the processing proceeds to step S208.

[0307] In step S208, the cardiogenic malfunction monitoring processing unit 211 registers that the processing target person is suspected of having clonic convulsions.

[0308] If it is determined in step S207 that the convulsion speed of the limb skeleton of the processing target person is not within the threshold range of the convulsion speed of the processing target person, the process proceeds to step S209.

[0309] In step S209, the cardiogenic malfunction monitoring processing unit 211 registers that the processing target person is not experiencing clonic convulsions.

[0310] In step S210, the cardiogenic malfunction monitoring processing unit 211 determines whether or not there is an unprocessed person, and if there is an unprocessed person, the process returns to step S204. That is, the processes from step S204 to S210 are repeated until it is determined whether or not clonic convulsions have occurred for all the people.

[0311] Then, in step S210, if there are no unprocessed persons, that is, if it is determined whether or not clonic convulsions have occurred for all the persons, the processing proceeds to step S211.

[0312] In step S211, the cardiogenic malfunction monitoring processing unit 211 determines whether or not a person suspected of having a clonic convulsion has been registered.

[0313] In step S211, if there is a person suspected of having a clonic convulsion, the process proceeds to step S212.

[0314] As mentioned above, there may be a case where a person is suspected of having clonic convulsions, but in this case, a comprehensive judgment may be made in addition to the various symptoms that may occur with clonic convulsions.

[0315] Symptoms that may accompany the aforementioned clonic convulsions include visual information such as clutching the chest with the hands, a pained facial expression, eyes rolling up, eyes wide open, mouth wide open, breathing with the shoulders held high, leaning back, crouching, hugging the knees, jugular venous distension, jugular venous pulsation, orthopnea, agonal breathing, and cold sweat. Symptoms that may accompany the aforementioned clonic convulsions also include audio information such as moans and calls for help.

[0316] As mentioned above, when a person is suspected of having clonic convulsions, this can be judged comprehensively in conjunction with the person's history over time, including their medical history, changes in wake-up time, reaction to lights, reaction to flashing lights, reaction to being called, reaction to warning sounds, changes in facial expression over time, changes in posture, changes in behavior, changes in voice quality, changes in complexion, and changes in facial expression.

[0317] In step S212, the cardiogenic disorder monitoring processing unit 211 identifies and presents the person suspected of having clonic convulsions by outputting and displaying information about the person registered as a person suspected of having clonic convulsions, such as a facial image of the person extracted from the stabilized frame RGB image, to the display unit 162.

[0318] As mentioned above, the identified person may be asked "Are you OK?" using a voice output device, or the identified person may be notified to an emergency system using a communication device, or the identified person may contact a pre-set contact such as their family doctor or family member, or the identified image may be sent, or a single judgment result may be sent to a subsequent judgment device, which may then make a judgment using multiple judgment results.

[0319] Furthermore, if it is determined in step S211 that there is no person suspected of having a clonic convulsion, the process of step S212 is skipped.

[0320] In step S213, the cardiogenic malfunction monitoring processing unit 211 determines whether an instruction to end the processing has been issued, and if it is determined that an instruction to end the processing has not been issued, the processing returns to step S201, and the subsequent processing is repeated. At this time, the registered information of persons suspected of having clonic convulsions and persons not considered to have clonic convulsions is initialized, and then the processing returns to step S201.

[0321] Although the registration information is initialized as described above, it is not necessary to initialize the registration information. Also, although the processing returns to step S201 as described above, it may return to any of steps S202 to S207.

[0322] If it is determined in step S213 that an instruction to end the process has been issued, the process ends.

[0323] Through the above processing, it is possible to set a highly accurate threshold value for the convulsion speed required to determine clonic convulsions based on height for each person, based on the metadata generated when the above-mentioned image processing is performed to correct rolling shutter distortion and display the image.

[0324] As a result, by measuring the convulsion speed of the limb skeleton for each person and comparing it with a threshold for clonic convulsions set based on each person's height, it is possible to appropriately monitor the presence or absence of clonic convulsions.

[0325] <<5. Example of Execution by Software>> The above-described series of processes can be executed by hardware, but can also be executed by software. When the series of processes is executed by software, the program constituting the software is installed from a recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0326] 22 shows an example of the configuration of a general-purpose computer. This computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0327] The input / output interface 1005 is connected to an input unit 1006 including input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 that outputs a processing operation screen and images of processing results to a display device, a storage unit 1008 including a hard disk drive or the like that stores programs and various data, and a communication unit 1009 including a LAN (Local Area Network) adapter or the like that executes communication processing via a network typified by the Internet. Also connected is a drive 1010 that reads and writes data from / to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0328] The CPU 1001 executes various processes in accordance with a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, installed in a storage unit 1008, and loaded from the storage unit 1008 into a RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as appropriate.

[0329] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0330] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0331] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting a removable storage medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0332] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0333] In addition, the CPU 1001 in Figure 22 realizes the functions of the thinning correction unit 127 and frame interpolation correction unit 161 in Figures 5 and 20, and the cardiogenic malfunction monitoring processing unit 211, and the memory unit 1008 realizes the functions of the memory unit 191 of the cloud storage 113.

[0334] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0335] Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0336] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0337] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0338] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0339] The present disclosure may also be configured as follows: <1> An image processing method including correcting rolling shutter distortion in an image captured by a rolling shutter method by combining thinning correction that corrects using a thinning method and frame interpolation correction that corrects using a frame interpolation method. <2> The image processing method according to <1>, further including setting a thinning correction weight that indicates the degree of the thinning correction and a frame interpolation correction weight that indicates the degree of the frame interpolation correction. <3> The image processing method according to <2>, further including identifying a subject in the image, wherein the thinning correction weight and the frame interpolation correction weight are set for each subject. <4> The image processing method according to <3>, further comprising: measuring a distance to the subject; mapping the distance to generate a distance map; mapping the reliability of the distance as a distance reliability based on a magnitude of a time series change of the distance in the distance map over a plurality of frames to generate a distance reliability map; and mapping a velocity of the subject based on the images of successive frames to generate a velocity map; wherein the thinning correction weight and the frame interpolation correction weight are set for each subject based on the distance map, the distance reliability map, and the velocity map. <5> The image processing method according to <4>, wherein a thinning distance weight that is a weight for the thinning correction weight is set for each of the subjects based on the distance map; a thinning distance reliability weight that is a weight for the thinning correction weight is set for each of the subjects based on the distance reliability map; a thinning speed weight that is a weight for the thinning correction weight is set for each of the subjects based on the velocity map; the thinning correction weight is set as a value obtained by multiplying the thinning distance weight, the thinning distance reliability weight, and the thinning speed weight for each of the subjects; and the frame interpolation correction weight is set as a value obtained by subtracting the thinning correction weight from 1.<6> The image processing method according to <5>, wherein the thinning distance weight is set to a value closer to 1 for each subject as the distance is shorter based on the distance map, and a value closer to 0 for each subject as the distance is longer. <7> The image processing method according to <5>, wherein the thinning distance confidence weight is set to a value closer to 1 for each subject as the change in distance is smaller based on the distance map of consecutive frames, and a value closer to 0 for each subject as the change in distance is larger, and a value closer to 0 for each subject as the change in distance is longer. <8> The image processing method according to <5>, wherein the thinning speed weight is set to a value closer to 1 for each subject as the speed is higher, and a value closer to 0 for each subject as the speed is lower based on the speed map. <9> The image processing method according to <4>, further including: detecting motion of an image sensor that captures the subject and acquiring the motion data; calculating a rolling shutter distortion amount based on the image, the distance map, the velocity map, and the motion data; calculating a thinning correction amount by multiplying the rolling shutter distortion amount by the thinning correction weight; and calculating a frame interpolation correction amount by multiplying the rolling shutter distortion amount by the frame interpolation correction weight. <10> The image processing method according to <9>, further including: controlling imaging of the image sensor so that, when capturing the image, pixel signals are thinned and read out by the number of lines corresponding to the thinning correction amount. <11> The image processing method according to <10>, further including multiplying the frame interpolation correction by the frame interpolation correction amount to the image captured by the image sensor, whose imaging is controlled so that pixel signals are thinned and read out by the number of lines corresponding to the thinning correction amount. <12> The image processing method according to <2>, further comprising: identifying a subject in the image; and setting a zone in the image in which the subject exists, using a line of an image sensor that captures an image of the subject as a unit; and setting the thinning-out correction weight and the frame interpolation correction weight on a zone-by-zone basis.<13> The image processing method according to <12>, wherein, when multiple subjects exist on the same line and therefore the zone includes multiple subjects, one of the subjects is set to be a reference subject based on a predetermined condition, and the thinning-out correction weight and the frame interpolation correction weight are set based on the reference subject, with the zone as a unit. <14> An image processing device comprising: a thinning-out correction unit that corrects rolling shutter distortion of an image captured by a rolling shutter method using a thinning-out method; and a frame interpolation correction unit that corrects the rolling shutter distortion using a frame interpolation method, and corrects the rolling shutter distortion by combining the correction by the thinning-out correction unit and the correction by the frame interpolation correction unit. <15> A program that causes a computer to function as a thinning-out correction unit that corrects rolling shutter distortion of an image captured by a rolling shutter method using a thinning-out method, and a frame interpolation correction unit that corrects the rolling shutter distortion by frame interpolation, and corrects the rolling shutter distortion by combining the correction by the thinning-out correction unit and the correction by the frame interpolation correction unit.

[0340] DESCRIPTION OF SYMBOLS 101, 101' Image processing system, 111 Imaging device, 112 Display processing unit, 113 Cloud storage, 121 Distance sensor, 122 Distance map generation unit, 123 RGB sensor, 124 RGB image signal processing unit, 125 6-axis sensor, 126 Motion detection unit, 127 Thinning correction unit, 141 Semantic segmentation processing unit, 142 Distance confidence map generation unit, 143 Velocity map generation unit, 144 Weighting processing unit, 145 Correction amount determination unit, 146 Thinning setting unit, 161 Frame interpolation correction unit, 181 Motion path estimation unit, 182 Target motion path estimation unit, 183 Motion correction unit, 191 Storage unit

Claims

1. An image processing method that includes correcting rolling shutter distortion in an image captured using a rolling shutter method by combining thinning correction, which corrects the distortion using a thinning method, and frame interpolation correction, which corrects the distortion using a frame interpolation method.

2. The image processing method according to claim 1, further comprising setting a thinning correction weight indicating a degree of said thinning correction, and a frame interpolation correction weight indicating a degree of said frame interpolation correction.

3. The image processing method according to claim 2, further comprising identifying an object in the image, wherein the thinning-out correction weight and the frame interpolation correction weight are set on a per-object basis.

4. The image processing method according to claim 3, further comprising: measuring a distance to the subject; mapping the distance to generate a distance map; mapping the confidence of the distance as a distance confidence based on a magnitude of time series change of the distance in the distance map over a plurality of frames to generate a distance confidence map; and mapping a velocity of the subject based on the images of successive frames to generate a velocity map; wherein the thinning-out correction weight and the frame interpolation correction weight are set for each subject based on the distance map, the distance confidence map, and the velocity map.

5. The image processing method of claim 4, wherein a thinning distance weight which is a weight for the thinning correction weight is set for each of the subjects based on the distance map; a thinning distance confidence weight which is a weight for the thinning correction weight is set for each of the subjects based on the distance confidence map; a thinning speed weight which is a weight for the thinning correction weight is set for each of the subjects based on the velocity map; the thinning correction weight is set as a value obtained by multiplying the thinning distance weight, the thinning distance confidence weight, and the thinning speed weight for each of the subjects; and the frame interpolation correction weight is set as a value obtained by subtracting the thinning correction weight from 1.

6. The image processing method according to claim 5, wherein the thinning distance weight is set to a value closer to 1 the closer the distance is, and to a value closer to 0 the farther the distance is, based on the distance map, with the subject as a unit.

7. The image processing method according to claim 5, wherein the thinning distance confidence weight is set so that, based on the distance map of successive frames, with the subject as a unit, the smaller the change in distance, the higher the distance confidence is considered to be, and a value closer to 1 is set, and the larger the change in distance, the lower the distance confidence is considered to be, and a value closer to 0 is set.

8. The image processing method according to claim 5, wherein the thinning speed weight is set to a value closer to 1 the higher the speed is, and a value closer to 0 the lower the speed is, based on the speed map and for each subject.

9. The image processing method according to claim 4, further comprising: detecting movement of an image sensor that images the subject and acquiring the movement data; calculating a rolling shutter distortion amount based on the image, the distance map, the velocity map, and the movement data; calculating a thinning-out correction amount by multiplying the rolling shutter distortion amount by the thinning-out correction weight; and calculating a frame interpolation correction amount by multiplying the rolling shutter distortion amount by the frame interpolation correction weight.

10. The image processing method according to claim 9, further comprising controlling imaging of the image sensor so that, when capturing the image, pixel signals are thinned out and read out by the number of lines corresponding to the amount of thinning correction.

11. The image processing method according to claim 10, further comprising: applying the frame interpolation correction amount to the image captured by the imaging element, the imaging being controlled so that pixel signals are thinned out and read out by the number of lines corresponding to the thinning correction amount.

12. The image processing method according to claim 2, further comprising: identifying a subject in the image; and setting a zone in the image in which the subject exists, the zone being determined by a line of an image sensor that captures the subject; and the thinning-out correction weight and the frame interpolation correction weight are set by using the zone as a unit.

13. The image processing method according to claim 12, wherein, when a plurality of subjects are present on the same line, resulting in a plurality of subjects being included in the zone, one of the subjects is set as a reference subject based on predetermined conditions, and the thinning-out correction weight and the frame interpolation correction weight are set based on the reference subject with the zone as a unit.

14. An image processing device comprising: a thinning-out correction unit that corrects rolling shutter distortion of an image captured by a rolling shutter method using a thinning-out method; and a frame interpolation correction unit that corrects the rolling shutter distortion using a frame interpolation method, and corrects the rolling shutter distortion by combining the correction by the thinning-out correction unit and the correction by the frame interpolation correction unit.

15. A program for causing a computer to function as a thinning-out correction unit that corrects rolling shutter distortion of an image captured by a rolling shutter method using a thinning-out method, and a frame interpolation correction unit that corrects the rolling shutter distortion using a frame interpolation method, and corrects the rolling shutter distortion by combining the correction by the thinning-out correction unit and the correction by the frame interpolation correction unit.

Images