Image correction method and program

Abstract

This image correction device compares a reference image and an input image, corrects the input image, and, if the amount of change between the reference image and each of a plurality of input images sequentially input is a value within a predetermined range within a predetermined period, updates the reference image on the basis of the input image input within the predetermined period.

Description

Image correction method and program

[0001] The disclosed technology relates to an image correction method and an image correction program.

[0002] In recent years, skeletal information of subjects recognized from video captured by cameras has been utilized in a variety of fields, including sports, surveillance, manufacturing, and nursing care. In skeletal recognition from video, it is important to accurately detect the positions of people and joints in the image. However, the camera may vibrate due to pedestrians moving near the camera. When camera vibration occurs, the positions of people and joints in the image may shift from their positions when the camera is not vibrating, i.e., from their intended positions. This shift may reduce the accuracy of skeletal recognition.

[0003] Therefore, there are vibration correction technologies for detecting camera vibration and correcting captured images. For example, an image processing device has been proposed that appropriately reduces the effects of changes in the position and orientation of the image capture device. This image processing device corrects the captured image based on a correction amount based on the feature points when an index related to the number of feature points in the captured image satisfies a predetermined condition. Furthermore, the image processing device corrects the captured image based on a past correction amount when the index related to the number of feature points in the captured image does not satisfy the predetermined condition. A case in which the index related to the number of feature points satisfies the predetermined condition refers to a case in which the ratio of the number of feature points in the captured image to the number of feature points in a reference image is equal to or greater than a threshold.

[0004] Japanese Patent Application Laid-Open No. 2022-012397

[0005] However, with the above-mentioned conventional technology, if there is a change in the environment of the shooting range compared to when the reference image, i.e., the standard image that serves as the basis for comparing the captured image, is shot, the accuracy of vibration correction may decrease.

[0006] As one aspect, the disclosed technology aims to suppress a decrease in accuracy of vibration correction even when an environment has changed since the reference image was captured.

[0007] In one aspect, the disclosed technology compares a reference image with an input image, corrects the input image, and updates the reference image based on the input images input within a predetermined period if the amount of change between the reference image and each of the plurality of input images input sequentially falls within a predetermined range within the predetermined period.

[0008] As one aspect, even if the environment has changed since the reference image was captured, it is possible to suppress a decrease in the accuracy of vibration correction.

[0009] 1 is a diagram for explaining the necessity of vibration correction; a diagram for explaining an example of vibration correction; a diagram for explaining an example of an environmental change; a functional block diagram of an image correction device which is a diagram for explaining an example of an environmental change; a diagram for explaining a first test; a diagram for explaining a second test; a diagram showing time series characteristics of correction vectors when vibration occurs, when a moving obstacle occurs, and after an environmental change; a diagram showing an example of a change amount buffer in the first embodiment; a block diagram showing a schematic configuration of a computer which functions as an image correction device; a flowchart showing an example of an image correction process; a flowchart showing an example of a test process; a flowchart showing an example of an update process; a diagram for explaining calculation of a differential pixel rate; a diagram showing time series characteristics of a differential pixel rate when vibration occurs and after an environmental change; a diagram showing an example of a change amount buffer in the second embodiment.

[0010] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0011] <Problems with Vibration Correction> Before describing each embodiment in detail, the necessity of vibration correction and problems with vibration correction will be described.

[0012] As described above, in skeletal structure recognition from video, it is important to accurately detect the positions of people and joints in the image. However, camera vibrations may occur due to pedestrians moving near the camera, etc. When camera vibrations occur, the positions of people and joints in the image may deviate from the positions captured when there is no vibration, i.e., the positions where they should be captured. This deviation may reduce the accuracy of skeletal structure recognition. For example, as shown in FIG. 1, a deviation occurs between the position of a person detected in an image captured when there is no vibration (solid rectangle in FIG. 1) and the position of a person detected in an image captured when vibrations are present (dashed rectangle in FIG. 1). In this case, the person may be erroneously recognized as having moved to a different position, even though they are stationary.

[0013] Therefore, a vibration correction technology for detecting camera vibration and correcting captured images is required. One example of a vibration correction technology is block matching vibration correction. In block matching vibration correction, a reference image is divided into multiple blocks, as shown in FIG. 2A, and an area in the input image to be corrected that is most similar to each block in the reference image is searched for. Then, as shown in FIG. 2B, a motion vector is calculated for each block, representing the position of the searched area relative to the block position. Then, an integrated motion vector representing the statistics of all the motion vectors, such as the average value, is calculated, and a correction vector in a direction that cancels out the integrated motion vector is calculated, as shown in FIG. 2C. The input image is then corrected based on the correction vector.

[0014] One possible method for correcting an image using a correction vector is to analyze vibration components (correction vectors) for multiple frames at once and then perform correction using a two-pass method based on the analysis results. However, while using the two-pass method can achieve high correction accuracy, since vibration components for multiple frames are analyzed once and then corrected retroactively, latency occurs between the time the input image is input and the time the corrected image obtained by correcting the input image is output. Therefore, this method cannot be applied to systems that require real-time performance.

[0015] As in the invention described in Patent Document 1, when correction is performed solely by comparing a reference image with an input image, real-time performance can be ensured. However, if there is a change in the environment when the input image was captured compared to when the reference image was captured, the correction accuracy may decrease. This is because, in the area where the environment has changed, a movement vector unrelated to vibration occurs, and the correction vector used for vibration correction is calculated including this unrelated movement vector.

[0016] Changes in the environment include changes in the position or shape of objects within the shooting range, such as the mat in the examples of Figures 3 and 4. Also included are changes in the state of objects constituting the background, such as the opening and closing of curtains, the installation of an object that was not present when the reference image was captured, the removal of an object that was present when the reference image was captured, and changes in illuminance.

[0017] Furthermore, as shown in Figure 4, a moving object such as a person in the vicinity may block the subject to be detected. Such a moving object will be referred to as a "moving obstruction" below. Even if a moving obstruction that is not included in the reference image appears in the input image, a movement vector unrelated to vibration will occur in the area of ​​the moving obstruction.

[0018] In order to respond to environmental changes since the time the reference image was captured, as described above, it is conceivable to apply a process in which a user interface displays and selects areas of the reference image captured during camera calibration for which correction amounts (correction vectors) are to be calculated. In this case, the user can select and exclude areas where errors may occur in the calculation of the correction amounts. However, there may be cases in which there is no time to operate the user interface in response to environmental changes that occur during operation, and in such cases, the above process cannot be applied. For example, in a system that supports the judging of gymnastics competitions based on skeletal information recognized from vibration-corrected images, it is not practical to operate a user interface during competition operation.

[0019] As mentioned above, real-time vibration correction requires a method that performs correction solely by comparing a reference image with an input image. However, in this case, the correction accuracy may decrease due to changes in the environment since the reference image was captured.

[0020] A simple method to prevent a decrease in correction accuracy without bothering the user is to periodically update the reference image with the latest input image. However, if an input image that contains vibrations or moving obstacles is used as the reference image, this is not appropriate because it can lead to incorrect correction.

[0021] Therefore, in the following embodiments, a method for correcting vibration by updating the reference image at an appropriate timing is provided. Each embodiment will be described in detail below.

[0022] 5, an image correction device 10 according to the first embodiment functionally includes an acquisition unit 11, a reading unit 12, a calculation unit 13, a correction unit 14, a testing unit 15, and an updating unit 16. In addition, an input image buffer 21, a reference image buffer 22, and a change amount buffer 23 are provided in a predetermined storage area of ​​the image correction device 10.

[0023] The acquisition unit 11 sequentially acquires each frame of a video captured by a camera as an input image. The acquisition unit 11 passes the acquired input image to the calculation unit 13 and stores it in the input image buffer 21.

[0024] The input image buffer 21 is a buffer in which input images acquired by the acquisition unit 11 are stored. The number of buffers in the input image buffer 21 is K. That is, the input image buffer 21 stores up to the most recent K input images. K may be, for example, 30. This corresponds to one second of input images when the video is at 30 fps.

[0025] The reading unit 12 reads the reference image stored in the reference image buffer 22 and passes the read reference image to the calculation unit 13 .

[0026] Reference images are stored in the reference image buffer 22. The reference images stored in the reference image buffer 22 are images captured during camera calibration and reference images updated by the update unit 16, which will be described later.

[0027] The calculation unit 13 calculates the amount of change between the input image and the reference image. In the first embodiment, the calculation unit 13 calculates the correction vector described with reference to FIG. 2 as the amount of change. For example, the calculation unit 13 calculates a correction vector in which a correction amount x in the horizontal direction of the image and a correction amount y in the vertical direction of the image are expressed as (x, y). The calculation unit 13 passes the input image, the reference image, and the calculated correction vector to the correction unit 14, and stores the correction vector in the change amount buffer 23.

[0028] The change amount buffer 23 is a buffer that stores the correction vector calculated by the calculation unit 13. The number of buffers in the change amount buffer 23 is K, which is the same as the number of buffers in the input image buffer 21.

[0029] The correction unit 14 compares the reference image with the input image and corrects the input image. Specifically, the correction unit 14 performs vibration correction on the input image in accordance with the correction vector passed from the calculation unit 13. For example, an affine transformation or the like may be used for the vibration correction. Specifically, the correction unit 14 generates a corrected image by applying a parallel translation by affine transformation to the input image in accordance with the horizontal component x and vertical component y of the correction vector. The correction unit 14 passes the input image, the reference image, and the generated corrected image to the testing unit 15.

[0030] If a change in the environment occurs due to the appearance of a moving obstacle or the like before the reference image is updated by the update unit 16, which will be described later, improper vibration correction may be performed. If the corrected image has the following characteristics, it is highly likely that improper vibration correction has been performed. (Characteristic Example 1) There is a large difference between the reference image and the corrected image. (Characteristic Example 2) The number of differential pixels between the corrected image and the reference image increases compared to the number of differential pixels between the input image and the reference image.

[0031] Therefore, the examination unit 15 examines the corrected image to determine whether or not there is inappropriate vibration correction. In this embodiment, the examination unit 15 performs two types of examinations: a first examination based on the above-described Feature Example 1, and a second examination based on the above-described Feature Example 2.

[0032] First, the first test will be described. As shown in A of Fig. 6, for example, during a period in which a moving obstacle is present in the input image, a correction vector is calculated that is caused by the moving obstacle, not by camera vibration. Also, as shown in B of Fig. 6, during a period in which a moving obstacle is present, the degree of difference between the reference image and the corrected image increases due to the influence of the moving obstacle that is not present in the reference image. In the first test, the corrected image is tested based on whether this degree of difference is equal to or greater than a threshold.

[0033] Specifically, the testing unit 15 calculates, for example, ZNCC (Zero-means Normalized Cross-Correlation) as the degree of difference between the reference image and the corrected image. If the ZNCC is less than a threshold, the testing unit 15 determines that the test is successful and adopts the corrected image. On the other hand, if the ZNCC is equal to or greater than the threshold, the testing unit 15 determines that the test is unsuccessful, determines that the reliability of the corrected image is low, and rejects the corrected image.

[0034] The degree of difference in the first test is not limited to ZNCC, and may be, for example, a difference in a color histogram or the like.

[0035] Next, the second test will be described. In the second test, it is determined that appropriate vibration correction has not been performed if the difference between the corrected image and the reference image, which should normally be reduced by vibration correction, has increased by more than the difference between the input image and the reference image.

[0036] 7, the testing unit 15 calculates a difference in pixel value for each pixel between the input image and the reference image, and calculates a difference pixel count C1, which is the number of pixels for which the difference is equal to or greater than a threshold value (e.g., 50). The testing unit 15 also calculates a difference in pixel value for each pixel between the corrected image and the reference image, and calculates a difference pixel count C2, which is the number of pixels for which the difference is equal to or greater than a threshold value (e.g., 50). If C1 is equal to or greater than C2, the testing unit 15 determines that the test has passed, and adopts the corrected image. On the other hand, if C1 is less than C2, the testing unit 15 determines that the test has failed, determines that the reliability of the corrected image is low, and discards the corrected image.

[0037] 7, the difference image is an image in which the difference between each pixel is binarized based on a threshold value, with white representing pixels whose difference is equal to or greater than the threshold value and black representing pixels whose difference is less than the threshold value. In the example of FIG. 7, the difference image between the input image and the reference image shows difference pixels occurring mainly in the area of ​​moving obstacles (C1 = approximately 480,000), while the difference image between the corrected image and the reference image shows difference pixels occurring not only in the area of ​​moving obstacles but also in the area of ​​stationary objects (C1 = approximately 620,000). This indicates that appropriate vibration correction has not been performed.

[0038] If both the first test and the second test are passed, the test unit 15 determines that the corrected image is highly reliable and outputs the corrected image. On the other hand, if either test is failed, the test unit 15 determines that the corrected image is unreliable, discards the corrected image, and outputs the input image. The output image is input to, for example, a skeleton recognition device and used for skeleton recognition processing.

[0039] The update unit 16 updates the reference image at an appropriate timing. As described above, updating the reference image using an input image when vibration or a moving obstacle occurs is inappropriate because it can lead to erroneous correction. Therefore, in this embodiment, the reference image is updated when no vibration or moving obstacle occurs and the image is in a stable state after an environmental change. To determine this timing, attention is focused on the time-series change in the amount of change between the reference image and the input image (in this embodiment, the correction vector). As shown in FIG. 8 , when vibration or a moving obstacle occurs, the correction vector has a characteristic of fluctuating over time. On the other hand, after an environmental change, the correction vector has a characteristic of maintaining a value within a predetermined range over time.

[0040] Therefore, the update unit 16 updates the reference image based on the input images input within the specified period when the amount of change between the reference image and each of the multiple input images input sequentially is within a specified range within a specified period.

[0041] Specifically, when correction vectors are stored in all buffers (K buffers) of the change amount buffer 23, the update unit 16 determines whether the correction vectors stored in the change amount buffer 23 are within a predetermined range. Here, the determination of whether or not they are within the predetermined range means determining whether the difference between the maximum and minimum values ​​of the horizontal component of the correction vector and the difference between the maximum and minimum values ​​of the vertical component are both within the predetermined range. For example, if the predetermined range is set to 0, this corresponds to determining whether all correction vectors stored in the change amount buffer 23 have the same value. When the correction vectors stored in the change amount buffer 23 are within the predetermined range, the update unit 16 determines that it is time to update the reference image.

[0042] For example, in the example in the upper part of Fig. 9, all correction vectors of the same value are stored in the change amount buffer 23, while in the example in the lower part, correction vectors of different values ​​are stored. Note that the example in Fig. 9 shows a case where the number of buffers K is 30. In this example, it is assumed that the predetermined range is 0, that is, when all correction vectors have the same value, it is determined that it is time to update the reference image. In this case, the update unit 16 determines that it is time to update in the example in the upper part, and determines that it is not time to update in the example in the lower part.

[0043] When it is determined that it is time to update the reference image, the update unit 16 creates a new reference image. For example, the update unit 16 creates an average image of the images stored in the input image buffer 21 as the reference image. By using the average image as the reference image in this way, it is possible to reduce the weight of the moving object area in the image. Note that the new reference image is not limited to the average image described above, and for example, the most recently input image stored in the input image buffer 21 may be used as the new reference image.

[0044] The update unit 16 stores the created new reference image in the reference image buffer 22. Furthermore, when the update unit 16 determines whether it is time to update the reference image, it clears the input image buffer 21 and the change amount buffer 23.

[0045] The image correction device 10 may be realized by, for example, a computer 40 shown in FIG. 10 . The computer 40 includes a CPU (Central Processing Unit) 41, a GPU (Graphics Processing Unit) 42, a memory 43 as a temporary storage area, and a non-volatile storage device 44. The computer 40 also includes an input / output device 45 such as an input device and a display device, and an R / W (Read / Write) device 46 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 47 that is connected to a network such as the Internet. The CPU 41, GPU 42, memory 43, storage device 44, input / output device 45, R / W device 46, and communication I / F 47 are connected to one another via a bus 48.

[0046] The storage device 44 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The storage device 44, which serves as a storage medium, stores an image correction program 50 for causing the computer 40 to function as the image correction device 10. The image correction program 50 includes an acquisition process control instruction 51, a read process control instruction 52, a calculation process control instruction 53, a correction process control instruction 54, an inspection process control instruction 55, and an update process control instruction 56. The storage device 44 also includes an information storage area 60 in which information constituting each of the input image buffer 21, the reference image buffer 22, and the change amount buffer 23 is stored.

[0047] The CPU 41 reads the image correction program 50 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions of the image correction program 50. The CPU 41 operates as the acquisition unit 11 shown in FIG. 5 by executing the acquisition process control instruction 51. The CPU 41 operates as the reading unit 12 shown in FIG. 5 by executing the reading process control instruction 52. The CPU 41 operates as the calculation unit 13 shown in FIG. 5 by executing the calculation process control instruction 53. The CPU 41 operates as the correction unit 14 shown in FIG. 5 by executing the correction process control instruction 54. The CPU 41 operates as the testing unit 15 shown in FIG. 5 by executing the testing process control instruction 55. The CPU 41 operates as the update unit 16 shown in FIG. 5 by executing the update process control instruction 56. Furthermore, the CPU 41 reads information from the information storage area 60 and loads the input image buffer 21, the reference image buffer 22, and the change amount buffer 23 into the memory 43. As a result, the computer 40 that executes the image correction program 50 functions as the image correction device 10. The CPU 41 that executes the program is hardware. Part of the program may also be executed by the GPU 42.

[0048] The functions realized by the image correction program 50 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0049] Next, the operation of the image correction device 10 according to the first embodiment will be described. When an input image is input to the image correction device 10, the image correction device 10 executes the image correction process shown in Fig. 11. Note that the image correction process is an example of an image correction method of the disclosed technology.

[0050] In step S11, the acquisition unit 11 acquires the input image input to the image correction device 10, and passes the acquired input image to the calculation unit 13 and stores it in the input image buffer 21. Next, in step S12, the reading unit 12 reads the reference image stored in the reference image buffer 22 and passes the read reference image to the calculation unit 13.

[0051] Next, in step S13, the calculation unit 13 calculates the motion vector of each block using a block matching method, and calculates an integrated motion vector from the average value of all the motion vectors, etc. Then, the calculation unit 13 calculates a correction vector in a direction that cancels out the integrated motion vector as the amount of change between the input image and the reference image.

[0052] Next, in step S14, the correction unit 14 performs vibration correction on the input image using, for example, affine transformation, in accordance with the correction vector calculated by the calculation unit 13. Next, in step S20, a testing process is executed. Here, the testing process will be described with reference to FIG. 12 .

[0053] In step S21, the testing unit 15 calculates, for example, ZNCC as the degree of difference between the reference image and the corrected image. Next, in step S22, the testing unit 15 determines whether the degree of difference is less than a threshold. If the degree of difference is less than the threshold, the process proceeds to step S23. If the degree of difference is equal to or greater than the threshold, the process proceeds to step S26.

[0054] In step S23, the testing unit 15 calculates the difference in pixel value of each pixel between the input image and the reference image, and calculates a difference pixel number C1, which is the number of pixels whose difference is equal to or greater than a threshold value (e.g., 50). The testing unit 15 also calculates the difference in pixel value of each pixel between the corrected image and the reference image, and calculates a difference pixel number C2, which is the number of pixels whose difference is equal to or greater than a threshold value (e.g., 50).

[0055] Next, in step S24, the testing unit 15 determines whether C1 is equal to or greater than C2. If C1≧C2, the process proceeds to step S25, and if C1<C2, the process proceeds to step S26.

[0056] In step S25, the inspection unit 15 determines that the corrected image is highly reliable (passes), and outputs the corrected image. On the other hand, in step S26, the inspection unit 15 determines that the corrected image is unreliable (fails), rejects the corrected image, and outputs the input image. Then, the process returns to the image correction process (FIG. 11).

[0057] Next, in step S30, an update process is executed, which will now be described with reference to FIG.

[0058] In step S31, the update unit 16 determines whether or not changes (in this embodiment, correction vectors) are stored in all the buffers (K buffers) of the change amount buffer 23. If changes are stored in all the buffers, the process proceeds to step S32, and if there is a buffer in which no change amount is stored, the process returns to the image correction process ( FIG. 11 ).

[0059] In step S32, the update unit 16 determines whether the amount of change stored in the change amount buffer 23 is within a predetermined range. If it is within the predetermined range, the process proceeds to step S33, and if it is not within the predetermined range, the process proceeds to step S35.

[0060] In step S33, the update unit 16 creates, for example, an average image of the images stored in the input image buffer 21 as a reference image. Next, in step S34, the update unit 16 stores the created new reference image in the reference image buffer 22. Next, in step S35, the update unit 16 clears the change amount buffer 23. Next, in step S36, the update unit 16 clears the input image buffer 21 and returns to the image correction process ( FIG. 11 ), whereupon the image correction process ends.

[0061] As described above, the image correction device according to the first embodiment compares a reference image with an input image to correct the input image. Furthermore, if the amount of change between the reference image and each of the multiple input images input sequentially falls within a predetermined range within a predetermined period, the image correction device updates the reference image based on the input images input within the predetermined period. That is, if the amount of change falls within a predetermined range within the predetermined period, it determines that there is a high probability that environmental changes, such as changes in the presence, position, shape, or illuminance of the subject, have occurred since the reference image was previously set, and sets the input image as a new reference image. This prevents a decrease in the accuracy of vibration correction even if changes have occurred in the environment since the reference image was captured.

[0062] Second Embodiment Next, a second embodiment will be described. In the image correction device according to the second embodiment, components similar to those of the image correction device 10 according to the first embodiment are denoted by the same reference numerals and descriptions thereof will be omitted, and functional units having some common functions are denoted by reference numerals having the same last two digits and descriptions of the common functions will be omitted.

[0063] 5, the image correction device 210 according to the second embodiment functionally includes an acquisition unit 11, a reading unit 12, a calculation unit 213, a correction unit 214, a testing unit 15, and an update unit 216. In addition, an input image buffer 21, a reference image buffer 22, and a change amount buffer 223 are provided in a predetermined storage area of ​​the image correction device 210.

[0064] The calculation unit 213 calculates the amount of change between the input image and the reference image. In the second embodiment, the calculation unit 213 calculates a pixel difference rate as the amount of change. Specifically, as shown in Fig. 14 , the calculation unit 213 calculates the difference between the pixel values ​​of each pixel between the input image and the reference image, and generates a difference image by binarizing the difference between each pixel based on a threshold value (e.g., 50). The calculation unit 213 calculates the pixel difference rate by dividing the number of pixels in the difference image whose difference is equal to or greater than the threshold value by the number of pixels in the entire image and multiplying the result by 100.

[0065] As shown in Figure 15, similar to the correction vector, the pixel difference rate also exhibits time-series variations when the timing of updating the reference image is inappropriate, such as during vibration or the occurrence of a moving obstacle. On the other hand, when the timing of updating after an environmental change is appropriate, the time-series variations are small. Therefore, similar to the correction vector, the pixel difference rate can be used to determine the timing of updating the reference image.

[0066] The calculation unit 213 passes the input image, the reference image, and the difference image between the input image and the reference image to the correction unit 214 , and stores the calculated difference pixel rate in the change amount buffer 223 .

[0067] The change amount buffer 223 is a buffer that stores the pixel difference ratio calculated by the calculation unit 213. It is similar to the change amount buffer 23 in the first embodiment, except that the stored values ​​are pixel difference ratios rather than correction vectors.

[0068] The correction unit 214 corrects the input image based on the difference image passed from the calculation unit 213. For example, the correction unit 214 corrects the input image based on the difference image so that the difference between the input image and the reference image becomes smaller.

[0069] When the pixel difference rates are stored in all buffers (K buffers) of the change amount buffer 223, the update unit 216 determines the timing to update the reference image based on whether the pixel difference rates stored in the change amount buffer 223 are within a predetermined range. Specifically, the update unit 216 calculates the difference between the maximum and minimum pixel difference rates stored in the change amount buffer 223 as a fluctuation range. When the calculated fluctuation range is equal to or less than a threshold value (e.g., 0.5%), the update unit 216 determines that it is time to update the reference image. The method for updating the reference image is the same as that of the update unit 16 in the first embodiment.

[0070] 16 , the maximum value of the difference pixel rate stored in the change amount buffer 223 is 3.6%, the minimum value is 3.4%, and the fluctuation range is 0.2%. In the example in the lower part, the maximum value of the difference pixel rate stored in the change amount buffer 223 is 4.7%, the minimum value is 2.2%, and the fluctuation range is 2.5%. In this example, when the threshold value is 0.5, the update unit 216 determines that it is time to update in the example in the upper part, and determines that it is not time to update in the example in the lower part.

[0071] The image correction device 210 may be realized by, for example, a computer 40 shown in Fig. 10. An image correction program 250 for causing the computer 40 to function as the image correction device 210 is stored in a storage device 44 of the computer 40. The image correction program 250 includes an acquisition process control instruction 51, a read process control instruction 52, a calculation process control instruction 253, a correction process control instruction 254, an inspection process control instruction 55, and an update process control instruction 256. The storage device 44 also includes an information storage area 60 in which information constituting each of the input image buffer 21, the reference image buffer 22, and the change amount buffer 223 is stored.

[0072] The CPU 41 reads the image correction program 250 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions contained in the image correction program 250. The CPU 41 operates as the calculation unit 213 shown in FIG. 5 by executing the calculation process control instruction 253. The CPU 41 also operates as the correction unit 214 shown in FIG. 5 by executing the correction process control instruction 254. The CPU 41 also operates as the update unit 216 shown in FIG. 5 by executing the update process control instruction 256. The other control instructions are the same as those of the image correction program 50 according to the first embodiment. As a result, the computer 40 that has executed the image correction program 250 functions as the image correction device 210.

[0073] The functions realized by the image correction program 250 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC, an FPGA, or the like.

[0074] Next, the operation of the image correction device 210 according to the second embodiment will be described. In the second embodiment, the image correction device 210 also executes the image correction process shown in FIG. 11 . However, the amount of change calculated in step S13 is a pixel difference ratio. Furthermore, in step S31 of the update process, the update unit 216 determines whether or not the pixel difference ratio is stored in all buffers (K buffers) of the change amount buffer 223. Furthermore, in step S32, the update unit 216 determines whether or not the fluctuation range of the pixel difference ratio is equal to or greater than a threshold. The processes in the other steps are the same as those in the image correction process according to the first embodiment.

[0075] As described above, in the second embodiment, the timing to update the reference image is determined based on the pixel difference ratio between the input image and the reference image, instead of the correction vector. As a result, as in the first embodiment, even if the environment has changed since the reference image was captured, it is possible to suppress a decrease in the accuracy of vibration correction.

[0076] The images output from the image correction device of each of the above embodiments can be used in a skeletal recognition system, etc. Furthermore, skeletal information recognized by the skeletal recognition system can be used in a variety of fields, such as gymnastics scoring support systems, analysis systems for other sports, monitoring systems, product manufacturing systems, and nursing care support systems. Furthermore, the images output from the image correction device of each of the above embodiments can be used for analysis and other processes other than skeletal recognition. Since the technology disclosed herein can suppress a decrease in the accuracy of vibration correction, it can suppress a decrease in the accuracy of subsequent processing regardless of the field in which it is used.

[0077] In the above embodiment, the change amount buffer and the input image buffer are cleared after determining whether it is time to update the reference image, but clearing is not essential and the most recent information may always be stored in each buffer. In this case, the reference image can be updated immediately when the condition that the change amount between the input image and the reference image is within a predetermined range within a predetermined period is satisfied.

[0078] In the above embodiment, the timing to update the reference image is determined when the amount of change between the input image and the reference image falls within a predetermined range within a predetermined period of time. However, this condition also holds when there is no change in the environment, such as immediately after calibration. Therefore, for a certain period of time, such as immediately after calibration, updating of the reference image may be skipped even if the above condition is met. Furthermore, the above condition may also include the amount of change between the input image and the reference image being equal to or greater than a predetermined value.

[0079] In addition, although the image correction program is pre-stored (installed) in the storage device in each of the above embodiments, this is not limiting. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0080] 10, 210 Image correction device 11 Acquisition unit 12 Reading unit 13, 213 Calculation unit 14, 214 Correction unit 15 Testing unit 16, 216 Update unit 21 Input image buffer 22 Reference image buffer 23, 223 Change amount buffer 40 Computer 41 CPU 42 GPU 43 Memory 44 Storage device 45 Input / output device 46 R / W device 47 Communication I / F 48 Bus 49 Storage medium 50, 250 Image correction program 51 Acquisition process control command 52 Reading process control command 53, 253 Calculation process control command 54, 254 Correction process control command 55 Testing process control command 56, 256 Update process control command 60 Information storage area

Claims

1. An image correction method in which a computer executes a process including: comparing a reference image with an input image, correcting the input image, and updating the reference image based on the input images input within a predetermined period if the amount of change between the reference image and each of a plurality of input images input sequentially is within a predetermined range within the predetermined period.

2. The image correction method according to claim 1, wherein the amount of change is a correction vector used when correcting the input image, the correction vector being calculated from a movement vector of the input image relative to the reference image.

3. The image correction method according to claim 1, wherein the amount of change is a pixel difference rate obtained by dividing the number of pixels that differ between the reference image and the input image by the number of pixels in the entire image.

4. An image correction method according to any one of claims 1 to 3, wherein the computer executes a process that further includes rejecting the corrected image when the degree of difference between the corrected image obtained by correcting the input image and the reference image is equal to or greater than a predetermined threshold.

5. An image correction method according to any one of claims 1 to 3, wherein the computer executes a process further comprising rejecting the corrected image when the number of pixels that differ between the corrected image obtained by correcting the input image and the reference image is greater than the number of pixels that differ between the input image and the reference image.

6. An image correction method according to any one of claims 1 to 3, wherein the process of updating the reference image includes updating the reference image with an average image of the input images input within the specified period.

7. An image correction program for causing a computer to execute a process including: comparing a reference image with an input image, correcting the input image, and updating the reference image based on the input images input within a predetermined period if the amount of change between the reference image and each of a plurality of input images input sequentially is within a predetermined range within the predetermined period.

8. The image correction program according to claim 7, wherein the amount of change is a correction vector calculated from a movement vector of the input image relative to the reference image when correcting the input image.

9. The image correction program according to claim 7, wherein the amount of change is a pixel difference rate obtained by dividing the number of pixels that differ between the reference image and the input image by the number of pixels in the entire image.

10. An image correction program according to any one of claims 7 to 9, for causing the computer to execute processing further including rejecting the corrected image when the degree of difference between the corrected image obtained by correcting the input image and the reference image is equal to or greater than a predetermined threshold.

11. An image correction program according to any one of claims 7 to 9, for causing the computer to execute processing further including rejecting the corrected image when the number of pixels difference between the corrected image obtained by correcting the input image and the reference image is greater than the number of pixels difference between the input image and the reference image.

12. An image correction program according to any one of claims 7 to 9, wherein the process of updating the reference image includes updating the reference image with an average image of the input images input within the specified period.

13. An image correction device comprising: a correction unit that compares a reference image with an input image and corrects the input image; and an update unit that updates the reference image based on the input images input within a specified period when the amount of change between the reference image and each of a plurality of input images input sequentially is within a specified range within the specified period.

14. The image correction device according to claim 13, wherein the amount of change is a correction vector used when correcting the input image, calculated from a movement vector of the input image relative to the reference image.

15. An image correction device according to claim 13, wherein the amount of change is a pixel difference rate obtained by dividing the number of pixels that differ between the reference image and the input image by the number of pixels in the entire image.

16. An image correction device according to any one of claims 13 to 15, further comprising a testing unit that rejects the corrected image when the degree of difference between the corrected image obtained by correcting the input image and the reference image is equal to or greater than a predetermined threshold.

17. An image correction device according to any one of claims 13 to 15, further comprising a testing unit that rejects the corrected image when the number of pixels that differ between the corrected image obtained by correcting the input image and the reference image is greater than the number of pixels that differ between the input image and the reference image.

18. An image correction device according to any one of claims 13 to 15, wherein the update unit updates the reference image with an average image of the input images input within the predetermined period.

19. A non-transitory storage medium storing an image correction program for causing a computer to execute a process including: comparing a reference image with an input image, correcting the input image, and updating the reference image based on the input images input within a predetermined period if the amount of change between the reference image and each of a plurality of input images input sequentially is within a predetermined range within the predetermined period.

Images