Image processing system and image processing device

Abstract

To provide an image processing system capable of improving measurement accuracy in a displacement amount of an object, and an image processing device.SOLUTION: An image processing system comprises: a first imaging device which captures a first video of an object; a second imaging device which captures a second video of the object from a direction different from the first imaging device; a first parallax-information generation unit which generates first parallax information representing the difference between the first video and the second video; a division unit which divides the first parallax information into a plurality of frames; and a second parallax-information generation unit which subjects each of the divided frames of the first parallax information to super-resolution processing and integrates each frame of the first parallax information having been subjected to super-resolution processing to generate second parallax information having a higher resolution than the first parallax information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing system and an image processing device that perform image processing of multiple videos captured by multiple image capture devices. [Background technology]

[0002] One known method for measuring the amount of displacement of an object when it is subjected to an external force is a technology called DIC (Digital Image Correlation). DIC captures video of the object's surface and calculates the amount of displacement of the object's surface for each frame through image processing. Furthermore, by using a stereo camera in DIC, it is possible to measure the amount of displacement in the out-of-plane direction of the object from the parallax information of the two videos (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-170831 Summary of the Invention [Problem to be solved by the invention]

[0004] Because DIC performs analysis based on video, there is a relationship between resolution and measurement accuracy; the higher the video resolution, the better the accuracy of measuring displacement in small areas. When measuring the displacement of an entire object, the camera must be moved farther away from the object, which reduces the resolution per unit area. As a result, the accuracy of measuring displacement in small areas decreases due to the above characteristics. A known method for increasing resolution is super-resolution processing, which uses neural networks to increase the resolution of video. However, when calculating the displacement of an object using two videos, if the two videos are super-resolution processed separately, the correlation between the two videos is lost, resulting in a decrease in the accuracy of measuring displacement.

[0005] The present disclosure is intended to solve the above-mentioned problems, and aims to provide an image processing system and an image processing device that can improve the accuracy of measuring the amount of displacement of an object. [Means for solving the problem]

[0006] The image processing system according to the present disclosure includes a first image capturing device that captures a first video of an object, a second image capturing device that captures a second video of the object from a different direction from the first image capturing device, a first parallax information generation unit that generates first parallax information that is the difference between the first video and the second video, a division unit that divides the first parallax information into a plurality of frames, and a second parallax information generation unit that performs super-resolution processing on each frame of the divided first parallax information and combines each frame of the first parallax information that has been subjected to super-resolution processing to generate second parallax information with a higher resolution than the first parallax information.

[0007] The image processing device according to the present disclosure includes a first parallax information generation unit that generates first parallax information, which is the difference between a first video of an object captured by a first imaging device and a second video of an object captured by a second imaging device from a different direction from the first imaging device; a division unit that divides the first parallax information into a plurality of frames; and a second parallax information generation unit that performs super-resolution processing on each frame of the divided first parallax information and combines the frames of the first parallax information that have been subjected to super-resolution processing to generate second parallax information with a higher resolution than the first parallax information. [Effects of the Invention]

[0008] According to the present disclosure, by generating second parallax information by performing super-resolution processing on each frame of first parallax information, which is the difference between the first video and the second video, it is possible to maintain the relationship between the two videos, thereby improving the accuracy of measuring the amount of displacement of an object using the second parallax information. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic configuration diagram of an image processing system according to a first embodiment. [Figure 2]FIG. 2 is a diagram illustrating an example of super-resolution processing according to the first embodiment. [Figure 3] 4 is a flowchart showing the flow of image processing in the first embodiment. [Figure 4] FIG. 2 is a conceptual diagram of image processing in the first embodiment. [Figure 5] FIG. 10 is a schematic configuration diagram of an image processing system according to a second embodiment. [Figure 6] 10 is a flowchart showing the flow of image processing in the second embodiment. [Figure 7] FIG. 10 is a conceptual diagram of image processing in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of an image processing system and an image processing device according to the present disclosure will be described with reference to the drawings. In each drawing, components denoted by the same reference numerals are identical or equivalent components, and this is common throughout the specification. Note that in each drawing, the relative dimensional relationships or shapes of each component may differ from those in reality.

[0011] Embodiment 1 Fig. 1 is a schematic configuration diagram of an image processing system 100 according to embodiment 1. As shown in Fig. 1, the image processing system 100 includes a first image capturing device 11, a second image capturing device 12, and an image processing device 2. The first image capturing device 11, the second image capturing device 12, and the image processing device 2 are connected to each other via wire or wirelessly so as to be able to communicate with each other.

[0012] The first and second image capturing devices 11 and 12 are horizontally installed stereo cameras, each equipped with an image capturing element such as a CCD or CMOS, and a lens that forms an image of the subject on the image capturing element. The first and second image capturing devices 11 and 12 capture moving images of an object 15 from two different directions. The object 15 is a measurement target whose displacement is to be measured. The first moving image MA captured by the first image capturing device 11 and the second moving image MB captured by the second image capturing device 12 are transmitted to the image processing device 2.

[0013] The image processing device 2 is a computer equipped with a processor such as a CPU or a GPU, a memory, an I / O port, etc. The image processing device 2 of this embodiment performs image processing on the first moving image MA and the second moving image MB, and generates high-resolution parallax information used to measure the amount of displacement of the object 15 in the out-of-plane direction.

[0014] 1, the image processing device 2 includes a first disparity information generation unit 21, a division unit 22, a second disparity information generation unit 23, and a storage unit 24. The first disparity information generation unit 21, the division unit 22, and the second disparity information generation unit 23 are functional units implemented by a processor executing a program. Alternatively, at least one of the first disparity information generation unit 21, the division unit 22, and the second disparity information generation unit 23 may be implemented by a processing circuit such as an ASIC or an FPGA.

[0015] The first parallax information generator 21 generates first parallax information, which is the difference between the first moving image MA and the second moving image MB. The first parallax information is a parallax video indicating the distance in the out-of-plane direction (depth direction) of the object 15, and is obtained by known stereo matching.

[0016] The dividing unit 22 divides into a plurality of frames the first parallax information generated by the first parallax information generating unit 21. In other words, the dividing unit 22 divides the parallax videos of the first moving image MA and the second moving image MB into a plurality of parallax images.

[0017] The second parallax information generation unit 23 generates second parallax information by increasing the resolution of the first parallax information. The second parallax information generation unit 23 includes a super-resolution processing unit 231 and an integration unit 232. The super-resolution processing unit 231 performs super-resolution processing on each frame (parallax image) of the divided first parallax information to increase the resolution. In other words, the super-resolution processing unit 231 increases the number of pixels in each frame of the first parallax information by an arbitrary multiple greater than 1 (for example, double) through the super-resolution processing.

[0018] The super-resolution processor 231 performs super-resolution processing using, for example, a trained neural network. The neural network algorithm used for super-resolution processing is Super-Resolution Convolutional Neural Network (SRCNN), Very Deep Super Resolution (VDSR), or Video Super Resolution (VSRnet), etc.

[0019] 2 is a diagram illustrating an example of the super-resolution processing according to the first embodiment. As shown in FIG. 2, the super-resolution processing unit 231 processes a target frame D1 t , target frame D1 t The previous frame D1 t-1 , and target frame D1 t Frame D1, one frame after t+1 Then, the spatial information and time series information of the three frames are combined and 3D convolution is performed. Finally, the target frame D1 t The result of the convolution operation is added to produce a high-resolution frame D2. t This makes it possible to increase the resolution of the parallax video while taking into account the time-series information of the video. Note that the super-resolution processing by the super-resolution processor 231 is not limited to that using a neural network, and various known high-resolution techniques such as compressed sensing can be used.

[0020] The integrating unit 232 integrates the frames of the first parallax information that have been subjected to the super-resolution process, and generates a high-resolution parallax video as second parallax information.

[0021] The storage unit 24 is, for example, a non-volatile semiconductor memory such as a ROM or flash memory, a volatile semiconductor memory such as a RAM, an HDD, or an SSD. The storage unit 24 stores programs used for image processing, as well as various data such as calculation formulas and thresholds used to execute the programs. The storage unit 24 also stores the first moving image MA and the second moving image MB received from the first image capture device 11 and the second image capture device 12, as well as various data input or generated during execution of image processing.

[0022] Fig. 3 is a flowchart showing the flow of image processing in embodiment 1. Fig. 4 is a conceptual diagram of image processing in embodiment 1. The image processing performed by the image processing device 2 will be described with reference to Figs. 3 and 4. First, a first moving image MA and a second moving image MB are acquired from the first image capturing device 11 and the second image capturing device 12, respectively (S1).

[0023] The first parallax information generation unit 21 performs stereo matching on each frame of the first moving image MA and the second moving image MB to generate first parallax information (S2). Here, a parallax video D1 is generated as the first parallax information. Next, the division unit 22 divides the first parallax information into a plurality of frames (S3). As shown in FIG. 4, the division unit 22 divides the parallax video D1 into a plurality of frames D11, D12, ... D1 N N is the number of frames in the parallax video D1.

[0024] Then, the super-resolution processing unit 231 of the second parallax information generation unit 23 divides each of the divided frames D11, D12, . . . D1 N As shown in Figure 4, the super-resolution processing results in high-resolution frames D21, D22, and D2 N The integrating unit 232 obtains the high-resolution frames D21, D22, ..., D2 N Then, the second disparity information is generated by integrating the second disparity information (S5). Here, a high-resolution disparity video D2 is generated as the second disparity information.

[0025] The second parallax information generated by the image processing device 2 is used to measure the amount of displacement of the object 15 in the out-of-plane direction (depth direction) in strain measurement using DIC, automatic driving of a car, or human position detection using an air conditioner.

[0026] The effects of the image processing system 100 of this embodiment will be described. When measuring the amount of displacement of an object 15 in the out-of-plane direction from multiple moving images in DIC or the like, one possible way to improve measurement accuracy is to increase the resolution of each of the multiple moving images and generate parallax information from the multiple high-resolution moving images. In this case, it is common to increase the resolution of each moving image frame by frame using super-resolution processing, but dividing the moving images into frames results in the loss of time-series information (relative changes over time) in each moving image, leading to a decrease in measurement accuracy.

[0027] In contrast, in this embodiment, first parallax information of the first moving image MA and the second moving image MB is generated, and then each frame of the first parallax information is subjected to super-resolution processing to obtain high-resolution second parallax information, thereby maintaining the time-series information of the first moving image MA and the second moving image MB. Furthermore, by using the high-resolution parallax information, the measurement accuracy of the amount of displacement of the object 15 in the out-of-plane direction can be improved.

[0028] Embodiment 2 A second embodiment will now be described. Fig. 5 is a schematic diagram of an image processing system 100A according to the second embodiment. As shown in Fig. 5, the image processing system 100A according to the second embodiment differs from that according to the first embodiment in the function of the image processing device 2A. The configuration of the image processing device 2A and the functions and configurations of the first and second image capturing devices 11 and 12 are the same as those according to the first embodiment.

[0029] 5, the image processing device 2A includes a first disparity information generation unit 21, a division unit 22, a second disparity information generation unit 23, a storage unit 24, a first high-resolution video generation unit 25, and a second high-resolution video generation unit 26. The first high-resolution video generation unit 25 and the second high-resolution video generation unit 26 are functional units implemented by a processor executing a program. Alternatively, at least one of the first high-resolution video generation unit 25 and the second high-resolution video generation unit 26 may be implemented by a processing circuit such as an ASIC or an FPGA.

[0030] The functions of the first parallax information generation unit 21, the second parallax information generation unit 23, and the storage unit 24 in this embodiment are the same as those in embodiment 1. The division unit 22 in this embodiment has a function of dividing the first parallax information generated by the first parallax information generation unit 21 into a plurality of frames, as in embodiment 1, and also has a function of dividing the first moving image MA into a plurality of frames (first images).

[0031] The first high-resolution moving image generation unit 25 generates a high-resolution first moving image by increasing the resolution of the first moving image MA. The first high-resolution moving image generation unit 25 has a super-resolution processing unit 251 and an integration unit 252. The super-resolution processing unit 251 performs super-resolution processing on each frame (first image) of the divided first moving image MA to increase the resolution. In other words, the super-resolution processing unit 251 increases the number of pixels of each frame of the first moving image MA by an arbitrary multiple greater than 1 (for example, double) through super-resolution processing.

[0032] The super-resolution processing unit 251 increases the resolution of each frame of the first moving image MA using a trained neural network or the like, as in embodiment 1. The multiple of the resolution achieved by the super-resolution processing unit 251 is the same as the multiple of the resolution achieved by the super-resolution processing unit 231. The method of super-resolution processing in the super-resolution processing unit 251 may be the same as or different from the method of super-resolution processing in the super-resolution processing unit 231.

[0033] The integrating unit 252 integrates the frames of the first moving image MA that have been subjected to the super-resolution processing to generate a high-resolution first moving image.

[0034] The second high-resolution video generator 26 generates a high-resolution second video from the second parallax information generated by the second parallax information generator 23 and the high-resolution first video generated by the first high-resolution video generator 25. Specifically, the second high-resolution video generator 26 generates a high-resolution second video by shifting the high-resolution first video by the difference indicated in the second parallax information. The high-resolution second video corresponds to a video obtained by increasing the resolution of the second video MB.

[0035] Fig. 6 is a flowchart showing the flow of image processing in the second embodiment. Fig. 7 is a conceptual diagram of the image processing in the second embodiment. The image processing performed by the image processing device 2A will be described with reference to Figs. 6 and 7. First, high-resolution second parallax information is generated by the same processes as in the first embodiment, steps S1 to S5.

[0036] Furthermore, in parallel with the processing of steps S1 to S5, the dividing unit 22 divides the first moving image MA into a plurality of frames (S11). As shown in FIG. 7, the dividing unit 22 divides the first moving image MA into a plurality of frames MA1, MA2, ... MA N N is the number of frames in the first moving image MA.

[0037] Then, the super-resolution processor 251 divides each of the divided frames MA11, MA2, ... MA N As shown in FIG. 7, the super-resolution processing is performed on the high-resolution frame MA. H1 , M.A. H2 ···MA HN The integrating unit 252 obtains each of the high-resolution frames MA H1 , M.A. H2 ···MA HN Integrating the first high-resolution video MA H is generated (S13).

[0038] Then, the second high-resolution video generator 26 combines the second parallax information generated in step S5 and the high-resolution first video MA generated in step S13. H And using high resolution second video MB His generated (S14).

[0039] High-resolution first video MA generated by image processing device 2A H and high-resolution second video MB H is used to measure the amount of displacement (strain) in the out-of-plane direction and in-plane direction of the object 15 in DIC or the like.

[0040] The effects of the image processing system 100 of this embodiment will be described. When measuring the amount of displacement of an object 15 in an in-plane direction from multiple videos in DIC or the like, one possible way to improve measurement accuracy is to increase the resolution of each of the multiple videos and measure changes over time. In this case, it is common to increase the resolution of each video frame by frame using super-resolution processing. However, when each video is subjected to super-resolution processing individually, spatial information in the multiple videos due to the positional relationship between the multiple cameras is lost, leading to a decrease in measurement accuracy.

[0041] In contrast to this, in the present embodiment, by obtaining a high-resolution second moving image from the high-resolution first moving image and the second parallax information obtained in embodiment 1, it is possible to maintain not only the time-series information of the first moving image MA and the second moving image MB, but also the spatial information. Then, by using the high-resolution first moving image, the high-resolution second moving image, and the parallax information, it is possible to improve the measurement accuracy of the displacement amount of the object 15 in the out-of-plane direction and the in-plane direction.

[0042] The above is a description of the embodiments, but the present disclosure is not limited to the above embodiments and can be modified in various ways without departing from the spirit of the present disclosure. Furthermore, the present disclosure includes all possible combinations of the configurations described in the above embodiments. For example, in the above embodiments, the first parallax information is a parallax video indicating the distance in the out-of-plane direction of the object 15, but this is not limited to this. For example, the first parallax information may be a numerical value such as a distance value or parallax value indicating the distance in the out-of-plane direction (depth direction) of each pixel.

[0043] The image processing devices 2 and 2A may also include a noise removal unit that removes noise from at least one of the first parallax information and the second parallax information. The noise removal unit removes noise by averaging the parallax values ​​within a predetermined range, or by adjusting the parallax value of a pixel to the parallax values ​​of the surrounding pixels when the difference between the parallax value of the pixel and the parallax values ​​of the surrounding pixels is equal to or greater than a predetermined threshold. This improves the accuracy of the first parallax information and the second parallax information, thereby further improving the measurement accuracy of the displacement amount of the object 15.

[0044] Furthermore, the image processing device 2 of the first embodiment may include a displacement amount measurement unit that measures the amount of displacement of the object 15 in the out-of-plane direction based on the second parallax information. The displacement amount measurement unit uses DIC to measure the amount of displacement of the object 15 in the out-of-plane direction from a time change in the parallax video, which is the second parallax information. Similarly, the image processing device 2A of the second embodiment may include a displacement amount measurement unit that measures the amount of displacement of the object 15 in the in-plane direction and the out-of-plane direction based on the second parallax information, the first high-resolution video, and the second high-resolution video. The displacement amount measurement unit uses DIC to measure the amount of displacement of the object 15 in the out-of-plane direction from a time change in the parallax video, which is the second parallax information, and measures the amount of displacement of the object 15 in the in-plane direction from a time change in the first high-resolution video and the second high-resolution video.

[0045] Furthermore, the functional units of the image processing devices 2 and 2A are not limited to those in the above-described embodiment. For example, the functional units of the image processing devices 2 and 2A may be implemented by dividing them into multiple computers.

[0046] Various aspects of the present disclosure are summarized below as appendices.

[0047] (Appendix 1) a first image capture device that captures a first video of the object; a second image capturing device that captures a second moving image of the object from a different direction from that of the first image capturing device; a first parallax information generating unit that generates first parallax information that is a difference between the first moving image and the second moving image; a division unit that divides the first parallax information into a plurality of frames; a second disparity information generation unit that performs super-resolution processing on each frame of the divided first disparity information, and integrates the frames of the first disparity information that have been subjected to the super-resolution processing to generate second disparity information having a higher resolution than the first disparity information. (Appendix 2) a first high-resolution video generator; a second high-resolution video generator; the dividing unit divides the first moving image into a plurality of frames, the first high-resolution moving image generation unit performs super-resolution processing on each frame of the divided first moving image, and integrates the frames of the first moving image that have been subjected to the super-resolution processing to generate a high-resolution first moving image; 2. The image processing system according to claim 1, wherein the second high-resolution video generator generates a high-resolution second video from the second disparity information and the high-resolution first video. (Appendix 3) 3. The image processing system according to claim 1, further comprising a noise removal unit that removes noise from at least one of the first disparity information and the second disparity information. (Appendix 4) 4. The image processing system according to any one of claims 1 to 3, further comprising a displacement amount measuring unit that measures an amount of displacement of the object in an out-of-plane direction from the second parallax information. (Appendix 5) 5. The image processing system according to any one of claims 2 to 4, further comprising a displacement amount measuring unit that measures displacement amounts in an in-plane direction and an out-of-plane direction of the object from the first high-resolution video and the second high-resolution video. [Explanation of symbols]

[0048] 2, 2A Image processing device, 11 First imaging device, 12 Second imaging device, 15 Object, 21 First parallax information generation unit, 22 Division unit, 23 Second parallax information generation unit, 24 Memory unit, 25 First high-resolution video generation unit, 26 Second high-resolution video generation unit, 100, 100A Image processing system, 231 Super-resolution processing unit, 232 Integration unit, 251 Super-resolution processing unit, 252 Integration unit.

Claims

1. a first image capture device that captures a first moving image of the object; a second image capturing device configured to capture a second moving image of the object from a different direction from that of the first image capturing device; a first parallax information generating unit that generates first parallax information that is a difference between the first moving image and the second moving image; a division unit that divides the first parallax information into a plurality of frames; a second disparity information generation unit that performs super-resolution processing on each frame of the divided first disparity information, and integrates the frames of the first disparity information that have been subjected to the super-resolution processing to generate second disparity information having a higher resolution than the first disparity information.

2. a first high-resolution video generator; a second high-resolution video generator; the dividing unit divides the first moving image into a plurality of frames, the first high-resolution moving image generation unit performs super-resolution processing on each frame of the divided first moving image, and integrates the frames of the first moving image that have been subjected to the super-resolution processing to generate a high-resolution first moving image; The image processing system according to claim 1 , wherein the second high-resolution video generator generates a second high-resolution video from the second parallax information and the first high-resolution video.

3. The image processing system according to claim 1 , further comprising a noise removal unit that removes noise from at least one of the first parallax information and the second parallax information.

4. The image processing system according to claim 1 , further comprising a displacement amount measuring unit that measures an amount of displacement of the object in an out-of-plane direction from the second parallax information.

5. The image processing system according to claim 2 , further comprising a displacement amount measuring unit that measures displacement amounts in an in-plane direction and an out-of-plane direction of the object from the first high-resolution video and the second high-resolution video.

6. a first parallax information generator that generates first parallax information that is a difference between a first video of an object captured by a first image capturing device and a second video of the object captured by a second image capturing device from a direction different from that of the first image capturing device; a division unit that divides the first parallax information into a plurality of frames; a second disparity information generation unit that performs super-resolution processing on each frame of the divided first disparity information, and integrates the frames of the first disparity information that have been subjected to the super-resolution processing to generate second disparity information having a higher resolution than the first disparity information.

Images