Calculation device, monitoring system, parallax calculation method

Abstract

To reduce error matching to improve calculation precision for parallax.SOLUTION: An arithmetic unit is connected to a first imaging unit and a second imaging unit, and generates parallax information between a first image photographed by the first imaging unit and a second image captured by the second imaging unit through a search in predetermined matching block units, and also comprises: a similarity generation unit which generates a first similarity of a matching block in a search range of the second image to a matching block in the first image in the matching block units; a similarity correction unit which corrects the first similarity using at least one of first correction information generated on the basis of the first similarity of an arbitrary adjacent matching block in the search range of the second image and second correction information generated on the basis of the parallax information so as to generate a second similarity; and a similarity determination unit which generates parallax information on the basis of the second similarity.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an arithmetic device, a monitoring system, and a parallax calculation method.

Background Art

[0002] A method is known in which a stereo camera is used to calculate the distance between the camera and an observation object, and recognition processing of the observation object is performed. In Patent Document 1, among the left and right captured images captured by a stereo camera that captures the same space from left and right viewpoints, each of a plurality of regions obtained by dividing one of the images is used as a reference block, and for each reference block, a search range is set in the other image, and similarity data is generated by associating the similarity between the image and the reference block with the positions within the search range. The similarity data generation unit, a similarity correction unit that smooths the similarity data in the spatial direction based on the similarity data generated for a predetermined number of reference blocks around the corresponding reference block, a result evaluation unit that detects the position at which the similarity becomes the maximum value in each smoothed similarity data, and using the detection result by the result evaluation unit, a parallax is obtained for each reference block, and based on this, a depth image is generated by calculating the position in the depth direction of the subject and associating it with the image plane. And an output information generation unit that performs predetermined information processing based on the position of the subject in the three-dimensional space using the depth image and outputs the result. An information processing apparatus is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the invention described in Patent Document 1, there is room for improvement in the calculation of parallax.

Means for Solving the Problem

[0005] The arithmetic unit according to the first aspect of the present invention is connected to a first imaging unit and a second imaging unit, and generates parallax information between a first image captured by the first imaging unit and a second image captured by the second imaging unit by searching in units of predetermined matching blocks. The arithmetic unit is Before Record the matching block in the first image And before And the matching block within the search range of the second image Based on the difference in luminance values between corresponding pixels in, the A similarity generation unit that generates a first similarity, a first correction information generated based on the first similarity of any adjacent matching blocks within the search range of the second image, and a second correction information generated based on the parallax information. A similarity correction unit that corrects the first similarity using at least one of them to generate a second similarity, and based on the second similarity The A similarity determination unit that generates parallax information, and is provided with. The parallax calculation method according to the second aspect of the present invention is a parallax calculation method executed by an arithmetic unit that is connected to a first imaging unit and a second imaging unit, and generates parallax information between a first image captured by the first imaging unit and a second image captured by the second imaging unit by searching in units of predetermined matching blocks. The parallax calculation method is Before Record the matching block in the first image And And the matching block within the search range of the second image Based on the difference in luminance values between corresponding pixels in, in units of the matching block Generating a first similarity, correcting the first similarity using at least one of the first correction information generated based on the first similarity of any adjacent matching blocks within the search range of the second image and the second correction information generated based on the parallax information to generate a second similarity, and based on the second similarity The Generating parallax information, and includes.

Advantages of the Invention

[0006] According to the present invention, false matching can be reduced and the calculation accuracy of parallax can be improved.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Best Mode for Carrying Out the Invention

[0008] —First Embodiment— Hereinafter, a first embodiment of the monitoring system will be described with reference to FIGS. 1 to 7.

[0009] FIG. 1 is a configuration diagram of a monitoring system S including a computing device 1. The vehicle peripheral monitoring system S includes a computing device 1, a first imaging unit 2, a second imaging unit 3, a recognition processing unit 4, and a vehicle control unit 5. The vehicle peripheral monitoring system S is mounted on a vehicle C. The computing device 1 is connected to the first imaging unit 2, the second imaging unit 3, and the recognition processing unit 4. The recognition processing unit 4 is connected to the computing device 1 and the vehicle control unit 5.

[0010] Each of the computing device 1, the recognition processing unit 4, and the vehicle control unit 5 is an ECU (Electronic Control Unit) mounted on the vehicle C. However, each of the computing device 1, the recognition processing unit 4, and the vehicle control unit 5 does not necessarily have to be an individual ECU, and two or more of them may be realized by the same ECU.

[0011] Both the first imaging unit 2 and the second imaging unit 3 are cameras, and output captured images, which are the images obtained by shooting, to the arithmetic unit 1. The captured image output by the first imaging unit 2 is called the first captured image 100, and the captured image output by the second imaging unit 3 is called the second captured image 101. The first imaging unit 2 and the second imaging unit 3 are arranged facing the same direction of the vehicle C and separated by a predetermined baseline length in the lateral direction. Hereinafter, the lateral direction in which the first imaging unit 2 and the second imaging unit 3 are arranged is also referred to as the "baseline direction". Based on the first captured image 100 and the second captured image 101, the arithmetic unit 1 generates parallax information 106 by the method described later and outputs it to the recognition processing unit 4.

[0012] The arithmetic unit 1 includes an input unit 10, a similarity generation unit 11, a similarity correction unit 12, and a similarity determination unit 13. The input unit 10 is realized by a hardware interface with the first imaging unit 2 and the second imaging unit 3, and an arithmetic processing device. The similarity generation unit 11, the similarity correction unit 12, and the similarity determination unit 13 are realized by an arithmetic processing device. The hardware interface is, for example, RJ45 or Camera Link.

[0013] The arithmetic processing device includes, for example, a CPU which is a central processing unit, a ROM which is a read-only storage device, and a RAM which is a readable and writable storage device. The CPU expands the program stored in the ROM to the RAM and executes it to perform the arithmetic operations described later. The arithmetic processing device may be realized by an FPGA (Field Programmable Gate Array) which is a rewritable logic circuit or an ASIC (Application Specific Integrated Circuit) which is an application-specific integrated circuit instead of the combination of the CPU, the ROM, and the RAM. Also, the arithmetic processing device may be realized by a combination of different configurations, for example, a combination of the CPU, the ROM, the RAM, and the FPGA instead of the combination of the CPU, the ROM, and the RAM.

[0014] The input unit 10 supplies the similarity generation unit 11 with the first image 102 obtained by performing image processing on the first captured image 100 and the second image 103 obtained by performing image processing on the second captured image 101. This image processing is executed using, for example, the internal parameters and external parameters of the camera, and eliminates the adverse effects on the image processing. Examples of the image processing include correction of lens distortion, correction of errors in the mounting position and mounting posture, and the like. However, since the processing in the input unit 10 in the present embodiment is not essential, the first captured image 100 and the first image 102 may be regarded as the same, or the second captured image 1021 and the second image 103 may be regarded as the same.

[0015] The similarity generation unit 11 takes the first image 102 and the second image 103 supplied from the input unit 10 as inputs, generates first similarity information 104, and supplies it to the similarity correction unit 12. The similarity correction unit 12 is information representing the similarity between the matching block in the first image 102 and the matching block in the second image 103. The first similarity information 104 is, for example, the amount of luminance variation between each pixel in the matching block of the first image 102 and the corresponding pixel in the matching block of the second image 103. As the method for calculating the amount of luminance variation, for example, methods such as SAD (Sum of Absolute Difference), ZSAD (Zero means Sum of Absolute Difference), SSD (Sum of Squared Difference), NCC (Normalized Cross Correlation), and ZNCC (Zero means Normalized Cross Correlation) can be used.

[0016] The first similarity information 104 may also be the amount of luminance variation, the direction of luminance variation, or the luminance variation pattern between each pixel in the matching block of the first image 102 and its surrounding pixels, and between each pixel in the matching block of the second image 103 and its surrounding pixels. As an example of the luminance variation pattern, there is an increase / decrease pattern from the left for a horizontal three-pixel region including the left and right adjacent pixels of the target pixel. For example, when the value of the left adjacent pixel is 10, the value of the target pixel is 25, and the value of the right adjacent pixel is 80, the increase / decrease pattern is "increase → increase".

[0017] Based on the first similarity information 104 or the parallax information 106, the similarity correction unit 12 generates the second similarity information 105 obtained by correcting the first similarity information 104. Based on the second similarity information 105, the similarity determination unit 13 detects the position with the highest similarity within the search range, and generates the parallax information 106 based on this position information. The similarity determination unit 13 outputs the generated parallax information 106 to the recognition processing unit 4 and the similarity correction unit 12.

[0018] Taking the parallax information 106 as an input, the recognition processing unit 4 performs various recognition processes. As an example of the recognition process in the recognition processing unit 4, there is stereo object detection using the parallax information 106. Also, as an example of the recognition target by the recognition processing unit 4, there are the position information, type information, motion information, and danger information of the subject. As an example of the position information, there are the direction and distance from the host vehicle. As an example of the type information, there are pedestrians, adults, children, elderly people, animals, falling rocks, bicycles, surrounding vehicles, surrounding structures, and curbstones. As an example of the motion information, there are the wobbling, jumping out, crossing, moving direction, moving speed, and moving trajectory of pedestrians and bicycles. As an example of the danger information, there are pedestrians jumping out, falling rocks, and abnormal motions of surrounding vehicles such as sudden stops, sudden decelerations, and sudden steering. The recognition information 107 generated by the recognition processing unit 4 is supplied to the vehicle control unit 5.

[0019] Based on the recognition information 107, various vehicle controls of the vehicle C are performed by the vehicle control unit 5. As an example of the vehicle control implemented by the vehicle control unit 5, there are brake control, steering control, accelerator control, in-vehicle lamp control, warning sound generation, in-vehicle camera control, and information output regarding the observation target around the imaging device to surrounding vehicles and remote center devices connected via a network. As a specific example, there is speed and brake control according to the parallax information 106 of an obstacle existing in front of the vehicle.

[0020] Note that distance information may be generated based on the parallax information and used instead of the parallax information 106. Although not described in this embodiment, the vehicle control unit 5 may perform a subject detection process based on the image processing result using the first image 102 or the second image 103. Further, the vehicle control unit 5 may display an image obtained via the first imaging unit 2 or the second imaging unit 3 or a display for the viewer on a display device connected to the vehicle control unit 5. Furthermore, the vehicle control unit 5 may supply information on the observed object detected based on the image processing result to an information device that processes traffic information such as map information and traffic jam information.

[0021] FIG. 2 is a configuration diagram of the similarity correction unit 12. The similarity correction unit 12 includes a first correction information generation unit 20, a second correction information generation unit 21, and a correction execution unit 22. The first correction information generation unit 20 generates first correction information 200 based on the first similarity information 104. The second correction information generation unit 21 generates second correction information 201 based on the parallax information 106. The correction execution unit 22 corrects the first similarity information 104 based on the first correction information 200 and the second correction information 201, and generates second similarity information 105.

[0022] The configuration of the first correction information generation unit 20 will be described. The first correction information generation unit 20 includes a correction determination unit 40 and a first generation unit 41. The correction determination unit 40 generates correction valid information 400 indicating the necessity of generating the first correction information 200 based on the first similarity information 104. For example, the correction determination unit 40 determines that it is necessary to generate the first correction information 200 when the following first determination condition is satisfied, and determines that it is unnecessary to generate the first correction information 200 when the first determination condition is not satisfied.

[0023] (Similarity S(i - 1) - Similarity S(i)) > TH0 and (Similarity S(i + 1) - Similarity S(i)) > TH1

[0024] Note that the similarity A(i) is the similarity at the search position A(i) to be determined within the search range, the similarity A(i-1) is the similarity at the search position A(i-1) adjacent to the left of the search position A(i), and the similarity A(i+1) is the similarity at the search position A(i+1) adjacent to the right of the search position A(i). Also, TH0 and TH1 are predefined threshold values. Furthermore, the magnitude determination in the first determination condition may be changed so that the condition is satisfied when the magnitudes are equal.

[0025] The reason for referring to the left and right instead of up and down in the first determination condition, that is, the reason for scanning in the horizontal direction, is that the baseline direction is horizontal. Also, the first determination condition may be determined using the similarities of five points as follows instead of just three points. However, here, TH2 to TH5, which are predefined threshold values, are used.

[0026] (Similarity S(i-2) - Similarity S(i-1)) > TH2 and (Similarity S(i-1) - Similarity S(i)) > TH3 and (Similarity S(i+1) - Similarity S(i)) > TH4 and (Similarity S(i+2) - Similarity S(i+1)) > TH5

[0027] The first generation unit 41 takes the first similarity information 104 and the correction valid information 400 as inputs, and generates the first correction information 200 based on the correction valid information 400. The first correction information 200 is, for example, a combination of the search position to be corrected and the corrected similarity information, or a combination of a newly generated search position and similarity information. The correction execution unit 22 corrects the similarity with reference to the first correction information 200. The operation of the first correction information generation unit 20 will be described with reference to FIG. 3.

[0028] Figure 3 is a diagram for explaining the first similarity information 104 and the first correction information 200. The upper part of Figure 3 shows the whole of the similarity curve 1010, and the lower part of Figure 3 shows the details of the position part of the similarity curve 1010. Specifically, the similarity curve 1012 shown in the lower part of Figure 3 indicates the range of the reference numeral 1011 in the upper part of Figure 3. In the similarity curve 1010 and the similarity curve 1012, the horizontal axis represents the horizontal search position within the search range, and the vertical axis represents the similarity. The lower the vertical axis, the higher the similarity.

[0029] The black circles shown in the similarity curve 1012 in the lower part of Figure 3 indicate the similarity at each search position. Since the similarity generation unit 11 performs the search in units of predetermined pixels, the similarity in the similarity curve 1012 is shown at intervals of the pixel units of the search. The pixel unit of the search can be arbitrarily set, but the finest is 1 pixel. The search positions in the central part of the illustration are 1003, 1001, 1000, 1002, and 1004 from the left.

[0030] The similarity S0 at the search position 1000 of the similarity curve 1012 is greater than the similarities at the left and right search positions. Therefore, the search position 1000 satisfies the first determination condition described above. As an example of the correction method of the similarity curve 1012, there is a method of replacing the similarity S0 at the search position 1000 with the similarity S1. As an example of another method, there is a method of generating a new similarity S1 at the search positions around the search position 1000, for example, the search position 1003.

[0031] The search position 1003 having the similarity S1 within the similarity curve 1012 is an example of the first correction information 200. As an example of the method of generating the similarity S1, there is a method of calculating by polynomial interpolation such as linear approximation or SSD parabola fitting based on a plurality of similarities in the vicinity. As an example of a plurality of similarities existing in the vicinity, there are three similarities at the search positions 1000, 1001, and 1002. The search position 1003 is a search position with a decimal precision located between the search position 1000 and the search position 1002.

[0032] In this way, the first correction information generation unit 20 scans in the baseline direction to search for a polarity change point where the sign of the difference in similarity changes, calculates a new similarity using the similarity in the vicinity of the polarity change point, and calculates the combination of the position of the polarity change point and the new similarity as the first correction information 200.

[0033] When the search position 1003 and the similarity S1 are included in the first correction information 200, there are at least the following three methods for the correction execution unit 22 to correct the similarity using the first correction information 200. The first method is to rewrite the information of the search position 1000 and the similarity S0 with the information of the search position 1003 and the similarity S1. The second method is to keep the information of the search position 1000 and the similarity S0 as they are and add the information of the search position 1003 and the similarity S1. The third method is to keep the position of the search position 1000 unchanged and rewrite the value of the similarity S0 to the value of S1. When adopting the third method, the information of the search position 1003 may be calculated by three-point approximation or the like.

[0034] Returning to FIG. 2, the configuration of the second correction information generation unit 21 will be described. The second correction information generation unit 21 includes a peripheral parallax continuity determination unit 50 and a second generation unit 51, and generates second correction information 301 based on the parallax information 106 or the parallax boundary information 300.

[0035] Each of FIGS. 4 and 5 is a diagram showing an example of the parallax information 106 used for generating the second correction information 201 in the second correction information generation unit 21. Hereinafter, since the parallax information 106 is set corresponding to positions, for example, for each pixel block, hereinafter, the region corresponding to the parallax information 106 used by the second correction information generation unit 21 for generating the second correction information 201 is referred to as a "reference peripheral parallax region".

[0036] FIG. 4 is a diagram showing a first example of the parallax information 106 used for generating the second correction information 201. In other words, FIG. 4 is a diagram showing a first example of determining a reference peripheral parallax region. FIG. 4 shows the process of calculating the parallax of the parallax generation target block 801 in the current frame 800 which is the latest captured image. The grid in FIG. 4 is a block which is a unit for calculating the parallax. The size of the block may be 1 pixel, or may be 4 pixels with 2 pixels horizontally and 2 pixels vertically, or may be 16 pixels with 4 pixels horizontally and 4 pixels vertically, or may be a square with 5 or more pixels on one side, or may be a rectangle composed of an arbitrary number of pixels.

[0037] In the example shown in FIG. 4, the parallax of the block at the upper left in the figure is calculated first, and the processing target is sequentially changed to the right direction as indicated by the arrow. After all the parallaxes in the uppermost row are calculated, the processing target is changed to the second row, and the processing target is sequentially changed from left to right in the same manner. Therefore, when calculating the parallax of the block indicated by the reference numeral 801, the parallaxes in the first to third rows shown in the figure are all calculated, and the left side of the fourth row is also the parallax generation target block 801. The parallax has been calculated. The region indicated by the diagonal lines in FIG. 4 is a region with a side length of 5 pixels centered on the parallax generation target block 801, in which the parallax has already been calculated at the time of calculating the parallax of the parallax generation target block 801. The parallax information 106 in this region is referred to as "reference peripheral parallax 802". In the example of FIG. 4, this reference peripheral parallax 802 is used for generating the second correction information 201. That is, in the first example shown in FIG. 4, the reference peripheral parallax region is the reference peripheral parallax 802.

[0038] FIG. 5 is a diagram showing a second example of the parallax information 106 used for generating the second correction information 201. In other words, FIG. 5 is a diagram showing a second example of determining a reference peripheral parallax region. FIG. 5 shows three images with different shooting times. The current frame 800 is the latest frame, the first past frame 901 is the frame acquired in the previous processing cycle, and the second past frame 902 is the frame acquired in the processing cycle two before. The block 801 in the current frame 800 is the parallax generation target block, and the corresponding blocks in the past frames are the block 903 and the block 904.

[0039] The parallax of the hatched area in the first past frame 901 is called the "first past reference peripheral parallax 905", and the parallax of the hatched area in the second past frame 902 is called the "second past reference peripheral parallax 906". The first past reference peripheral parallax 905 and the second past reference peripheral parallax 906 are the second examples of the parallax information 106 used for generating the second correction information 201. Since these parallax information are included in past frames, they have already been calculated at the time of generating the parallax generation target block 801, and are the parallax information around the blocks 903 and 904 of the past frame corresponding to the parallax generation target block 801. That is, in the second example shown in FIG. 5, the reference peripheral parallax area is only the first past reference peripheral parallax 905, or both the first past reference peripheral parallax 905 and the second past reference peripheral parallax 906. In the example of FIG. 5, two past frames are referred to, but only one past frame or three or more past frames may be referred to.

[0040] Based on the parallax information 106, the peripheral parallax continuity determination unit 50 generates second correction necessity information 500 indicating the necessity of generating the second correction information 301. As an example of the second correction necessity information 500, there are peripheral parallax information and correction necessity information.

[0041] As an example of the peripheral parallax information, there is an average value of the parallax values in the reference peripheral parallax area. As an example of the calculation method of this parallax value, there are the average parallax value of the reference peripheral parallax 802 shown in FIG. 4, the average parallax value of the first past reference peripheral parallax 905 shown in FIG. 5, and the average parallax value of the first past reference peripheral parallax 905 and the second past reference peripheral parallax 906 shown in FIG. 5. Also, only the parallax value in the horizontal direction may be referred to, or only the parallax value in the vertical direction may be referred to. Further, the parallax isolated within the reference peripheral parallax area may be excluded from the reference target parallax. As an example of the detection method of the isolated parallax, there is a method of determining that the parallax is isolated when the difference between the value of the detection target parallax and the value of the parallax around the detection target parallax is larger than a predetermined value.

[0042] The second correction necessity information 500 is information indicating whether the second correction information 301 is generated in the second generation unit 51. When the second correction necessity information 500 is "Generation: Required", the correction execution unit 22 performs similarity correction using the second correction information 301. When the second correction information 301 is "Generation: Not Required", the correction execution unit 22 does not perform similarity correction using the second correction information 301.

[0043] For example, the peripheral parallax continuity determination unit 50 calculates the degree of variation of the parallax within the reference peripheral parallax region, and when the degree of variation is large, sets the second correction necessity information 500 to "Generation: Not Required". Examples of the degree of variation include the sum of differences between the average of the parallax values within the reference peripheral parallax region and each parallax value, and the variance value within the reference peripheral parallax region. For generating the degree of variation, only the parallax values in the horizontal direction may be referred to, or only the parallax values in the vertical direction may be referred to. As another example of the method for generating the correction necessity information, the continuity of the parallax within the reference peripheral parallax region is calculated, and when the continuity is small, the correction necessity information may be set to no correction. Examples of the continuity include the average and variance values of the difference values between the values of each parallax within the reference peripheral parallax region. For generating the continuity, only the parallax values in the horizontal direction may be referred to, or only the parallax values in the vertical direction may be referred to.

[0044] The second generation unit 51 generates the second correction information 301 based on the second correction necessity information 500. An example of the second correction information 301 is a correction coefficient K corresponding to the search position. Referring to FIG. 6, the role of this correction coefficient K will be described. The calculation method of the correction coefficient K will be described later.

[0045] FIG. 6 is a diagram for explaining the correction coefficient K. The graph 1110 shown at the upper part of FIG. 6 is an example of the second correction information 301 and is a set of correction coefficients K for each search position. The graph 1111 shown at the lower part of FIG. 6 shows the similarity for each search position and shows information of the same type as FIG. 3. However, in FIG. 6, the similarity before correction is shown by a solid line, and the correction coefficient after correction is shown by a dashed line. The upper and lower parts of FIG. 6 will be described in order.

[0046] In the upper part of FIG. 6, the horizontal axis of graph 1110 represents the search position within the search range, and the vertical axis represents the correction coefficient K for the first similarity information 104. The correction coefficient K0 is the value when the first similarity information 104 is not corrected. When the correction coefficient is smaller than the correction coefficient K0, the first similarity information 104 is corrected so that the similarity becomes higher. Also, when the correction coefficient is larger than the correction coefficient K0, the first similarity information 104 is corrected so that the similarity becomes smaller.

[0047] When the correction coefficient K is used by multiplying it with the first similarity information 104 before correction, for example, the correction coefficient K0 is "1", the correction coefficient K1 is a real number smaller than "1" and larger than "0", and the correction coefficient K2 is a real number larger than "1". When the correction coefficient K is used by multiplying it with the first similarity information 104 before correction, for example, the correction coefficient K0 is "1", the correction coefficient K1 is a real number smaller than "1" and larger than "0", and the correction coefficient K2 is a real number larger than "1". When the correction coefficient K is used by adding it to the first similarity information 104 before correction, for example, the correction coefficient K0 is "0", the correction coefficient K1 is a positive real number, and the correction coefficient K2 is a negative real number.

[0048] In the lower part of FIG. 6, the horizontal axis of graph 1111 represents the search position within the search range, and the vertical axis represents the similarity. The first similarity curve 1104 shown by the solid line is the similarity curve created using the first similarity information 104 as it is. The second similarity curve 1105 shown by the dashed line is the similarity curve created using the second similarity information 105 generated based on the second correction information 1100 and the first similarity information 104.

[0049] As shown in graph 1110, the correction coefficient K at the search position A1 is "K0" as shown in the element 1101 of the second correction information 301. Therefore, as shown in the element 1106 on the second similarity curve 1105, the values of the first similarity curve 1104 and the second similarity curve 1105 are the same at the search position A1.

[0050] The correction coefficient K at the search position A2 is "K1", which is a value smaller than K0 as shown in the element 1102 of the second correction information 301. Therefore, as shown in the element 1107 on the second similarity curve 1105, the second similarity is higher than the first similarity at the search position A2. The correction coefficient at the search position A3 is "K2", which is a value larger than K0 as shown in the element 1103 of the second correction information 301. Therefore, as shown in the element 1108 on the second similarity curve 1105, the second similarity is lower than the first similarity at the search position A3. The correction of the first similarity information 104 using the correction coefficient K is as follows, for example.

[0051] Sa(Ai) = Sb(Ai)×K(i)

[0052] Here, Sb(i) is the similarity at the search position (i) on the first similarity curve 1104, Sa(i) is the similarity at the search position (i) on the second similarity curve 1105, and K(i) is the correction coefficient K on the second correction information 1100 at the search position (i). In this case, the similarity at the search position A2 in FIG. 6 has the following relationship.

[0053] S1 = S0×K1

[0054] The correction execution unit 22 reads the correction coefficient K for each search position with reference to the second correction information 301, and corrects the first similarity by multiplying it with the corresponding similarity as in the above formula. Note that in the correction execution unit 22, the correction of the similarity using the second correction information 301 is performed after the correction of the similarity using the first correction information 200.

[0055] Referring to FIG. 7, the generation of the second correction information 301 by the second generation unit 51, specifically, a specific example of the generation method of the correction coefficient K corresponding to the search position will be described. FIG. 7(a) is a diagram showing a first example of calculating the correction coefficient K. The first row shows the parallax position, the second row shows the parallax value, the third row shows the continuity, the fourth row shows the function, and the fifth row shows the correction coefficient K. Here, if the difference from the parallax value adjacent to the left is less than or equal to a predetermined threshold value of "2", it is determined as continuous and the continuity is increased by "1", and if the difference is greater than or equal to the threshold value, the continuity is decreased by "1". The upper limit of the continuity is set to "3".

[0056] Also, the function means "0" for no correction, "1" for weak correction, and "2" for strong correction. When the continuity is "0" and "1", the function is set to "0", when the continuity is "2", the function is set to "1", and when the continuity is "3", the function is set to "2". The correction coefficient K is described numerically differently from FIG. 6 for convenience of drawing. "5" corresponds to "K0" in FIG. 6, "3" corresponds to "K1" in FIG. 6, and "4" corresponds to the value between "K1" and "K0" in FIG. 6. The correction coefficient is set according to the value of the function. When the function is "0", the correction coefficient K is set to "5", when the function is "1", the correction coefficient K is set to "4", and when the function is "2", the correction coefficient K is set to "3".

[0057] In the example of FIG. 7(a), the parallax values "8" are continuous at the parallax positions A to G. Therefore, the continuity increases by 1 from the initial value of "0" to the maximum value of "3". After the parallax position H, the parallax value switches to "2". Since the difference from "8" is greater than the threshold value of "2", it decreases by "1" at the parallax positions H, I, and J, and the continuity becomes "0" at the position J. After that, since the parallax value "2" continues, the continuity turns to increase as meaning the continuity of "2" and continues to the maximum value of "3". The function is linked to the continuity, and the correction coefficient K is linked to the function. Therefore, in FIG. 7(a) as well, the value of the function varies according to the continuity, and the value of the correction coefficient K varies according to the value of the function.

[0058] FIG. 7(b) is a diagram showing a second example of calculating the correction coefficient K. The first row shows the parallax position, the second row shows the parallax value, the third row shows the continuity degree A, the fourth row shows the continuity degree B, the fifth row shows the maximum continuity degree, the sixth row shows the function, and the seventh row shows the correction coefficient K. The continuity degree A and the continuity degree B count the continuous states of different parallax values. The maximum continuity degree is the larger value of the continuity degree A and the continuity degree B. The function is set in conjunction with the maximum continuity degree in this example.

[0059] In the example of FIG. 7(b), the parallax values "8" are continuous at the parallax positions A to G, the parallax values "2" are continuous at the parallax positions H to M, and the parallax values "5" are continuous at the parallax positions N to Q. Therefore, at the parallax positions A to G, the continuity degree A increases in the same manner as the continuity degree in FIG. 7(a) and maintains the maximum value of "3". At this time, since there is only one type of parallax value and the continuity degree B is not used yet, an asterisk indicating no value is described.

[0060] When the parallax value switches to "2" at the parallax position H, since the difference from the previous parallax value "8" is larger than the threshold value "2", the continuity degree A decreases and becomes the lower limit "0" in this example. The continuity degree B becomes the initial value "0" at the parallax position H, and then increases as the parallax value "2" is continuous, and becomes the maximum value "3" at the parallax position K. Thereafter, when the parallax value changes to "5" at the parallax position N, the continuity degree B decreases, and the continuity degree A increases in an alternating manner. Since the maximum continuity degree is the maximum value of the continuity degree A and the continuity degree B as described above, the maximum continuity degree is the same as the continuity degree A until the parallax position G where the value of the continuity degree B does not exist, and the larger value of the continuity degree B becomes the maximum continuity degree from the parallax position J to N. The value of the function is determined according to the maximum continuity degree, and the value of the correction coefficient K is determined according to the value of the function.

[0061] In the example shown in FIG. 7, the correction coefficient K only takes values corresponding to K0 to K1, but by changing the relationship between the function and the correction coefficient K as follows, the correction coefficient K may be changed to take values of K0 to K2. For example, when the function is "0", the correction coefficient K is "7", that is, set to K2, when the function is "1", the correction coefficient K is "5", that is, set to K0, and when the function is "2", the correction coefficient K is "3", that is, set to K1.

[0062] According to the above-described first embodiment, the following operational effects can be obtained. (1) The arithmetic unit 1 is connected to the first imaging unit 2 and the second imaging unit 3, and generates parallax information between the first image 100 captured by the first imaging unit 2 and the second image 101 captured by the second imaging unit 3 by searching in units of predetermined matching blocks. The arithmetic unit 1 includes a similarity generation unit 11 that generates a first similarity between a matching block in the first image 100 and a matching block within the search range of the second image 101 in units of matching blocks, a first correction information 200 generated based on the first similarity of any adjacent matching blocks within the search range of the second image 101, and a similarity correction unit 12 that corrects the first similarity information 104 using at least one of the second correction information 301 generated based on the parallax information 106 to generate second similarity information 105, and a similarity determination unit 13 that generates parallax information 106 based on the second similarity information 105. Therefore, by correcting the similarity using the first correction information 200 and the second correction information 301, false matching can be prevented and the calculation accuracy of the parallax can be improved.

[0063] (2) The first imaging unit 2 and the second imaging unit 3 generate the first image 100 and the second image 101 at every predetermined processing cycle. The arithmetic unit 1 includes a second correction information generation unit 21 that generates second correction information, which is a correction coefficient K multiplied by the similarity based on the parallax information 106 within the peripheral region of the matching block in the latest first image 100 and the latest second image 101, or the first image 100 and the second image 101 from a predetermined cycle before, as shown in FIGS. 4 and 5. Therefore, when using the latest captured image, a correction coefficient K based on parallax information 106 without positional deviation can be calculated, and when using a past captured image, a correction coefficient can be calculated using the parallax information of the entire frame.

[0064] (3) The second correction information 301 includes a correction coefficient for correcting so as to increase the first similarity corresponding to the disparity information in the peripheral region when the degree of continuity of the disparity information in the peripheral region is equal to or greater than a threshold value, as in the search position A2 in FIG. 6. Therefore, by increasing the similarity of the matching blocks where the surroundings and the disparity are similar and are expected to be similar, false matching can be prevented and the calculation accuracy of the disparity can be improved.

[0065] (4) The second correction information 301 includes a correction coefficient for correcting so as to decrease the first similarity corresponding to the disparity information in the peripheral disparity when the degree of continuity of the disparity information in the peripheral region is equal to or less than a threshold value, as in the search position A3 in FIG. 6. Therefore, by decreasing the similarity of the matching blocks where the surroundings and the disparity are not similar, it becomes difficult to match, false matching can be prevented, and the calculation accuracy of the disparity can be improved.

[0066] (5) The correction execution unit 22 of the similarity correction unit 12 generates a second similarity by further correcting the first similarity information 104 based on the second correction information 301 after correcting the first similarity information 104 based on the first correction information 200. Therefore, the arithmetic unit 1 can utilize the result of the first correction that performs local similarity correction for the second correction that performs wide-area similarity correction.

[0067] (6) The first imaging unit 2 and the second imaging unit 3 are arranged in the baseline direction. The first correction information generation unit 20 is provided, which scans in the baseline direction to search for a polarity change point where the positive / negative of the difference in similarity switches, calculates a new similarity using the similarity around the polarity change point, and calculates the combination of the position of the polarity change point and the new similarity as the first correction information 200. Therefore, local similarity correction can be performed more finely than the unit of similarity calculation. For example, if the similarity is calculated for each pixel, since the search position 1003 has a fractional coordinate value, it can be said that the similarity can be calculated with sub-pixel accuracy.

[0068] (7) The monitoring system S includes the arithmetic unit 1, the first imaging unit 2, and the second imaging unit 3. Therefore, the monitoring system S has few false matches and good calculation accuracy of the disparity.

[0069] (Modification Example 1) In the above-described first embodiment, the similarity correction unit 12 of the arithmetic unit 1 includes the first correction information generation unit 20 and the second correction information generation unit 21. However, the similarity correction unit 12 may include only one of the first correction information generation unit 20 and the second correction information generation unit 21.

[0070] (Modification Example 2) In the above-described first embodiment, since the first imaging unit 2 and the second imaging unit 3 are arranged side by side horizontally, the baseline direction is horizontal, and the first generation unit 41 scanned the similarity in the horizontal direction as shown in FIG. 3. However, the first imaging unit 2 and the second imaging unit 3 may be arranged vertically or diagonally. In this case, the first generation unit 41 may scan the similarity along the baseline direction, that is, along the epipolar line.

[0071] - Second Embodiment - Referring to FIG. 8, a second embodiment of the monitoring system will be described. In the following description, the same components as those in the first embodiment are denoted by the same reference numerals, and the differences will be mainly described. For points not particularly described, they are the same as those in the first embodiment. In this embodiment, it is mainly different from the first embodiment in that it includes a correction information invalidation unit.

[0072] FIG. 8 is a configuration diagram of the correction execution unit 22A in the second embodiment. The correction execution unit 22A includes a correction information invalidation unit 60 and a correction unit 61. The correction unit 61 includes a first correction unit 70 and a second correction unit 71.

[0073] In the correction execution unit 22 in the first embodiment, the first similarity information 104, the first correction information 200, and the second correction information 301 are input. However, in the correction execution unit 22A in this embodiment, in addition to the above three, an invalidation instruction 602 is further input. The correction execution unit 22A performs correction processing on the first similarity information 104 based on the invalidation instruction 602, the first correction information 200, and the second correction information 301, and generates the second similarity information 105.

[0074] In the correction information invalidation unit 60, based on the invalidation instruction 602, at least one of the first correction information 200 and the second correction information 301 can be invalidated. For example, the correction information invalidation unit 60 may invalidate both the first correction information 200 and the second correction information 301, may invalidate only one of them, or may not invalidate either of them. As an example of the invalidation instruction 602, there is information indicating the validity or invalidity of the first correction information 200, and information indicating the validity or invalidity of the second correction information 301. That is, the correction information invalidation unit 60 accepts the first correction information 200 output by the first correction information generation unit 20 and the second correction information 301 output by the second correction information generation unit 31 as the second valid correction information 601 as well.

[0075] When the correction information invalidation unit 60 receives an invalidation instruction 602 indicating that the first correction information 200 is valid and the second correction information 301 is invalid, for example, it performs the following processing. That is, the correction information invalidation unit 60 outputs the first correction information 200 as the first valid correction information 600, and outputs the second valid correction information 601 as information in which the second correction information 301 is invalidated.

[0076] The correction unit 61 performs correction processing on the first similarity information 104 based on the first valid correction information 600 and the second valid correction information 601, and generates the second similarity information 105. As an example of the correction processing for the first valid correction information 600, there is a method of generating the second similarity information 105 by replacing the similarity at the search position to be corrected or inserting the similarity at a new search position using the first correction information 200. As an example of the correction processing for the second valid correction information 601, there is the method of generating the second similarity curve 1105 described above with reference to FIG. 6. Also, in the first valid correction information 600, when the first correction information 200 is invalidated, the correction processing using the first correction information 200 is not performed. In the second valid correction information 601, when the first correction information 200 is invalidated, the correction processing using the first correction information 200 is not performed.

[0077] The first correction unit 70 performs correction processing on the first similarity information 104 based on the first valid correction information 600, and generates intermediate similarity information 700. The second correction unit 71 performs correction processing on the intermediate similarity information 700 based on the second valid correction information 601, and generates second similarity information 105. Since the correction processing in the first correction unit 70 and the correction processing in the second correction unit 71 are as described with reference to FIGS. 3 and 6, the details are omitted.

[0078] According to the second embodiment described above, appropriate corrections can be used properly.

[0079] - Third Embodiment - Referring to FIG. 9, a third embodiment of the monitoring system will be described. In the following description, the same components as those in the first embodiment are denoted by the same reference numerals, and the differences will be mainly described. For points not particularly described, they are the same as those in the first embodiment. In this embodiment, it is mainly different from the first embodiment in that it does not detect the boundary of the disparity to generate the second correction information.

[0080] FIG. 9 is a configuration diagram of the similarity correction unit 12B in the third embodiment. The similarity correction unit 12B includes a second correction information generation unit 21B instead of the second correction information generation unit 21, and newly includes a disparity boundary detection unit 30. The disparity boundary detection unit 30 refers to the calculated disparity information 106, and when detecting the boundary of the disparity in the matching block, outputs the disparity boundary information 300 to the second correction information generation unit 21B. For the presence or absence of the disparity boundary in the matching block, for example, any of the following three methods can be used.

[0081] The first method is to individually check the disparity of each block within the matching block in the current frame. Specifically, the disparity of each block within the matching block is compared with the disparities of other blocks on the left and right and above and below, and when the difference is equal to or greater than a predetermined threshold, it is determined that there is a disparity boundary. For any block, when the difference from the disparities of other blocks on the left and right and above and below is less than the predetermined threshold, it is determined that there is no disparity boundary. In this first method, since the current frame is used, the latest information is utilized, but the difference from a block for which the disparity has not yet been calculated cannot be evaluated.

[0082] The second method is to individually check the disparity of each block within the matching block in the past frame obtained immediately before. The difference from the first method is the target frame. Since all blocks in the past frame have had their disparities calculated, the second method can calculate the difference in disparities for blocks for which the difference in disparities could not be calculated in the first method. This second method embodies the idea that it is assumed that there is no significant change in the position of the subject within the time of one frame, and the benefit of being able to calculate the difference in disparities for all blocks outweighs the loss due to the change in the position of the subject.

[0083] The third method is a method for detecting the disparity boundary by hierarchical search. This method may be applied to the current frame as in the first method or to the past frame as in the second method. In hierarchical search, first, at the first level, the disparity is calculated for a block larger than the matching block, for example, a block with both vertical and horizontal sizes twice that of the matching block. Hereinafter, this block will be referred to as a large-size block.

[0084] Next, at the second layer, the disparity of each block within the matching block is calculated based on the disparity of the large-size block. Specifically, the sum of the disparities of three blocks, namely, the disparity of the large-size block containing the block for which the disparity is to be calculated, and the disparities of the left and right large-size blocks adjacent to that large-size block, are used as disparity candidates. The disparity that is within a predetermined range and most likely among the disparity candidates is set as the disparity of the block to be calculated. Using the disparity calculated through hierarchical search in this manner, it is determined whether the difference from the disparities of other blocks in the horizontal and vertical directions is equal to or greater than a predetermined threshold to determine the presence or absence of a disparity boundary.

[0085] When the peripheral disparity continuity determination unit 50 that has received c from the disparity boundary detection unit 30 determines that there is a disparity boundary within the matching block, it creates second correction necessity information 500 indicating that generation of the second correction information 301 is unnecessary and transmits it to the second generation unit 51. Since the second generation unit 51 that has received this second correction necessity information 500 does not generate the second correction information 301, correction of the first similarity information 104 based on the second correction information 301 by the correction execution unit 22 is not performed either.

[0086] According to the third embodiment described above, the following operational effects can be obtained. (8) The arithmetic unit 1 includes a disparity boundary detection unit 30 that generates disparity boundary information 300 based on the luminance information or disparity information of the first image 100 or the second image 101 within the peripheral region. The second correction information generation unit 21 invalidates the generation of the second correction information 201 based on the disparity boundary information 300. Therefore, when there is a disparity boundary in the surroundings, correction of the similarity by the second correction information 301 is not performed, and the adverse effect of the correction of the similarity by the second correction information 301 can be prevented.

[0087] (Modification Example of the Third Embodiment) In the above-described third embodiment, the disparity boundary detection unit 30 determined the presence or absence of a disparity boundary using the disparity information 106. However, the disparity boundary detection unit 30 may simply determine the presence or absence of a disparity boundary by using the presence or absence of luminance changes using the first image 102 or the second image 103. The disparity boundary detection unit 30 refers to the first image 102 or the second image 103 and compares the luminance of each block in the matching block with the luminance of other blocks on the left and right and above and below. When the difference is equal to or greater than a predetermined threshold, it is determined that there is a disparity boundary. For any block, if the difference from the luminance of other blocks on the left and right and above and below is less than the predetermined threshold, it is determined that there is no luminance boundary.

[0088] In each of the above-described embodiments and modifications, the configuration of the functional blocks is merely an example. Some functional configurations shown as separate functional blocks may be integrated, or the configuration represented by one functional block diagram may be divided into two or more functions. Also, a configuration may be adopted in which a part of the functions of each functional block is provided by other functional blocks.

[0089] In each of the above-described embodiments and modifications, it was assumed that the program was stored in a ROM (not shown), but the program may be stored in a non-volatile storage device (not shown), such as a flash memory. Further, the arithmetic unit may include an input / output interface (not shown), and when necessary, the program may be read from another device via a medium that can be used by the input / output interface and the arithmetic unit. Here, the medium refers to, for example, a storage medium detachable from the input / output interface, or a communication medium, that is, a wired, wireless, optical, or other network, or a carrier wave or digital signal propagating through the network. Also, part or all of the functions realized by the program may be realized by a hardware circuit or an FPGA.

[0090] Each of the above-described embodiments and modifications may be combined. Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

Explanation of Reference Numerals

[0091] S… Surveillance system 1… Arithmetic unit 10… Input unit 11… Similarity generation unit 12, 12B… Similarity correction unit 13… Similarity determination unit 20… First correction information generation unit 21, 21B… Second correction information generation unit 22, 22A… Correction execution unit 30… Parallax boundary detection unit 31… Second correction information generation unit 40… Correction determination unit 41… First generation unit 50… Peripheral parallax continuity determination unit 51… Second generation unit 60… Correction information invalidation unit 61… Correction unit 70… First correction unit 71… Second correction unit

Claims

1. An arithmetic unit connected to a first imaging unit and a second imaging unit, which generates disparity information between a first image captured by the first imaging unit and a second image captured by the second imaging unit by searching in units of predetermined matching blocks, a similarity generation unit that generates a first similarity in units of the matching blocks based on the difference in luminance values between corresponding pixels in a matching block in the first image and a matching block within a search range of the second image; a similarity correction unit that corrects the first similarity using at least one of first correction information generated based on the first similarity of any adjacent matching blocks within the search range of the second image and second correction information generated based on the disparity information to generate a second similarity; a similarity determination unit that generates the disparity information based on the second similarity, and an arithmetic unit comprising the same.

2. The arithmetic unit according to claim 1, wherein the first imaging unit and the second imaging unit generate the first image and the second image at every predetermined processing cycle, and further comprises a second correction information generation unit that generates the second correction information, which is a correction coefficient multiplied by the first similarity based on the disparity information within a peripheral region of the matching block, in the latest first image and the latest second image, or the first image and the second image before a predetermined period.

3. The arithmetic unit according to claim 2, wherein the second correction information includes a correction coefficient that corrects the first similarity corresponding to the disparity information within the peripheral region to increase it when the continuity of the disparity information within the peripheral region is equal to or greater than a threshold value.

4. The arithmetic unit according to claim 2, wherein the second correction information includes a correction coefficient that corrects the first similarity corresponding to the disparity information within the peripheral region to decrease it when the continuity of the disparity information within the peripheral region is less than a threshold value.

5. The arithmetic unit according to claim 2, further comprising a disparity boundary detection unit that generates disparity boundary information based on luminance information or the disparity information of the first image within the peripheral region, wherein the second correction information generation unit invalidates the generation of the second correction information based on the disparity boundary information.

6. The arithmetic unit according to claim 1, The similarity correction unit generates the second similarity by correcting the first similarity based on the first correction information and then further correcting the first similarity based on the second correction information.

7. The arithmetic unit according to claim 1, wherein the first imaging unit and the second imaging unit are arranged in the baseline direction, search for a switching point where the sign of the difference in similarity switches by scanning in the baseline direction, calculate a new similarity using the similarity around the switching point, and calculate a combination of the position of the switching point and the new similarity as first correction information. The arithmetic unit further includes a first correction information generation unit.

8. The arithmetic unit according to claim 1, the first imaging unit, and the second imaging unit, and a monitoring system comprising the same.

9. A parallax calculation method executed by an arithmetic unit that is connected to a first imaging unit and a second imaging unit and generates parallax information between a first image captured by the first imaging unit and a second image captured by the second imaging unit by searching in units of predetermined matching blocks, generating a first similarity in units of the matching blocks based on the difference in luminance values between corresponding pixels in the matching block in the first image and the matching block within the search range of the second image; correcting the first similarity using at least one of first correction information generated based on the first similarity of any adjacent matching blocks within the search range of the second image and second correction information generated based on the parallax information to generate a second similarity; and generating the parallax information based on the second similarity.

Images