Camera calibration support device, camera calibration support method, and program

Abstract

To determine the quality of a variable value estimated in camera calibration.SOLUTION: A camera calibration support device includes an acquisition unit configured to acquire a plurality of variable values representing positions and orientations of a plurality of cameras installed in a vehicle and estimated on the basis of a plurality of images obtained by imaging a predetermined imaging target serving as a reference by the plurality of cameras having different imaging directions, image information representing a plurality of images obtained by imaging a worker positioned in the vicinity of the vehicle by the plurality of cameras, identification information for identifying the worker, a synthesis unit configured to synthesize a plurality of images based on the image information into a bird's-eye view image representing the vicinity of the vehicle, a determination unit configured to determine whether or not the plurality of variable values are good based on the bird's-eye view image, and a determination result output unit configured to output a determination result of whether or not the plurality of variables are good.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a camera calibration support device, a camera calibration support method, and a program. [Background technology]

[0002] Patent Document 1 discloses a calibration method for calculating camera parameters based on the positional deviations between multiple markers obtained as a result of generating an overhead image on a road surface or a plane parallel to the road surface, reflecting the height of each marker, in camera calibration for generating an overhead (bird's-eye) image of the area around a vehicle. Note that in the synthesis of overhead images described in Patent Document 1, the images are aligned by translating and rotating them so that identical markers in different images overlap, and then the images are synthesized.

[0003] Camera calibration (or simply referred to as calibration) is a process of estimating the parameter values ​​of a camera model (hereinafter, these parameter values ​​are also referred to as variable values) from an image captured by a camera of a reference object whose coordinates are known (for example, a calibration board or white line). The camera model parameters include external parameters that define the position and orientation of the camera, and internal parameters that define the camera's unique characteristics (focal length, pixel size, image origin, optical axis position, etc.). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6009894 Summary of the Invention [Problem to be solved by the invention]

[0005] The calibration method described in Patent Document 1 has a problem in that, for example, in cases where there is a deviation in the height direction between multiple markers, or in cases where multiple markers can be captured with difficulty but the position or orientation of the camera deviates significantly from the design values, it is not possible to correctly synthesize an overhead image and therefore to correctly estimate camera parameters.

[0006] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a camera calibration support device, a camera calibration support method, and a program that can determine the quality of variable values ​​estimated in camera calibration. [Means for solving the problem]

[0007] In order to solve the above problem, the camera calibration support device according to the present disclosure includes an acquisition unit that acquires a plurality of variable values ​​that represent the position and attitude of each camera, estimated based on a plurality of images of a predetermined target that serves as a reference, captured by a plurality of cameras with different imaging directions that are installed on a vehicle; image information that represents a plurality of images of a worker positioned around the vehicle, captured by the plurality of cameras; and identification information that identifies the position of the worker; a synthesis unit that synthesizes the plurality of images based on the image information into an overhead image that represents the periphery of the vehicle, based on the plurality of variable values; a recognition unit that recognizes the worker from the overhead image; a judgment unit that judges whether the plurality of variable values ​​are good or bad, based on the recognition result of the worker and the identification information; and a judgment result output unit that outputs the judgment result of whether the plurality of variable values ​​are good or bad.

[0008] The camera calibration support method according to the present disclosure includes the steps of acquiring a plurality of variable values ​​representing the position and attitude of each camera, estimated based on a plurality of images of a predetermined target image serving as a reference captured by a plurality of cameras mounted on a vehicle with different imaging directions, image information representing a plurality of images of a worker positioned in the vicinity of the vehicle captured by the plurality of cameras, and identification information identifying the position of the worker, combining the plurality of images based on the image information into an overhead image representing the vicinity of the vehicle based on the plurality of variable values, recognizing the worker from the overhead image, determining whether the plurality of variable values ​​are good or bad based on the recognition result of the worker and the identification information, and outputting the determination result of whether the plurality of variable values ​​are good or bad.

[0009] The program according to the present disclosure causes a computer to execute the following steps: acquiring a plurality of variable values ​​representing the position and attitude of each camera estimated based on a plurality of images of a predetermined reference target captured by a plurality of cameras with different imaging directions installed on a vehicle; image information representing a plurality of images of a worker positioned in the vicinity of the vehicle captured by the plurality of cameras; and identification information identifying the position of the worker; combining the plurality of images based on the image information into an overhead image representing the vicinity of the vehicle based on the plurality of variable values; recognizing the worker from the overhead image; judging whether the plurality of variable values ​​are good or bad based on the recognition result of the worker and the identification information; and outputting the judgment result of whether the plurality of variable values ​​are good or bad. [Effects of the Invention]

[0010] According to the camera calibration support device, camera calibration support method, and program of the present disclosure, it is possible to determine whether the variable values ​​estimated in camera calibration are good or bad. Therefore, if the result of the determination is that the variable values ​​are bad, the camera parameters can be correctly estimated by performing camera calibration again after adjusting the attitude and position of the markers and camera. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration of an information processing system according to a first embodiment of the present disclosure. [Figure 2] 1 is a side view of an industrial vehicle according to a first embodiment of the present disclosure. [Figure 3] 1 is a plan view schematically showing an industrial vehicle and an imaging range of a camera according to a first embodiment of the present disclosure. FIG. [Figure 4] 3 is a schematic diagram showing a configuration example of specific information according to the first embodiment of the present disclosure. FIG. [Figure 5] FIG. 2 is a schematic diagram showing a plan view included in specific information according to the first embodiment of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram showing an overhead image according to the first embodiment of the present disclosure. [Figure 7] FIG. 3 is a schematic diagram showing an output example according to the first embodiment of the present disclosure. [Figure 8] FIG. 10 is a schematic diagram showing another output example according to the first embodiment of the present disclosure. [Figure 9] 4 is a flowchart showing an operation example of the information processing system according to the first embodiment of the present disclosure. [Figure 10] FIG. 10 is a schematic diagram showing an output example according to the second embodiment of the present disclosure. [Figure 11] FIG. 10 is a block diagram showing a schematic configuration of an information processing system according to a third embodiment of the present disclosure. [Figure 12] FIG. 11 is a schematic diagram showing a configuration example of specific information according to a third embodiment of the present disclosure. [Figure 13] FIG. 10 is a block diagram showing a schematic configuration of an information processing system according to a fourth embodiment of the present disclosure. [Figure 14] 10 is a flowchart illustrating an example of operation of the information processing device according to the fourth embodiment of the present disclosure. [Figure 15] FIG. 11 is a schematic diagram showing an example of a three-dimensional model generated by a generation unit according to a fourth embodiment of the present disclosure. [Figure 16] FIG. 10 is a schematic diagram showing an output example according to the fourth embodiment of the present disclosure. [Figure 17]FIG. 13 is a schematic diagram showing another output example according to the fourth embodiment of the present disclosure. [Figure 18] FIG. 13 is a schematic diagram showing another output example according to the fourth embodiment of the present disclosure. [Figure 19] FIG. 1 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, a camera calibration support device, a camera calibration support method, and a program according to an embodiment of the present disclosure will be described with reference to Figures 1 to 19. Note that the same or corresponding components in each figure are designated by the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0013] First Embodiment FIG. 1 is a block diagram showing a schematic configuration of an information processing system 10 according to a first embodiment of the present disclosure. The information processing system 10 is a system that determines whether variable values ​​(camera parameters) estimated in camera calibration are good or bad. In the embodiment of the present disclosure, the information processing system 10 determines whether extrinsic parameters, among the camera parameters, are good or bad. That is, in the present embodiment, the camera parameters that are the subject of the determination of whether they are good or bad are extrinsic parameters. The extrinsic parameters are parameters that represent the rotation and translation of the camera in three-dimensional space, and are parameters that represent the position (three-dimensional position) and attitude (camera line of sight (roll angle, pitch angle, and yaw angle)) of the camera. Hereinafter, in the present disclosure, the camera parameters are also referred to as multiple variable values.

[0014] The information processing system 10 shown in Fig. 1 includes an information processing device 1, three cameras 2a to 2c, a mobile terminal 3, an input device 4, and a display device 5. The three cameras 2a to 2c are installed on the left, right, and rear of an industrial vehicle 7, as shown in Figs. 2 and 3. Fig. 2 is a side view of the industrial vehicle 7 according to the first embodiment of the present disclosure. Fig. 3 is a plan view schematically showing the industrial vehicle 7 according to the first embodiment of the present disclosure and the imaging ranges of the cameras 2a to 2c.

[0015] The industrial vehicle 7 shown in Figures 2 and 3 is a forklift (hereinafter also referred to as a forklift 7). Industrial vehicles are vehicles used to transport cargo within a yard or the like, and include forklifts, yard transport vehicles, yard towing vehicles, etc. This embodiment is more effective in industrial vehicles where there is a relatively high possibility that a worker will be located near the vehicle, but the vehicle to which this embodiment is applicable can be any vehicle in general and is not limited to industrial vehicles.

[0016] 2, the forklift 7 includes a vehicle body 71, a front tire 72, a rear tire 73, forks 74, a head guard 75, and a seat 76. The forklift 7 also includes a display device 5 at a position facing the operator sitting on the seat 76.

[0017] Camera 2a is installed on the left side of head guard 75 as viewed in FIG. 3 so as to capture an image of imaging range A-2a shown in FIG. 3. Camera 2b is installed on the right side of head guard 75 as viewed in FIG. 3 so as to capture an image of imaging range A-2b. Camera 2c is installed on vehicle body 71 so as to capture an image of imaging range A-2c. In this embodiment, cameras 2a to 2c are equipped with wide-angle lenses with an angle of view of, for example, approximately 180 degrees and are cameras that capture moving images. Image signals representing the moving images captured by cameras 2a to 2c are output to information processing device 1. Note that the number of cameras 2a to 2c is not limited to three. For example, two or more cameras may be used. The positions and orientations of cameras 2a to 2c are fixed before camera calibration is performed, and are then adjusted and re-fixed if readjustment is required after camera calibration is performed.

[0018] The mobile terminal 3 is a portable information processing device such as a smartphone or a tablet terminal, and is carried by, for example, a worker 6 (FIG. 7). The mobile terminal 3 transmits and receives predetermined information to and from the information processing device 1 via a communication line such as short-range wireless communication. However, the information processing system 10 does not necessarily have to include the mobile terminal 3.

[0019] The input device 4 includes, for example, an operating device such as a numeric keypad, a push button switch, etc., and outputs information input by an operator to the information processing device 1.

[0020] The display device 5 is a display that displays moving images, still images, characters, etc. output by the information processing device 1. The display device 5 and the input device 4 may be integrated into one unit, for example, as a touch panel or the like.

[0021] The information processing device 1 is one example of the configuration of the "camera calibration support device" of the present disclosure, and can be configured using a computer such as a personal computer. The information processing device 1 includes the following functional blocks configured by a combination of hardware such as a computer and peripheral devices, and software such as a program executed by the computer. That is, the information processing device 1 includes, as functional blocks, a captured image acquisition unit 101, an estimation unit 102, an acquisition unit 103, a synthesis unit 104, a recognition unit 105, a determination unit 106, a determination result output unit 107, and a storage unit 108.

[0022] The captured image acquiring unit 101 receives output signals from the cameras 2a to 2c and updates the captured images representing the moving images captured by the cameras 2a to 2c over a certain period of time, for example, while storing the images as captured images 108A in the storage unit 108. The captured image acquiring unit 101 also receives output signals from the cameras 2a to 2c from the time an operator or the like issues an instruction to start capturing images through the input device 4 until the time an instruction to end capturing images is issued, and stores the captured images representing the moving images captured by the cameras 2a to 2c over the period from the start to the end of capturing images as image information 108C in the storage unit 108. The image information 108C is information used by the estimation unit 102 when estimating multiple variable values ​​(information representing multiple images of a predetermined target to be captured as a reference), and is also information used by the composition unit 104 when combining the captured images of the worker 6 ( FIG. 7 ) positioned around the industrial vehicle 7 captured by the three cameras 2a to 2c into a bird's-eye view image representing the periphery of the industrial vehicle 7. In this embodiment, the "surroundings" of the vehicle refers to, for example, a part or all of the space surrounding the vehicle. In this case, "nearby" refers to a space within a predetermined distance from the vehicle, for example, within a range in which the camera 2a to 2c can capture an image of the worker 6 at a size that can be recognized by the camera 2a to 2c. In this embodiment, the "surroundings" may be read as "surroundings."

[0023] The estimation unit 102 estimates multiple variable values ​​(external parameters) representing the position and orientation of each camera based on multiple images of a predetermined reference object captured by multiple cameras 2a to 2c installed on the industrial vehicle 7 and with different imaging directions. Alternatively, the estimation unit 102 estimates multiple variable values ​​(external parameters) and internal parameters. Note that when the estimation unit 102 estimates only the external parameters, it may use internal parameters estimated in advance using images captured by the cameras 2a to 2c before installing the cameras 2a to 2c on the industrial vehicle 7. As described above, the predetermined reference object may be, for example, a calibration board or a white line. There is no limitation on the method by which the estimation unit 102 estimates the multiple variable values. For example, it may use the method of estimating external parameters using a calibration index, as described in Japanese Patent Application Laid-Open No. 2012-015576. The estimation unit 102 stores multiple variable values ​​(external parameters) representing the estimated position and orientation of each camera and internal parameters in the memory unit 108 as information 108D representing the multiple variable values ​​(hereinafter also referred to as multiple variable values ​​108D).

[0024] The acquisition unit 103 acquires information 108D representing a plurality of variable values ​​representing the position and attitude of each of the cameras 2a to 2c estimated based on a plurality of images of a predetermined target to be captured by the plurality of cameras 2a to 2c installed on the industrial vehicle 7 with different imaging directions, image information 108C representing a plurality of images of a worker 6 (FIG. 7) positioned in the vicinity of the industrial vehicle 7 captured by the plurality of cameras 2a to 2c, identification information 108B identifying the position of the worker 6, and the plurality of variable values ​​108D. Here, the acquisition by the acquisition unit 103 means making the information available for use in processing after acquisition, and for example, means making a file in the storage unit 108 available or expanding it in a predetermined storage area.

[0025] Here, a configuration example of the identification information 108B will be described with reference to FIGS. 4 and 5. FIG. 4 is a schematic diagram illustrating a configuration example of the identification information 108B according to the first embodiment of the present disclosure. FIG. 5 is a schematic diagram illustrating a plan view 108B-2 included in the identification information 108B according to the first embodiment of the present disclosure. In this embodiment, the identification information 108B is information for identifying the position of the worker 6 when the multiple cameras 2a to 2c capture images of the worker 6. The identification information 108B illustrated in FIG. 4 includes information 108B-1 related to the worker's movement route and the stop time at a predetermined position on the movement route, and a plan view 108B-2 representing the movement route and the stop time. The example illustrated in FIGS. 4 and 5 is an example in which a worker moves around a periphery a distance L1 away from the industrial vehicle 7. The worker moves to each of the circled positions 1 to 9 in numerical order. The worker pauses at each position for 10 or 5 seconds. Information 108B-1 is a table containing the coordinate values ​​in the vehicle coordinate system (XL-YL coordinate system in FIG. 5) of each of positions 1 to 9 on plan view 108B-2, the stop times, and numbers, in association with each other. Note that information 108B-1 may also include information on, for example, the travel time between predetermined positions in addition to the travel route and the stop times (or excluding the stop times).

[0026] Based on a plurality of variable values ​​108D, the composition unit 104 combines a plurality of images based on image information 108C representing a plurality of images captured by a plurality of cameras 2a to 2c of a worker 6 positioned around the industrial vehicle 7 into an overhead image representing the periphery of the industrial vehicle 7. Fig. 6 is a schematic diagram showing an example of an overhead image BEV1 combined by the composition unit 104 according to the first embodiment of the present disclosure. 6 includes an image region IM-2a obtained by correcting an image captured by camera 2a based on a plurality of variable values ​​108D; an image region IM-2b obtained by correcting an image captured by camera 2b based on a plurality of variable values ​​108D; an image region IM-2c obtained by correcting an image captured by camera 2c based on a plurality of variable values ​​108D; an image IM-6 of worker 6; a CG (computer graphics) image IM-7 representing an image of industrial vehicle 7 captured in advance from above or an image of industrial vehicle 7 viewed from above; and a CG image IM-NI representing an area in front of industrial vehicle 7 that is not included in the image capture range of cameras 2a to 2c (a blind spot area). The method for synthesizing the overhead images is not limited, and existing techniques can be used (e.g., JP 10-211849 A). The synthesizing unit 104 synthesizes a plurality of images for a predetermined period of time based on image information 108C, and stores the generated overhead images as information 108E in the storage unit 108.

[0027] Furthermore, the synthesis unit 104 can have a function of synthesizing the captured image 108A into an overhead image based on a plurality of variable values ​​108D, and displaying the image on the display device 5 in almost real time.

[0028] The recognition unit 105 recognizes the worker 6 from the multiple overhead images based on the information 108E. There are no limitations on the image recognition processing, and the recognition unit 105 may recognize the worker 6 included in the multiple overhead images by, for example, image recognition using pattern matching or image recognition using a trained machine learning model.

[0029] The determination unit 106 determines whether the multiple variable values ​​108D are acceptable based on the recognition result of the worker 6 by the recognition unit 105 and the identification information 108B. If the recognition unit 105 does not recognize the worker 6 at the position identified by the identification information 108B, the determination unit 106 determines that the multiple variable values ​​108D are unacceptable. For example, if the worker 6 is recognized for approximately 10 seconds at position "1" shown in FIG. 5 based on the recognition result, the determination unit 106 estimates the approximate positions of the worker 6 from position "2" to position "9" and the time spent at each position. If the worker 6 is recognized at all estimated positions "1" to "9" for the estimated time, the determination unit 106 determines that the multiple variable values ​​108D are acceptable. If the worker 6 is not recognized at any of the estimated positions "1" to "9" for the estimated time, the determination unit 106 determines that the multiple variable values ​​108D are unacceptable.

[0030] The judgment result output unit 107 outputs the judgment result of the judgment unit 106, indicating whether the multiple variable values ​​108D are good or bad, to, for example, the display device 5. FIG. 7 is a schematic diagram showing an example of output by the judgment result output unit 107 according to the first embodiment of the present disclosure. FIG. 8 is a schematic diagram showing another example of output by the judgment result output unit 107 according to the first embodiment of the present disclosure. FIG. 7 shows an example of output when the worker 6 is recognized at all positions at the estimated time. The overhead image BEV2 shown in FIG. 7 is an overhead image obtained by combining images of the worker 6 recognized at positions 1 to 9. FIG. 8 shows an example of output when the worker 6 is not recognized at position "2" at the estimated time. The overhead image BEV3 shown in FIG. 8 is an overhead image obtained by combining images of the worker 6 recognized at positions 1 and 3 to 9, and combining, at position "2," a circled "2" indicating the position of "2."

[0031] Next, an example of the operation of the information processing device 1 will be described with reference to Fig. 9. The example of the operation shown in Fig. 9 starts in a state where, for example, the initial installation of the cameras 2a to 2c is completed, external parameters and the like are estimated by the estimation unit 102 using a predetermined reference imaging target, and the estimated camera parameters are stored as a plurality of variable values ​​108D in the storage unit 108. It is also assumed that the cameras 2a to 2c are constantly capturing images after the information processing device 1 is started up.

[0032] For example, when the worker 6 performs a predetermined operation on the input device 4 to instruct the start of the process shown in FIG. 9, the process shown in FIG. 9 is started. First, the worker 6 performs a predetermined input operation on the input device 4 to instruct the start of image capture (step S10). The worker 6 moves around the industrial vehicle 7 while referring to the floor plan 108B-2 included in the identification information 108B (for example, referring to the floor plan 108B-2 printed on a paper medium or the floor plan 108B-2 displayed on the mobile terminal 3). The captured image acquisition unit 101 acquires images captured by the cameras 2a to 2c and records them in the storage unit 108 as image information 108C (step S11). Next, when the worker 6 performs a predetermined input operation on the input device 4 to instruct the end of image capture, the captured image acquisition unit 101 ends recording of the image information 108C (step S12).

[0033] Next, acquisition unit 103 acquires a plurality of variable values ​​108D from storage unit 108 (step S13). Next, acquisition unit 103 acquires image information 108C from storage unit 108 (step S14). Next, acquisition unit 103 acquires identification information 108B from storage unit 108 (step S15). Next, synthesis unit 104 synthesizes overhead images based on the plurality of variable values ​​108D, and stores the generated plurality of overhead images in storage unit 108 as information 108E (step S16).

[0034] Next, recognition unit 105 recognizes worker 6 from multiple overhead images based on information 108E (step S17). Next, determination unit 106 determines whether multiple variable values ​​108D are acceptable or unacceptable based on the recognition result of worker 6 and identification information 108B (step S18). Next, determination result output unit 107 outputs the pass / fail determination result made by determination unit 106 (step S19), and the process shown in FIG. 9 ends.

[0035] As described above, according to this embodiment, the information processing device 1 (camera calibration support device) includes an acquisition unit 103, a synthesis unit 104, a recognition unit 105, a determination unit 106, and a determination result output unit 107. The acquisition unit 103 acquires a plurality of variable values ​​108D representing the positions and orientations of the cameras 2a to 2c, which are estimated based on a plurality of images of a predetermined target image serving as a reference captured by the cameras 2a to 2c and installed on the industrial vehicle 7 with different imaging directions; image information 108C representing a plurality of images of a worker 6 positioned in the vicinity of the industrial vehicle 7 captured by the cameras 2a to 2c; and identification information 108B identifying the position of the worker 6. The synthesis unit 104 synthesizes the plurality of images based on the image information 108C onto an overhead image BEV1 representing the vicinity of the industrial vehicle 7, based on the plurality of variable values ​​108D. The recognition unit 105 recognizes the worker 6 from the overhead image BEV1. The determination unit 106 determines whether the multiple variable values ​​108D are acceptable based on the recognition result of the worker 6 and the identification information 108B. The determination result output unit 107 outputs the judgment result. With this configuration, it is possible to determine whether the variable values ​​estimated in the camera calibration are acceptable. Therefore, if the judgment result indicates that the variable values ​​are unacceptable, the camera parameters can be correctly estimated by adjusting or adding markers, adjusting the camera attitude or position, and then performing camera calibration again.

[0036] If the recognition unit 105 does not recognize the worker 6 at the position identified by the identification information 108B, the determination unit 106 determines that the plurality of variable values ​​108D are defective.

[0037] Furthermore, the identification information 108B includes information relating to the movement route and stop time of the worker 6.

[0038] Since the main use of overhead video in industrial vehicles is safety confirmation, it is desirable to judge the quality of camera calibration based on whether it "impairs the safety confirmation function." For example, when judging the quality of overhead video generated by calculating camera parameters using a general calibration board, the criterion is to move a "person" within the video and determine whether the "person" can be seen correctly. The "position of the person that must be visible" is defined individually for each vehicle, but could be, for example, "all around the vehicle at a distance of 50 cm" or "a specific location behind the vehicle."

[0039] By using whether or not a person can be seen as the criterion for judgment, it becomes possible to check for cases where blind spots occur due to calibration errors, and it becomes possible to satisfy the performance required for safety confirmation, which is the purpose of overhead video in industrial vehicles.

[0040] However, visually checking whether a "person" appears correctly is difficult, and there is a risk of human error, such as overlooking a person. Therefore, in this embodiment, whether a "person" appears correctly is automated using "person detection" through image processing. Furthermore, to confirm whether the "detection results by the system" are detecting the actual person that should be detected, the "person position = correct" is input into the system (information on the location and order (specific information 108B) is input in advance, for example, in file format). In this case, for example, the "person positions" are numbered in advance, and by changing the positions in order from the initial position every specific time (for example, 10 seconds), the positions are automatically compared with the detection results by the system.

[0041] It is now possible to automatically determine whether a "person" is visible using image processing, thereby reducing errors in pass / fail judgment due to human error.

[0042] The image processing human detection method may be an existing method such as a human detection method that uses a feature vector called HOG (Histogram of Oriented Gradients), but it would be even better if it is tuned specifically for determining whether the calibration is correct, for example, by performing machine learning using multiple training data sets consisting of a pair of overhead images of a worker and information indicating the worker's area in image recognition using a trained machine learning model.

[0043] Second Embodiment Next, a second embodiment of the present disclosure will be described with reference to Fig. 10. Fig. 10 is a schematic diagram showing an example of output by the determination result output unit 107 according to the second embodiment of the present disclosure. In the first embodiment and the second embodiment, the configuration of the information processing device 1 shown in Fig. 1 is basically the same. However, the operation of the determination result output unit 107 is partially different between the first embodiment and the second embodiment.

[0044] In the first embodiment, the worker 6 was recognized using a simple binary value: whether or not the "person" was visible. In contrast, in the second embodiment, the "degree of visibility" is determined to achieve better calibration. Therefore, in the second embodiment, a score is also output when the "person" is detected by image processing to quantify the visibility. This score may be, for example, the reliability obtained by deep learning image processing, or the area (number of pixels) of the "person" portion. When the number of pixels is used, the score may be, for example, the absolute value of the difference between the average number of pixels in the area of ​​the worker 6 and the relevant pixel number, divided by the average number of pixels, and then subtracted from 1. Note that the confidence score in object recognition is an index that indicates the accuracy of whether an object is included in the divided area and whether the area is accurately enclosed, and the predicted probability of each class (the likelihood that an object in the image belongs to a specific class).

[0045] The determination result output unit 107 of the second embodiment outputs a predetermined index value representing the visibility of the worker 6. Fig. 10 shows an example of output when the worker 6 is recognized at all positions at the estimated time. The overhead image BEV4 shown in Fig. 10 is an overhead image obtained by combining images of the worker 6 recognized at each of positions 1 to 9. The overhead image BEV4 shown in Fig. 10 includes a score value of 85 near each recognized worker 6.

[0046] According to the second embodiment, it is possible to assign a score to the visibility of "people," and it is possible to determine the quality of the calibration in more detail.

[0047] As for the method of calculating the score, other methods such as the difference from a "human template" that indicates "how it should look at this position" may also be used.

[0048] Third Embodiment Next, a third embodiment of the present disclosure will be described with reference to FIGS. 11 and 12. FIG. 11 is a block diagram illustrating a schematic configuration of an information processing system 10A according to a third embodiment of the present disclosure. FIG. 12 is a schematic diagram illustrating a configuration example of identification information 108B according to the third embodiment of the present disclosure. The third embodiment differs from the first embodiment in that the mobile terminal 3 illustrated in FIG. 11 includes a location information acquisition unit 31, the configuration of the identification information 108B is different, the information processing device 1A corresponding to the information processing device 1 includes a location information acquisition unit 109, and the operation of the determination unit 106 is partially different. As illustrated in FIG. 12, the identification information 108B of the third embodiment includes a time series 108B-3 of the location information of the worker 6 and a time series 108B-4 of the industrial vehicle 7. The time series 108B-3 of the location information of the worker 6 includes time series information that combines, for example, coordinates (latitude and longitude) in a world coordinate system acquired by the location information acquisition unit 31 of the mobile terminal 3 and time information (year, month, day, hour, minute, second) at the time of acquisition. The position information 108B-4 of the industrial vehicle 7 includes position information of two different positions of the industrial vehicle 7.

[0049] The position information acquisition unit 31 of the mobile terminal 3 acquires position information from a positioning device that uses, for example, the Global Navigation Satellite System (GNSS). The positioning device (receiving device) may be built into the mobile terminal 3, or may be connected to the mobile terminal 3 via a wired or wireless communication line. Furthermore, when the cameras 2a to 2c capture images of a worker 6 (carrying the mobile terminal 3) located in the vicinity of the industrial vehicle 7, the position information acquisition unit 31 sequentially acquires position information of the mobile terminal 3 and transmits the acquired time series of the position information to the information processing device 1A. The information processing device 1A stores the received time series of the position information in the storage unit 108 as identification information 108B (time series 108B-3 of the position information of the worker 6).

[0050] Furthermore, when the cameras 2a to 2c capture images of the worker 6 positioned around the industrial vehicle 7, the position information acquisition unit 109 of the information processing device 1A acquires position information at two or more different positions on the industrial vehicle 7 (for example, positions in front of the industrial vehicle 7 and positions behind the industrial vehicle 7, which are preferably at least a certain distance apart) from two or more positioning devices using GNSS, and stores the acquired position information in the storage unit 108 as specific information 108B (position information 108B-4 of the industrial vehicle 7). The position information acquisition unit 109 may acquire the position information from a positioning device attached to the industrial vehicle 7, or may acquire the position information by, for example, placing the mobile terminal 3 at a predetermined position on (or in a predetermined vicinity of) the industrial vehicle 7. The position and orientation of the industrial vehicle 7 can be determined from the position information from two different points.

[0051] Furthermore, the determination unit 106 of the information processing device 1A determines whether the multiple variable values ​​108D are acceptable or not, based on the recognition result of the recognition unit 105 and the time series of the position information included in the identification information 108B.

[0052] According to the third embodiment, if it is determined that a "person" is not visible (or difficult to see), knowing where that person is located allows for focused corrections to be made when readjusting camera parameters. Therefore, in the third embodiment, a means capable of acquiring position information of a "person" relative to the industrial vehicle 7 is used to make it possible to acquire the position of the "person" in each image. Note that the means for acquiring position information is not limited to GNSS, and existing methods such as an IMU (Inertial Measurement Unit) can also be used.

[0053] According to the third embodiment, it is possible to obtain position information of invisible (or hard-to-see) "people," and it is possible to identify cameras that require recalibration and parameters that need to be adjusted.

[0054] <Fourth embodiment> While the previous embodiments were able to determine whether a person was correctly seen, they were unable to identify the cause. If a person was not seen, the cause could be either (1) the actual camera being too misaligned or (2) the camera's calibration parameters being different from the actual location. To make this determination, a CG simulator is utilized. An industrial vehicle and camera are recreated in a virtual space using the determined calibration parameters, and a CG image of a person placed in the "missing location" is checked. If the person is not visible in the CG image, (1) the actual camera is too misaligned, so the actual camera is adjusted to match the ideal camera position determined in advance. On the other hand, if the person is visible in the CG image, (2) the calibration parameters are different, so calibration is performed again, for example, by adding more calibration boards within the field of view (or both). This eliminates the need to blindly adjust the actual camera or calibration parameters when a person is not seen, thereby improving efficiency. In other words, according to the fourth embodiment, if a person is not seen in a location where it should be seen, it is possible to determine whether the actual camera or the calibration parameters should be adjusted, thereby improving the efficiency of calibration readjustment. If the actual image and the CG reproduction match (i.e., the person is not visible in either), adjust the position and angle of the actual camera. On the other hand, if the person is visible in the CG, increase the number of boards and recalibrate to correct the camera parameters.

[0055] The fourth embodiment will be described with reference to Figs. 13 to 18. Fig. 13 is a block diagram showing a schematic configuration of an information processing system 10B according to the fourth embodiment of the present disclosure. Fig. 14 is a flowchart showing an example of operation of an information processing device 1B according to the fourth embodiment of the present disclosure. Fig. 15 is a schematic diagram showing an example of a three-dimensional model generated by a generation unit 110 according to the fourth embodiment of the present disclosure. Fig. 16 is a schematic diagram showing an example of output according to the fourth embodiment of the present disclosure. Figs. 17 and 18 are schematic diagrams showing other example of output according to the fourth embodiment of the present disclosure.

[0056] 1, an information processing device 1B of the fourth embodiment shown in Fig. 13 newly includes a generation unit 110, a virtual synthesis unit 111, and an adjustment target identification unit 112. In addition, a determination result output unit 107 included in the information processing device 1B outputs the result of identification of the adjustment target by the adjustment target identification unit 112, etc.

[0057] As shown in FIG. 15, the generation unit 110 generates a three-dimensional model M1 that reproduces in a virtual space V1 a vehicle model M-7 that imitates an industrial vehicle 7, multiple camera models M-2a to M-2c that imitate multiple cameras 2a to 2c having positions and attitudes based on multiple variable values ​​108D, and a worker model M-6 that imitates a worker 6 based on specific information 108B.

[0058] The virtual synthesis unit 111 synthesizes multiple virtual images of the worker model M-6 taken by multiple camera models M-2a to M-2c into virtual overhead images VBEV1 (Figure 16), VBEV2 (Figure 17), etc. that represent the periphery of the vehicle model M-7.

[0059] When the recognition unit 105 does not recognize the worker 6 at the position identified by the identification information 108B, the adjustment target identification unit 112 identifies an adjustment target that requires adjustment based on at least one of the positions and orientations of the multiple cameras 2a to 2c and the multiple variable values ​​108D, based on how the worker model M-6 is included at that position in the virtual overhead-view image VBEV1. Here, the way the worker model M-6 is included can be, for example, when the worker model M-6 is included in the virtual overhead-view image VBEV1 with a predetermined accuracy or higher (hereinafter referred to as a first accuracy), when the worker model M-6 is not included in the virtual overhead-view image VBEV1 with a predetermined accuracy or higher (hereinafter referred to as a second accuracy), or when neither of the above applies. Note that the accuracy can be, for example, the reliability score in the object recognition described above. Furthermore, the first accuracy and the second accuracy may be the same or different. The adjustment target may be at least one of the positions and orientations of the cameras 2a to 2c, the multiple variable values ​​108D, or both the positions and orientations of the cameras 2a to 2c and the multiple variable values ​​108D. If the recognition unit 105 does not recognize the worker 6 at the position identified by the identification information 108B, and if the virtual overhead-view image VBEV1 does not include the worker model M-6 at that position with a predetermined degree of accuracy, the adjustment target identification unit 112 identifies at least one of the positions and orientations of the cameras 2a to 2c as the adjustment target. If the virtual overhead-view image VBEV1 includes the worker model M-6 with a predetermined degree of accuracy, the adjustment target identification unit 112 identifies both as the adjustment target.

[0060] Fig. 14 shows an example of the operation of information processing device 1B. The process shown in Fig. 14 is executed, for example, between step S18 and step S19 in Fig. 9. In the following example of operation, it is assumed that the recognition result of recognition unit 105 does not recognize worker 6 at position "2," as in overhead image BEV3 shown in Fig. 8.

[0061] 14, first, the adjustment target specifying unit 112 determines whether or not the worker 6 has been recognized by the recognition unit 105 at each position specified by the identification information 108B (step S101). If the worker 6 has been recognized at each position (step S101: YES), the adjustment target specifying unit 112 ends the process shown in FIG.

[0062] If the worker 6 is not recognized at any position (step S101: NO), the generation unit 110 generates a three-dimensional model M1 by reproducing the vehicle model M-7, the camera models M-2a to M-2c based on the multiple variable values ​​108D, and the worker model M-6 (at the unrecognized position) based on the identification information 108B in the virtual space V1 (step S102). Next, the virtual synthesis unit 111 synthesizes multiple virtual images of the worker model M-6 captured by the multiple camera models M-2a to M-2c into virtual overhead images VBEV1, VBEV2, etc. that represent the periphery of the vehicle model M-7 (step S103).

[0063] Next, the adjustment target specifying unit 112 determines whether or not the worker model M-6 is included in the virtual overhead images VBEV1, VBEV2, etc. (step S104).

[0064] If the worker model M-6 is not clearly (= with a predetermined accuracy or higher) included in the virtual overhead images VBEV1, VBEV2, etc. (step S104: not clearly included), the adjustment target identification unit 112 identifies at least one of the positions and orientations of the multiple cameras 2a to 2c as the adjustment target (step S105), and displays, for example, a virtual overhead image VBEV1 as shown in Figure 16 on the display device 5, and terminates the processing shown in Figure 14.

[0065] If the worker model M-6 is clearly (= with a predetermined accuracy or higher) included in the virtual overhead images VBEV1, VBEV2, etc. (step S104: clearly included), the adjustment target identification unit 112 identifies multiple variable values ​​108D as being to be adjusted (step S106), and displays, for example, a virtual overhead image VBEV2 as shown in Figure 17 on the display device 5, and terminates the processing shown in Figure 14.

[0066] The virtual overhead image VBEV1 shown in Fig. 16 indicates that the worker model M-6 is not included at the position "2" in the identification information 108B. The virtual overhead image VBEV2 shown in Fig. 17 indicates that the worker model M-6 is included at the position "2" in the identification information 108B.

[0067] If the case does not fall into either the case where worker model M-6 is clearly included or not clearly included (step S104: neither), adjustment target identification unit 112 identifies at least one of the positions and orientations of multiple cameras 2a to 2c and multiple variable values ​​108D as adjustment targets (step S107), and displays a virtual overhead image VBEV3 as shown in Fig. 18 on display device 5, for example, and terminates the processing shown in Fig. 14. The virtual overhead image VBEV3 shown in Fig. 18 indicates that worker model M-6 is unclearly included at position "2" in identification information 108B.

[0068] <Other embodiments> Although the embodiments of the present disclosure have been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment and includes design modifications within the scope of the present disclosure. Furthermore, the above-described embodiments can be combined as appropriate. Furthermore, the worker's position or movement path is not limited to the entire or nearly entire perimeter of the work vehicle, but may be one or more specific positions. Furthermore, in the above-described embodiment, the overhead image is an image showing the left, right, and rear of the work vehicle. However, for example, the overhead image may be an image showing the entire perimeter of the work vehicle, or an image showing the left, right, and front of the work vehicle. Furthermore, in step S106 of FIG. 14, only multiple variable values ​​are identified as adjustment targets, but both multiple variable values ​​and the position and orientation of the camera may be identified as adjustment targets.

[0069] <Computer configuration> FIG. 19 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. The computer 90 includes a processor 91 , a main memory 92 , a storage 93 , and an interface 94 . The above-described information processing devices 1, 1A, and 1B are implemented in a computer 90. The operations of the above-described processing units are stored in the form of a program in a storage 93. A processor 91 reads the program from the storage 93, loads it into a main memory 92, and executes the above-described processing in accordance with the program. The processor 91 also allocates storage areas in the main memory 92 corresponding to the above-described storage units in accordance with the program.

[0070] The program may be for realizing some of the functions to be performed by the computer 90. For example, the program may be combined with other programs already stored in storage or other programs implemented in other devices to perform the functions. In other embodiments, the computer may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the processor may be realized by the integrated circuit.

[0071] Examples of storage 93 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 93 may be an internal medium directly connected to the bus of computer 90, or an external medium connected to computer 90 via interface 94 or a communication line. Furthermore, when this program is distributed to computer 90 via a communication line, computer 90 that receives the program may load the program into main memory 92 and execute the above-described processing. In at least one embodiment, storage 93 is a non-transitory tangible storage medium.

[0072] <Additional Notes> The camera calibration support device (information processing devices 1, 1A, and 1B) described in each embodiment can be understood, for example, as follows.

[0073] (1) A camera calibration support device according to a first aspect includes: an acquisition unit that acquires multiple variable values ​​representing the position and orientation of each camera, estimated based on multiple images of a predetermined target captured by multiple cameras mounted on a vehicle with different imaging directions; image information representing multiple images of a worker positioned around the vehicle captured by the multiple cameras; and identification information identifying the position of the worker; a synthesis unit that synthesizes the multiple images based on the image information into an overhead image representing the area around the vehicle based on the multiple variable values; a recognition unit that recognizes the worker from the overhead image; a determination unit that determines whether the multiple variable values ​​are acceptable based on the worker recognition result and the identification information; and a determination result output unit that outputs the determination result. According to this aspect and the following aspects, the quality of the variable values ​​estimated in camera calibration can be determined. Therefore, if the determination result indicates that the variable values ​​are unacceptable, the camera parameters can be correctly estimated by adjusting the markers and the camera's attitude and position, and then performing camera calibration again.

[0074] (2) A camera calibration support device according to a second aspect is a camera calibration support device according to (1), wherein the determination unit determines that the multiple variable values ​​are poor when the worker is not recognized by the recognition unit at the position identified by the identification information.

[0075] (3) A camera calibration support device according to a third aspect is the camera calibration support device of (1) or (2), wherein the judgment result output unit outputs a predetermined index value representing the ease of visibility for the worker.

[0076] (4) A camera calibration support device according to a fourth aspect is the camera calibration support device according to any one of (1) to (3), wherein the specific information includes information relating to the movement route and stop time of the worker.

[0077] (5) A camera calibration support device according to a fifth aspect is the camera calibration support device according to any one of (1) to (4), wherein the specific information includes a time series of position information of the worker.

[0078] (6) A camera calibration support device according to a sixth aspect is a camera calibration support device according to any one of (1) to (5), further comprising: a generation unit that generates a three-dimensional model that reproduces in a virtual space a vehicle model that imitates the vehicle, a plurality of camera models that imitate the plurality of cameras having positions and attitudes based on the plurality of variable values, and a worker model that imitates the worker based on the specific information; a virtual synthesis unit that synthesizes a plurality of virtual images of the worker model captured by the plurality of camera models into a virtual overhead image that represents the periphery of the vehicle model; and an adjustment target identification unit that, when the worker is not recognized by the recognition unit at the position identified by the specific information, identifies an adjustment target that needs to be adjusted from at least one of the positions and attitudes of the plurality of cameras and the plurality of variable values, based on how the worker model is included in the virtual overhead image.

[0079] (7) A camera calibration support device according to a seventh aspect is a camera calibration support device according to any one of (1) to (6), wherein the adjustment target identification unit identifies at least one of the positions and orientations of the plurality of cameras as the adjustment target when the worker is not recognized by the recognition unit at the position identified by the identification information and when the worker model is not included in the virtual overhead image with a predetermined accuracy or higher, and identifies the plurality of variable values ​​as the adjustment target when the worker model is included in the virtual overhead image with a predetermined accuracy or higher. [Explanation of symbols]

[0080] 1, 1A, 1B...Information processing device (camera calibration support device) 2a~2d...Camera 7...Industrial vehicles 103…Acquisition part 104...Synthesis section 105...Recognition part 106...Judgment section 107...Determination result output unit 108B…Specific information 108C...Image information 108D...Multiple variable values 110...Generation section 111...Virtual synthesis section 112...Adjustment target specific part

Claims

1. an acquisition unit that acquires a plurality of variable values ​​that represent the position and attitude of each camera, estimated based on a plurality of images of a predetermined target imaged as a reference taken by a plurality of cameras with different imaging directions that are installed on a vehicle; image information that represents a plurality of images of a worker positioned around the vehicle taken by the plurality of cameras; and identification information that identifies the position of the worker; a synthesis unit that synthesizes the plurality of images based on the image information into an overhead image representing the periphery of the vehicle based on the plurality of variable values; a recognition unit that recognizes the worker from the overhead image; a determination unit that determines whether the plurality of variable values ​​are good or bad based on the recognition result of the worker and the specific information; a judgment result output unit that outputs the pass / fail judgment result; A camera calibration support device comprising:

2. The determination unit determines that the plurality of variable values ​​are defective when the worker is not recognized by the recognition unit at the position identified by the identification information. The camera calibration support device according to claim 1 .

3. The determination result output unit outputs a predetermined index value representing the visibility of the worker. The camera calibration support device according to claim 2 .

4. The specific information includes information regarding the movement route and stop time of the worker. The camera calibration support device according to claim 3 .

5. The specific information includes a time series of the location information of the worker. The camera calibration support device according to claim 3 .

6. a generation unit that generates a three-dimensional model in which a vehicle model that imitates the vehicle, a plurality of camera models that imitate the plurality of cameras having positions and orientations based on the plurality of variable values, and a worker model that imitates the worker based on the specific information are reproduced in a virtual space; a virtual synthesis unit that synthesizes a plurality of virtual images of the worker model captured by the plurality of camera models into a virtual overhead image that represents the periphery of the vehicle model; an adjustment target identification unit that identifies an adjustment target that needs to be adjusted from at least one of the positions and orientations of the plurality of cameras and the plurality of variable values ​​based on how the worker model is included in the virtual overhead image, when the recognition unit does not recognize the worker at the position identified by the identification information; The camera calibration support device according to any one of claims 1 to 5, further comprising:

7. The adjustment target identification unit identifies at least one of the positions and orientations of the plurality of cameras as the adjustment target when the worker is not recognized by the recognition unit at the position identified by the identification information and when the worker model is not included in the virtual overhead image with a predetermined accuracy or higher, and identifies the plurality of variable values ​​as the adjustment target when the worker model is included in the virtual overhead image with a predetermined accuracy or higher. The camera calibration support device according to claim 6.

8. a step of acquiring a plurality of variable values ​​representing the position and attitude of each camera, estimated based on a plurality of images of a predetermined target imaged as a reference taken by a plurality of cameras with different imaging directions installed on a vehicle, image information representing a plurality of images of a worker positioned around the vehicle taken by the plurality of cameras, and identification information for identifying the position of the worker; synthesizing the plurality of images based on the image information into an overhead image representing the periphery of the vehicle based on the plurality of variable values; Recognizing the worker from the overhead image; determining whether the plurality of variable values ​​are acceptable or not based on the recognition result of the worker and the specific information; outputting the pass / fail judgment result; A camera calibration assistance method including:

9. a step of acquiring a plurality of variable values ​​representing the position and attitude of each camera, estimated based on a plurality of images of a predetermined target imaged as a reference taken by a plurality of cameras with different imaging directions installed on a vehicle, image information representing a plurality of images of a worker positioned around the vehicle taken by the plurality of cameras, and identification information for identifying the position of the worker; synthesizing the plurality of images based on the image information into an overhead image representing the periphery of the vehicle based on the plurality of variable values; Recognizing the worker from the overhead image; determining whether the plurality of variable values ​​are acceptable or not based on the recognition result of the worker and the specific information; outputting the pass / fail judgment result; A program that causes a computer to execute the following.

Images