Photography support device, photography system, and photography support method

Abstract

It assists in determining the position and orientation for photographing changes in the appearance of an object. [Solution] The shooting support device 100 supports the shooting of an object using a plurality of first position and orientations, which are the position and orientation of the camera that captured the object and are stored in the position and orientation database 130. The shooting support device 100 includes a receiving unit 111 that receives a second position and orientation, which is the position and orientation of the camera that captured the object and is

Description

Technical Field

[0001] The present invention relates to a photographing support device, a photographing system, and a photographing support method.

Background Art

[0002] For traffic infrastructure such as roads and railways, and facilities with a long design life such as plants, inspection and maintenance are important. In such facilities, visual inspection is performed to visually confirm the presence or absence of appearance defects such as discoloration, damage, rust, and corrosion on the surface of the facility, and to determine the degree of influence on the facility. In large-scale infrastructure facilities, the inspection targets (inspection points) are diverse, and the appearance defects for each target are also diverse. Furthermore, defect discrimination requires skilled know-how. Therefore, a method for improving the efficiency of the appearance inspection of such infrastructure facilities is required.

[0003] As an invention for inspecting defects of an inspection object, there is an inspection device described in Patent Document 1. This inspection device includes an acquisition unit that acquires a plurality of captured images obtained by capturing an inspection object from a plurality of angles, an assignment unit that assigns a label corresponding to the corresponding angle to each of the captured images, an encoder that inputs the captured image as an input image and the corresponding label to generate a latent variable, and a decoder that inputs the label and the latent variable to generate a restored image. A learning processing unit that learns a neural network so that the input image and the restored image are the same, and a restoration inspection image that is the restored image obtained by inputting the captured image as an inspection image and the corresponding label to the neural network during inspection of the inspection object, and an inspection processing unit that performs an inspection process for defects of the inspection object based on the difference between the inspection image and the restoration inspection image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] According to the inspection device described in Patent Document 1, defects can be inspected with high accuracy using captured images of the object to be inspected. However, it is difficult to acquire training / learning image data and labels from all directions (shooting position and orientation, position and orientation) for a wide variety of objects to be inspected. It is desirable that inspection / checking be possible based on training / learning images taken at shooting positions and orientations within the range of visual inspection.

[0006] In image generation using machine learning techniques, if the position and orientation of the image taken during inspection / checking deviate from the position and orientation of the training image taken under normal conditions, the reconstructed inspection image will be blurry or the generation will fail. This makes it impossible to compare the captured image with the reconstructed inspection image. Therefore, it is desirable to take the image during inspection / checking at a position and orientation similar to the position and orientation when the object being inspected / checked is under normal conditions. The desire to take multiple images taken at the same or similar position and orientation when there is a time gap between them is not limited to inspection / checking. For example, it is desirable to take images at a similar position and orientation when detecting changes in the layout of a facility or when monitoring / observing changes in the appearance of equipment, facilities, or manufactured goods.

[0007] This invention was made in view of the above background, and aims to provide a shooting support device, shooting system, and shooting support method that assist in determining the position and orientation for photographing changes in the appearance of an object. [Means for solving the problem]

[0008] To solve the above-mentioned problems, the present invention provides a shooting support device in which a plurality of first positional orientations, which are the position and orientation of a camera that has captured an image of an object, are stored in a storage unit, and the shooting support device provides support for shooting the object using the first positional orientations, comprising: a receiving unit that receives a second positional orientation, which is the position and orientation of a camera that has captured an image of an object; a determination unit that determines whether the second positional orientation is appropriate or not according to the number of first positional orientations in a space including the first positional orientation and the second positional orientation that are within a predetermined distance from the second positional orientation; a determination unit that, when the determination unit determines that the second positional orientation is not appropriate, determines a direction in which the position and orientation will be changed to approach a positional orientation that the determination unit determines to be appropriate, starting from the second positional orientation; and an output control unit that outputs the direction determined by the determination unit. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a shooting support device, shooting system, and shooting support method that assist in determining the position and orientation for photographing changes in the appearance of an object. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments. [Brief explanation of the drawing]

[0010] [Figure 1] This is a functional block diagram of the shooting support device according to this embodiment. [Figure 2] This is a screen configuration diagram of the visual inspection support screen according to this embodiment. [Figure 3] This is a screen configuration diagram of the visual inspection support screen according to this embodiment. [Figure 4] This is a flowchart of the normal image registration process according to this embodiment. [Figure 5] This is a flowchart of the visual inspection support process according to this embodiment. [Figure 6] This is an overall configuration diagram of a modified photographic system according to this embodiment. [Figure 7] This is a hardware configuration diagram showing an example of a computer that implements the functions of the imaging support device according to the above embodiment. [Modes for carrying out the invention]

[0011] ≪Overview of the imaging support device≫ The following describes an overview of the imaging support device in an embodiment for carrying out the present invention. In this embodiment, the imaging support device is configured to function as an "external appearance inspection support device" that assists in inspecting the external appearance of an object. Details of the imaging support device, including its configuration and functions, will be described later.

[0012] The inspector, who is a user of the camera support device, takes a picture of the inspection target (inspection point) under normal conditions and records (registers) the image as a normal image in the camera support device, which is an external inspection support device. At this time, the camera support device also records (registers) the position and orientation of the camera used to take the picture, in addition to the captured image (normal image).

[0013] Once the registration of normal images and the position and orientation of the camera is complete, the camera support device generates an image generation model, which is a machine learning model, using training data in which the explanatory variable is the position and orientation of the camera and the target variable is the image taken at that position and orientation. By using this image generation model, the camera support device can generate an image of the inspection target taken from that position and orientation based on the position and orientation.

[0014] When an inspector photographs the object being inspected during an inspection, the photography support device generates an image using an image generation model based on the position and orientation of the photograph. This image is then compared with the captured image, and any differing areas are output as potential defect locations (see area 421 in Figure 3 below).

[0015] Regarding the images generated by the image generation model, the generated images in the regions where the distribution (density of the distribution) of the position and orientation at the time of registering the normal-time image is high are of high quality. High quality means that the evaluation score of the position and orientation calculated by the evaluation function constructed during the generation of the image model is high. It may be considered that the similarity between the image captured of the inspection target from the position and orientation and the generated image is high. However, the generated images in the regions where the distribution is low or there is no distribution are of low quality. In other words, at the time of inspection, it is required to capture images in the same or nearly the same position and orientation as when registering the normal-time image.

[0016] To correspond to this, the imaging support device guides the inspector to capture images in the same / nearly the same position and orientation as when registering the normal-time image at the time of inspection. To explain in detail, when the inspector captures an image in a position and orientation deviating from that at the time of registering the normal-time image, the imaging support device indicates the direction of the same / nearly the same position and orientation as when registering the normal-time image (see arrows 411 and 412 in FIG. 2 described later).

[0017] By using such an imaging support device, the inspector can also capture images at the time of inspection in the same / nearly the same position and orientation as when capturing images at the time of registration. Then, the imaging support device can generate high-quality images of the inspection target, so it can output defect candidate locations with high accuracy, and the appearance inspection work is made more efficient. Hereinafter, the position and orientation when capturing the normal-time image may be described as the first position and orientation, and the position and orientation at the time of appearance inspection may be described as the second position and orientation.

[0018] ≪Configuration of the Imaging Support Device≫ FIG. 1 is a functional block diagram of an imaging support device 100 according to the present embodiment. The imaging support device 100 is a computer and includes a control unit 110, a storage unit 120, and an input / output unit 180. A user interface device 820 such as a touch panel display is connected to the input / output unit 180. The input / output unit 180 may include a communication device and be capable of data transmission and reception with other devices. Also, a media drive may be connected to the input / output unit 180, and data exchange using a recording medium may be possible.

[0019] The input / output unit 180 is also connected to a camera 810 that captures the inspection target. The camera 810 may be connected using wireless communication. The camera 810 includes, for example, a receiving antenna of a global navigation satellite system, an acceleration sensor, and a gyro sensor, and can measure its own position and orientation. In addition to the captured image of the inspection target, the camera 810 transmits its own position and orientation (position and orientation) at the time of shooting to the shooting support device 100. Note that the camera 810 may capture a moving image as a series of still images, not limited to still images. An example of such a camera 810 is a smartphone.

[0020] ≪Shooting Support Device: Storage Unit≫ The storage unit 120 is configured to include storage devices such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). The storage unit 120 stores a position and orientation database 130, an image database 140, an image generation model 121, and a program 128. Note that various storage contents of the storage unit 120 may be read from an external storage device such as a cloud server as needed. The program 128 includes descriptions of the processing of functional units provided in the control unit 110 described later. Note that the storage unit 120 may be outside the configuration of the shooting support device 100.

[0021] ≪Storage Unit: Position and Orientation Database, Image Database≫ The position and orientation database 130 stores the position and orientation (first position and orientation) of the camera 810 when capturing an image of the inspection target in a normal state (normal-time image). The position and orientation are a three-dimensional coordinate indicating the position and a two-dimensional direction vector indicating the orientation, and are a five-dimensional numerical vector. The position and orientation can be regarded as a point in a position and orientation space, which is a five-dimensional numerical vector space. In the position and orientation space, a distance (for example, a geometric distance) is appropriately defined. The image database 140 stores the normal-time image in association with the position and orientation when the normal-time image was captured.

[0022] ≪Memory Unit: Image Generation Model≫ Image generation model 121 is a machine learning model trained and generated using training data in which position and orientation (see position and orientation database 130) is the explanatory variable and a normal image (see image database 140) corresponding to that position and orientation is the target variable (ground truth). By using image generation model 121, it is possible to generate an image of the inspection target taken at a given position and orientation under normal conditions, based on the position and orientation.

[0023] In the position-orientation space, the more position-orientation data from normal images is available near a specified position-orientation, the more the generated image will reflect the normal image and result in a higher quality image. High quality means that the evaluation function constructed during the generation of the image generation model calculates a high evaluation score. This can be considered as a high degree of similarity between the generated image and an image actually taken of the inspection target from the specified position-orientation. Conversely, if a position-orientation outside of the position-orientation data from normal images is specified, the generated image will be a low-resolution image or an image with low similarity to an image actually taken of the inspection target from the specified position-orientation.

[0024] As described above, the imaging support device 100 (as an external inspection support device) stores multiple first position and orientation data, which are the position and orientation of the camera 810 that captured an image of the object, in the storage unit 120 (see position and orientation database 130). The memory unit 120 also stores an image generation model 121, which is a machine learning model generated using training data in which the first position and orientation is used as the explanatory variable and the image of the object captured at the said first position and orientation is used as the target variable.

[0025] <<Image Capture Support Device: Control Unit>> The control unit 110 includes a CPU (Central Processing Unit) and comprises a receiving unit 111, a model generation unit 112, a determination unit 113, a decision unit 114, an image generation unit 115, an image comparison unit 116, a defect determination unit 117, and a display control unit 118. The control unit 110 may also include a GPU (Graphics Processing Unit), an NPU (Neural (network) Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc.

[0026] ≪Control Unit: Receiving Unit≫ The receiving unit 111 receives the image of the object to be inspected transmitted by the camera 810, as well as the position and orientation (first position and second position) at the time the image was taken. In the stage of registering the normal image of the object to be inspected, the receiving unit 111 stores the received position and orientation in the position and orientation database 130, and the image (normal image) associated with the position and orientation (first position and orientation) in the image database 140.

[0027] As described above, the imaging support device 100 includes a receiving unit 111 that receives a second position and orientation, which is the position and orientation of the camera 810 that has captured an image of the object. The receiving unit 111 receives the second position and orientation, along with the captured image, which is an image of the object taken at the second position and orientation.

[0028] ≪Control Unit: Model Generation Unit≫ The model generation unit 112 trains and generates an image generation model 121 using training data in which position and orientation are explanatory variables and normal images associated with said position and orientation are the target variable (ground truth).

[0029] ≪Control Unit: Determination Unit≫ The determination unit 113 compares the position and orientation of the camera 810 during the visual inspection (second position and orientation) with the position and orientation of the camera 810 during normal image registration (first position and orientation) to determine whether the position and orientation (second position and orientation) is appropriate. For example, the determination unit 113 determines that it is appropriate if there are a predetermined number or more of the position and orientations during normal image registration within a predetermined distance of the position and orientation during the visual inspection. Here, the predetermined distance is the distance within the position and orientation space, and is set by the user of the shooting support device along with the predetermined number.

[0030] The determination unit 113 may determine that the position is acceptable if the sum of the distances from the position and orientation at the time of visual inspection to a predetermined number of positions and orientations at the time of registration of normal images, starting from the closest position and orientation, is within a predetermined distance. The determination unit 113 may also determine compliance based on the distance between the position and orientation during the visual inspection and the position and orientation during normal image capture, and the number of positions and orientations during normal image capture that are within that distance. For example, compliance may be determined based on multiple distances and the number of positions and orientations during normal image capture, such as 3 or more positions and orientations during normal image capture within a distance of 5, and 8 or more within a distance of 10.

[0031] Thus, the determination unit 113 determines that the camera 810 is suitable if its position and orientation during normal image registration are distributed in the vicinity of the camera 810's position and orientation during the visual inspection in a manner that satisfies the above-described conditions. The settings for the predetermined distance and predetermined number used in the above determination will be described later.

[0032] As described above, the imaging support device 100 includes a determination unit 113 that determines whether the second positional orientation is appropriate according to the number of first positional orientations in the space including the first positional orientation and the second positional orientation that are within a predetermined distance from the second positional orientation. The determination unit 113 determines whether the second position orientation is appropriate based on the distance between the second position orientation and the first position orientation, and the number of first position orientations within that distance.

[0033] ≪Control Unit: Decision-Making Unit≫ The determination unit 114, if the judgment unit 113 determines that the camera 810 is suitable, finds a position and orientation that would make it suitable, and determines the direction in which to change the position and orientation to achieve that position and orientation. The display control unit 118, described later, outputs this direction to the display connected to the input / output unit 180. For example, the determination unit 114 finds a position and orientation that is closest to the camera 810 and that the judgment unit 113 determines as suitable (also referred to as the appropriate position and orientation), and determines the direction in which to change the position and orientation to approach the appropriate position and orientation (see arrows 411 and 412 in Figure 2, described later).

[0034] The determination unit 114 may pre-determine a region of position and orientation in which the judgment unit 113 determines suitability after the registration of normal images is complete. If the judgment unit 113 determines suitability is not suitable, the determination unit 114 may decide to change the position and orientation in a direction that brings it closer to that region. The determination unit 114 may also decide to change the position and orientation in a direction that results in a position and orientation that increases the evaluation function.

[0035] The destination to which the display control unit 118 outputs the direction is not limited to the display connected to the input / output unit 180. If the camera 810 is equipped with a display, the display control unit 118 may transmit the direction for changing the position and orientation to the camera 810 so that the camera 810 displays it on its own display.

[0036] Figure 2 is a screen configuration diagram of the visual inspection support screen 410 according to this embodiment. The captured image is displayed in area 413 of the visual inspection support screen 410. Incidentally, the captured image shown in Figure 2 is an image taken during a visual inspection of the underside of a train car body. The visual inspection support screen 410 is the screen displayed when the judgment of the judgment unit 113 is negative, and arrows 411 and 412 are displayed at the top of the visual inspection support screen 410 to guide the inspector to the appropriate position and posture. Arrow 411 indicates the direction of the appropriate position and instructs the inspector to move the camera 810 to the right. Arrow 412 indicates the direction of the appropriate posture (orientation) and instructs the inspector to point the camera 810 to the right (clockwise).

[0037] As described above, the imaging support device 100 includes a determination unit 114 that, when the determination unit 113 determines that the second position and orientation is unsuitable, determines the direction in which to change the position and orientation so as to approach the position and orientation that the determination unit 113 determines to be suitable, starting from the second position and orientation.

[0038] ≪Control Unit: Image Generation Unit≫ Returning to Figure 1, let's continue the explanation of the control unit 110. The image generation unit 115 generates an image using the image generation model 121 based on the position and orientation of the camera 810 during the visual inspection. The more of the position and orientations from when the normal image was registered that are close to the position and orientation during the visual inspection, the higher the quality of the generated image. An image is generated when the camera 810 is in an appropriate position and orientation. For this reason, the determination unit 113 is configured to set predetermined distances and numbers used in the determination so that it determines that a position and orientation that generates a high-quality image is appropriate.

[0039] ≪Control Unit: Image Comparison Unit≫ The image comparison unit 116 compares the image generated by the image generation unit 115 based on the position and orientation of the camera 810 during the visual inspection with the image captured by the camera 810 at that position and orientation, and calculates the difference in pixels. The image comparison unit 116 calculates the difference by comparing, for example, the RGB values ​​of the images. The image comparison unit 116 may also calculate the difference based on brightness, or it may calculate the difference by comparing images after image processing such as edge detection.

[0040] ≪Control Unit: Defect Detection Unit≫ The defect detection unit 117 determines the presence or absence of a defect based on the pixel difference calculated by the image comparison unit 116. The defect detection unit 117 determines that there is a defect in the area where the difference exists, for example, if the magnitude of consecutive differences (e.g., the number of pixels) is greater than or equal to a predetermined number. The display control unit 118 outputs the area in question to the display as a defective area (candidate defect area) (see area 421 in Figure 3 described later).

[0041] Figure 3 is a screen configuration diagram of the visual inspection support screen 420 according to this embodiment. The captured image is displayed in area 422 of the visual inspection support screen 420. The visual inspection support screen 420 is the screen displayed when a defect is determined to exist, and area 421 (candidate defect area) is displayed overlaid on the captured image as the location of the defect.

[0042] ≪Control Unit: Display Control Unit≫ The display control unit 118 (output control unit) outputs an appearance inspection support screen 410 that includes the direction in which to change the position and orientation of the camera 810 determined by the determination unit 114 (see arrows 411 and 412 in Figure 2). The display control unit 118 also outputs an appearance inspection support screen 420 that includes the area determined by the defect determination unit 117 to have a defect (see area 421 in Figure 3). The output destination is, for example, a display connected to the input / output unit 180.

[0043] As described above, the shooting support device 100 includes an image generation unit 115 that, when the suitability of the second position and orientation is deemed appropriate, uses the image generation model 121 to generate a generated image, which is an image of the object as it would appear if it were photographed in the second position and orientation. The shooting support device 100 also includes an image comparison unit 116 that compares the captured image with the generated image and calculates the difference.

[0044] Furthermore, the imaging support device 100 includes a defect determination unit 117 that determines that there is a defect in the region (see region 421) where the difference is located if the difference satisfies predetermined conditions. The imaging support device 100 includes an output control unit (display control unit 118) that outputs the direction determined by the determination unit 114. The output control unit outputs the region where there is a difference.

[0045] ≪Normal Image Registration Process≫ Figure 4 is a flowchart of the normal image registration process according to this embodiment. When registering a normal image, the inspector uses the camera 810 to photograph the normal visual inspection target (visual inspection point) from one or more directions. Note that there is not limited to one visual inspection point; if there are multiple visual inspection points on one piece of equipment, the inspector may photograph the visual inspection points one after another.

[0046] In step S11, the receiving unit 111 starts repeating steps S12 to S13 while the inspector is taking normal images of the visual inspection points with the camera 810. In step S12, the receiving unit 111 receives the captured image and the position and orientation of the camera 810 at the time the image was taken from the camera 810.

[0047] In step S13, the receiving unit 111 stores the position and orientation received in step S12 in the position and orientation database 130, and the captured image (normal image) associated with the position and orientation in the image database 140. In step S14, the model generation unit 112 trains and generates an image generation model 121 using training data in which the position and orientation in the image database 140 are used as explanatory variables and the normal image corresponding to the said position and orientation is used as the target variable (ground truth).

[0048] <<Visual Inspection Support Processing>> Figure 5 is a flowchart of the visual inspection support process according to this embodiment. During the visual inspection, the inspector uses the camera 810 to photograph the object to be inspected (visual inspection area) from one or more directions.

[0049] In step S21, the receiving unit 111 starts repeating steps S22 to S29 while the inspector is taking pictures of the visual inspection area with the camera 810. In step S22, the receiving unit 111 receives the captured image and the position and orientation of the camera 810 at the time the image was taken from the camera 810.

[0050] In step S23, the determination unit 113 compares the position and orientation of the camera 810 with the position and orientation at the time of normal image registration in the position and orientation database 130 to determine whether it is appropriate or not. If the determination unit 113 determines it is appropriate (step S23 → appropriate), it proceeds to step S26; if it determines it is not appropriate (step S23 → not appropriate), it proceeds to step S24.

[0051] In step S24, the determination unit 114 determines an appropriate position and orientation, and calculates the direction in which to change the position and orientation to approach that position and orientation. In step S25, the display control unit 118 outputs the direction calculated in step S24 (see Figure 2) to the display and returns the process to step S22.

[0052] In step S26, the image generation unit 115 generates an image using the image generation model 121 based on the position and orientation of the camera 810 received in step S22. In step S27, the image comparison unit 116 compares the image captured by the camera 810 (captured image) received in step S22 with the image generated in step S26 (generated image) and calculates the difference in pixels.

[0053] In step S28, the defect determination unit 117 determines whether or not there is a defect based on the difference calculated in step S27. If the defect determination unit 117 determines that there is a defect (step S28 → present), it proceeds to step S29; if it determines that there is no defect (step S28 → absent), it returns to step S22. In step S29, the display control unit 118 outputs the area on the captured image with defects (see Figure 3) to the display and returns the process to step S22.

[0054] Features of the imaging support device When the inspector takes a picture during inspection at a position (second position) that deviates from the position (first position) used when registering a normal image, the camera support device 100 instructs the inspector to take a picture at the same or a similar position as when registering a normal image. By changing the position of the camera 810 according to this instruction, the inspector can take a picture at the same or a similar position as when registering a normal image.

[0055] The imaging support device 100 generates an image of the object to be inspected based on the position and orientation of the camera 810, and by comparing this generated image with the image taken by the inspector, outputs the difference between the two images as a candidate for defects.

[0056] Inspectors are instructed to take photographs in the same or similar position and orientation as when registering normal images (see Figure 2). By taking photographs in this position and orientation, the photographing support device 100 can generate images similar to those of a normal inspection target taken in that position and orientation. Since the photographing support device 100 outputs defect candidates (see Figure 3) based on such images, the accuracy of the defect candidates is expected to be high, and the visual inspection work can be made more efficient. In addition, since the inspector can identify the location of the defect candidate (the area of ​​the photographed image in which the defect candidate is visible), they can directly visually check the location of the inspection target to confirm whether or not there is a defect.

[0057] ≪Modified Version: Shooting Support Device≫ The imaging support device 100 can be used not only for visual inspection, but also for detecting changes in the layout of a facility, or for monitoring / observing changes in the appearance of equipment, facilities, or manufactured goods. For such applications, an imaging support device comprising a receiving unit 111, a determination unit 113, and a decision unit 114 may be used.

[0058] ≪Differentiation: Camera≫ In the embodiment described above, the camera 810 is carried by the inspector to photograph the object to be visually inspected. Alternatively, a robot 500 (see Figure 6 below) may be equipped with a camera to photograph the object to be visually inspected instead of the inspector.

[0059] Figure 6 is an overall configuration diagram of a modified image capture system 50 according to this embodiment. The image capture system 50 comprises a robot 500 and an image capture support device 100. The robot 500 comprises a control unit 510, a communication unit 520, a sensor 530, and a camera 540. The robot 500 is equipped with wheels or tracks for movement, an arm to grasp the camera 540 and change its position and orientation, and actuators to drive the wheels / tracks and arm, but these are not shown in Figure 6.

[0060] The control unit 510, for example, includes a microprocessor and controls the robot 500. The communication unit 520 is equipped with communication equipment and can communicate data with the shooting support device 100. The data includes the captured image taken by the camera 540, the position and orientation of the camera 540 when the image was taken, and the direction in which the position and orientation will be changed to approach the appropriate position and orientation determined by the determination unit 114 of the shooting support device 100.

[0061] Sensor 530 is, for example, a receiving antenna for a global navigation satellite system, an accelerometer, or a gyroscope. Based on the measurement results of the sensors, the control unit 510 calculates the position and attitude of the camera 540 and transmits it to the shooting support device 100 along with the image of the object to be inspected captured by the camera 540. The display control unit 118 of the shooting support device 100 transmits the direction determined by the determination unit 114 to the robot 500.

[0062] In a visual inspection support system, which is a photographic system 50 that supports visual inspection and includes a photographic support device 100 and a robot 500, arrows 411 and 412 do not need to be displayed on the visual inspection support screen 410 (see Figure 2). The robot 500 (control unit 510) changes its posture (orientation) by moving itself and the camera 540 so that it becomes the appropriate position and posture calculated by the determination unit 114 while changing the object to be visually inspected.

[0063] The system does not necessarily require a single robot; it could include multiple robots, such as robots that move on the ground / floor or drones that fly. Using such a visual inspection support system can significantly reduce the labor involved in visual inspections. It also enables visual inspection of areas that would be dangerous to inspect manually.

[0064] As described above, the shooting system 50 includes a camera 540 and a robot 500 that photographs an object while changing the position and orientation of the camera 540. The imaging system 50 also includes a receiving unit 111, a determination unit 113, and a decision unit 114. Furthermore, the shooting system 50 includes an output control unit (display control unit 118) that outputs the direction determined by the determination unit 114 to the robot 500. The robot 500 changes the position and orientation of the camera 540 according to the direction (which changes the position and orientation output by the output control unit).

[0065] ≪Variantization: Judgment Section / Decision Section≫ In the embodiment described above, the determination unit 113 determines that a position or orientation during visual inspection (second position or orientation) is acceptable if a predetermined number of position or orientations during normal image registration (first position or orientation) are within a predetermined distance of that position or orientation. The determination unit 113 may also make a determination based on an evaluation score calculated based on the distance between the position or orientation during visual inspection and the position or orientation during normal image registration that is closest to that position or orientation.

[0066] For example, a distance evaluation model, which is a machine learning model, is constructed using training data in which the distance to the closest normal image registration location and location for various positions and orientations is used as the explanatory variable, and the evaluation score of the position and orientation calculated by the evaluation function during image generation of the image generation model 121 is used as the target variable. The input (explanatory variable) of this distance evaluation model is distance, and the output (target variable) is the evaluation score.

[0067] The determination unit 113 may determine compliance based on an evaluation score calculated using a distance evaluation model from the distance between the position and orientation at the time of visual inspection and the position and orientation at the time of registration of the closest normal image. For example, the appropriate distance is defined as the distance at which the evaluation score output by the distance evaluation model is equal to or greater than a predetermined evaluation score. The determination unit 113 may determine compliance if the distance between the position and orientation at the time of visual inspection and the position and orientation at the time of registration of the closest normal image is within the appropriate distance. The distance may also be the average of the distances to a predetermined number of position and orientations at the time of registration of normal images, starting from the closest one, instead of the distance to the closest position and orientation at the time of registration of the closest normal image.

[0068] The determination unit 113 may determine whether the position and orientation during the visual inspection are acceptable or not based on an evaluation score obtained from an evaluation function for the position and orientation. For example, if the evaluation score for the position and orientation is equal to or greater than a predetermined evaluation score, the position and orientation may be determined to be acceptable.

[0069] For example, if the position during visual inspection is shifted 20 cm to the left compared to the appropriate position, the determination unit 114 may decide to change the position so that the position during visual inspection is shifted 20 cm to the right. The display control unit 118 may also display the amount of change along with an arrow 411 (see Figure 2) indicating the direction of the appropriate position. The amount of change is the difference (distance) between the appropriate position and the position during visual inspection, for example, 20 cm.

[0070] For example, if the position and orientation during visual inspection is 45 degrees to the left compared to the appropriate position and orientation, the determination unit 114 may decide to change the orientation (direction) so that the position and orientation during visual inspection is 45 degrees to the right. The display control unit 118 may also display the amount of change along with the arrow 412 indicating the direction of the appropriate orientation. The amount of change is the difference (distance) between the appropriate position and orientation and the position and orientation during visual inspection, for example, 45 degrees.

[0071] As described above, the determination unit 113 determines whether the image is suitable or not based on the evaluation score of the second position and orientation, which is obtained using the evaluation function constructed when the image generation model 121 is generated. Furthermore, the determination unit 113 uses a distance evaluation model to determine whether the position is appropriate or not based on an evaluation score calculated based on the distance between the second position and attitude and the first position and attitude that is most recent to the second position and attitude. The distance evaluation model is a machine learning model generated using training data in which the distance between the second position and orientation and the most recent first position and orientation is used as the explanatory variable, and the evaluation score of the second position and orientation, using an evaluation function constructed during the generation of the image generation model 121, is used as the objective variable.

[0072] The determination unit 114 determines the amount of change to the second position and attitude based on the distance between the second position and attitude and the position and attitude that the judgment unit 113 determines to be appropriate.

[0073] <<Other variations>> Although several embodiments and modifications of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. For example, the imaging support device 100 includes a model generation unit 112 to generate an image generation model 121, but the image generation unit 115 may generate an image using an image generation model 121 generated by another device.

[0074] The external inspection support screen 410 (see Figure 2) displays arrows 411 and 412 indicating the direction to change the position and orientation of the camera 810 to the appropriate position and orientation. Alternatively, instead of directions, numerical values ​​(coordinates in position and orientation space) indicating the position and orientation may be displayed. Furthermore, while the imaging support device is configured to function as an "external inspection support device" to assist in inspecting the external appearance of an object, its use is not limited to that of an external inspection support device.

[0075] The present invention can take on various other embodiments, and furthermore, various modifications such as omissions and substitutions can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention as described herein, and are also included in the scope of the invention and its equivalents as described in the claims.

[0076] Hardware Configuration The imaging support device 100 according to the above embodiment is implemented by a computer 900 having a configuration such as that shown in Figure 7. Figure 7 is a hardware configuration diagram showing an example of a computer 900 that implements the functions of the imaging support device 100 according to the above embodiment. The computer 900 includes a CPU 901, ROM 902, RAM 903, SSD 904, input / output interface 905 (labeled as input / output I / F (Interface) in Figure 7), communication interface 906 (labeled as communication I / F in Figure 7), and media interface 907 (labeled as media I / F in Figure 7). The computer 900 may be equipped with an HDD (Hard Disc Drive) instead of the SSD 904, or it may be equipped with an HDD in addition to the SSD 904.

[0077] The CPU 901 operates based on programs stored in the ROM 902 or SSD 904 and is controlled by the control unit 110 in Figure 1. The ROM 902 stores boot programs executed by the CPU 901 when the computer 900 starts up, as well as programs related to the computer 900's hardware.

[0078] The CPU 901 controls input devices 910, such as a mouse and keyboard, and output devices 911, such as a display and printer, via the input / output interface 905. The CPU 901 acquires data from the input devices 910 and outputs the generated data to the output devices 911 via the input / output interface 905.

[0079] SSD904 stores programs executed by CPU901 and data used by those programs. Communication interface906 receives data from other devices (e.g., camera810) not shown via the communication network and outputs it to CPU901, and also transmits data generated by CPU901 to other devices via the communication network.

[0080] The media interface 907 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads the program from the recording medium 912 onto the RAM 903 via the media interface 907 and executes the loaded program. The recording medium 912 can be an optical recording medium such as a DVD (Digital Versatile Disk), a magneto-optical recording medium such as an MO (Magneto Optical Disk), a magnetic recording medium, a conductive memory tape medium, or a semiconductor memory.

[0081] For example, when the computer 900 functions as the imaging support device 100 according to the above embodiment, the CPU 901 of the computer 900 realizes the functions of the imaging support device 100 by executing the program 128 (see Figure 1) loaded on the RAM 903. The CPU 901 reads the program from the recording medium 912 and executes it. In addition, the CPU 901 may read the program from another device via a communication network, or it may install the program 128 from the recording medium 912 to the SSD 904 and execute it. [Explanation of Symbols]

[0082] 100 Imaging support device 111 Receiving Unit 112 Model Generation Unit 113 Judgment section 114 Decision Section 115 Image generation unit 116 Image Comparison Section 117 Defect detection unit 118 Display Control Unit (Output Control Unit) 121 Image Generation Models 130 Position and Orientation Database 140 Image Database 50 shooting systems 500 robots 540 Camera 810 Camera

Claims

1. A shooting support device that stores multiple first position and orientation values, which are the position and orientation of a camera that has captured an image of an object, in a storage unit, and uses these first position and orientation values ​​to support the shooting of the object, A receiving unit that receives a second position and orientation, which is the position and orientation of the camera that captured the aforementioned object, A determination unit that determines whether the second positional orientation is appropriate according to the number of first positional orientations in the space including the first positional orientation and the second positional orientation that are within a predetermined distance from the second positional orientation, If the determination unit determines that the second position and orientation is unsuitable, the determination unit determines the direction in which to change the position and orientation so as to approach the position and orientation that the determination unit determines to be suitable, The system comprises an output control unit that outputs the direction determined by the determination unit. A device to assist in photography.

2. The aforementioned storage unit is The system further stores an image generation model, which is a machine learning model generated using training data in which the first position and orientation is used as the explanatory variable and the image of the object captured at the said first position and orientation is used as the target variable. The receiving unit is Along with the second position and orientation, the system receives the captured image, which is an image of the object taken in the second position and orientation. If the suitability of the second position and orientation is deemed appropriate, an image generation unit generates a generated image, which is an image of the object taken at the second position and orientation, using the image generation model. An image comparison unit that compares the captured image and the generated image and calculates the difference, The system further includes a defect determination unit that determines that there is a defect in the region where the difference exists if the difference satisfies predetermined conditions, The output control unit outputs the region where the difference exists. The imaging support device according to claim 1.

3. The determination unit, The suitability of the second position and orientation evaluation points is determined based on the evaluation function constructed during the generation of the aforementioned image generation model. The imaging support device according to claim 2.

4. The determination unit, Using a distance evaluation model, suitability is determined according to an evaluation score calculated based on the distance between the second position and attitude and the first position and attitude that is most recent to the second position and attitude. The aforementioned distance evaluation model, This is a machine learning model generated using training data in which the distance between the second position / orientation and the first position / orientation that is most recent to the second position / orientation is used as an explanatory variable, and the evaluation score of the second position / orientation, using an evaluation function constructed during the generation of the image generation model, is used as the objective variable. The imaging support device according to claim 2.

5. The determination unit, The suitability of the second positional attitude is determined based on the distance between the second positional attitude and the first positional attitude, and the number of first positional attitudes within that distance. The imaging support device according to claim 1.

6. The aforementioned determination unit, Based on the distance between the second position and attitude and the position and attitude that the determination unit determines to be appropriate, the amount of change in the second position and attitude is determined. The imaging support device according to claim 4 or 5.

7. A shooting system in which a plurality of first position and orientation values, which are the position and orientation of a camera that has captured an image of an object, are stored in a memory unit, and the system uses the first position and orientation values ​​to support the shooting of the object, A robot equipped with a camera that photographs the object while changing the position and orientation of the camera, A receiving unit that receives a second position and orientation, which is the position and orientation of the camera that captured the aforementioned object, A determination unit that determines whether the second positional orientation is appropriate according to the number of first positional orientations in the space including the first positional orientation and the second positional orientation that are within a predetermined distance from the second positional orientation, If the determination unit determines that the second position and orientation is unsuitable, the determination unit determines the direction in which to change the position and orientation so as to approach the position and orientation that the determination unit determines to be suitable, The system includes an output control unit that outputs the direction determined by the determination unit to the robot, The robot changes the position and orientation of the camera according to the direction. Filming system.

8. Multiple first position and orientation values, which are the position and orientation of the camera that captured the image of the object, are stored in the memory unit, and a shooting support device that uses these first position and orientation values ​​to support the shooting of the object, The steps include receiving a second position and orientation, which is the position and orientation of the camera that captured the aforementioned object, A step of determining whether the second positional attitude is appropriate in a space including the first positional attitude and the second positional attitude, according to the number of first positional attitudes that are within a predetermined distance from the second positional attitude, If the second position and attitude is determined to be unsuitable, the process involves determining a direction to change the position and attitude to approach the position and attitude determined to be suitable, starting from the second position and attitude. The step of outputting the aforementioned direction and the step of performing the following Filming support methods.

Images