Imaging support device, imaging system, and imaging support method

Abstract

An imaging support device (100) supports imaging of an object by using a plurality of first positions / postures, which are the positions and postures of a camera that has captured an image of the object and which are stored in a position / posture database (130). The imaging support device (100) comprises: a reception unit (111) that receives a second position / posture, which is the position and posture of the camera that has captured an image of the object; a determination unit (113) that determines suitability of the second position / posture in accordance with the number of the first positions / postures in which the distance from the second position / posture is within a predetermined distance in a space including the first positions / postures and the second position / posture; a decision unit (114) that, when the suitability of the second position / posture is negative, determines a direction in which the position and posture are changed so as to approach the position / posture determined to be suitable by the determination unit (113), with the second position / posture as a starting point; and an output control unit (display control unit (118)) that outputs the direction.

Description

Imaging Support Device, Imaging System, and Imaging Support Method

[0001] The present invention relates to an imaging support device, an imaging system, and an imaging support method.

[0002] In traffic infrastructure such as roads and railways, and facilities with a long design life such as plants, inspections and maintenance are important. In such facilities, visual inspections are performed to visually confirm the presence or absence of appearance defects such as discoloration, damage, rust, and corrosion on the equipment surface, and to determine the degree of influence on the equipment. In large-scale infrastructure facilities, the inspection targets (inspection locations) are diverse, and the appearance defects for each target also vary. Furthermore, defect discrimination requires skilled know-how. Therefore, there is a need for a method to improve the efficiency of visual inspections of such infrastructure facilities.

[0003] As an invention for inspecting defects in an inspection object, there is an inspection device described in Patent Document 1. This inspection device includes an acquisition unit that acquires a plurality of captured images obtained by capturing an inspection object from a plurality of angles, an assignment unit that assigns a label corresponding to the corresponding angle to each of the captured images, an encoder that inputs the captured image as an input image and the corresponding label to generate a latent variable, and a decoder that inputs the label and the latent variable to generate a restored image, and a learning processing unit that learns a neural network so that the input image and the restored image are the same. During the inspection of the inspection object, an inspection processing unit that performs an inspection process for defects in the inspection object based on the difference between the restored inspection image, which is the restored image obtained by inputting the captured image as an inspection image and the corresponding label to the neural network, and the inspection image.

[0004] Japanese Patent Application Laid-Open No. 2022-123733

[0005] According to the inspection device described in Patent Document 1, defects in the object to be inspected / checked can be inspected with high accuracy using captured images. However, it is difficult to acquire training / learning image data and labels from all directions (shooting position and orientation, position and orientation) for a wide variety of objects to be inspected / checked. It is desirable that inspection / checking be possible based on training / learning images taken at shooting positions and orientations within the range of visual inspection.

[0006] In image generation using machine learning techniques, if the position and orientation of the image taken during inspection / checking deviate from the position and orientation of the training image taken under normal conditions, the reconstructed inspection image will be blurry or the generation will fail. This makes it impossible to compare the captured image with the reconstructed inspection image. Therefore, it is desirable to take the image during inspection / checking at a position and orientation similar to the position and orientation when the object being inspected / checked is under normal conditions. The desire to take multiple images taken at the same or similar position and orientation when there is a period of time between them is not limited to inspection / checking. For example, it is desirable to take images at a similar position and orientation when detecting changes in the layout of a facility or when monitoring / observing changes in the appearance of equipment, facilities, or manufactured goods.

[0007] This invention was made in view of the above background, and aims to provide a shooting support device, shooting system, and shooting support method that assist in determining the position and orientation for photographing changes in the appearance of an object.

[0008] To solve the above-mentioned problems, the present invention provides a shooting support device in which a plurality of first positional orientations, which are the position and orientation of a camera that has captured an image of an object, are stored in a storage unit, and the shooting support device provides support for shooting the object using the first positional orientations, comprising: a receiving unit that receives a second positional orientation, which is the position and orientation of a camera that has captured an image of an object; a determination unit that determines whether the second positional orientation is appropriate or not according to the number of first positional orientations in a space including the first positional orientation and the second positional orientation that are within a predetermined distance from the second positional orientation; a determination unit that, when the determination unit determines that the second positional orientation is not appropriate, determines a direction in which the position and orientation will be changed to approach a positional orientation that the determination unit determines to be appropriate, starting from the second positional orientation; and an output control unit that outputs the direction determined by the determination unit.

[0009] According to the present invention, it is possible to provide a shooting support device, shooting system, and shooting support method that assist in determining the position and orientation for photographing changes in the appearance of an object. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

[0010] This is a functional block diagram of the shooting support device according to this embodiment. This is a screen configuration diagram of the visual inspection support screen according to this embodiment. This is a screen configuration diagram of the visual inspection support screen according to this embodiment. This is a flowchart of the normal image registration process according to this embodiment. This is a flowchart of the visual inspection support process according to this embodiment. This is an overall configuration diagram of the shooting system according to a modified example of this embodiment. This is a hardware configuration diagram showing an example of a computer that realizes the functions of the shooting support device according to the above embodiment.

[0011] <<Overview of the Photography Support Device>> The overview of the photography support device in an embodiment for carrying out the present invention is described below. In this embodiment, the photography support device is configured to function as an "external inspection support device" that assists in inspecting the external appearance of an object, but details of the photography support device, including its configuration and functions, will be described later.

[0012] The inspector, who is a user of the camera support device, takes a picture of the inspection target (inspection point) under normal conditions and records (registers) the image as a normal image in the camera support device, which is an external inspection support device. At this time, the camera support device also records (registers) the position and orientation of the camera used to take the picture, in addition to the captured image (normal image).

[0013] Once the registration of normal images and the position and orientation of the camera is complete, the camera support device generates an image generation model, which is a machine learning model, using training data in which the explanatory variable is the position and orientation of the camera and the target variable is the image taken at that position and orientation. By using this image generation model, the camera support device can generate an image of the inspection target taken from that position and orientation based on the position and orientation.

[0014] When an inspector photographs the object being inspected during an inspection, the photography support device generates an image using an image generation model based on the position and orientation of the photograph. This image is then compared with the captured image, and any differing areas are output as potential defect locations (see area 421 in Figure 3 below).

[0015] Images generated by the image generation model are of high quality when the position and orientation distribution (distribution density) is high in the region where the normal image was registered. High quality means that the evaluation score calculated by the evaluation function constructed during the generation of the image model is high for that position and orientation. It can also be said that the similarity between the image taken of the inspection target from that position and orientation and the generated image is high. However, images generated from positions and orientations in regions with low or no distribution are of low quality. In other words, during inspection, it is required to take the image from the same or a similar position and orientation as when the normal image was registered.

[0016] To address this, the imaging support device guides the inspector to take a photograph at the same or a similar position and orientation as when the normal image was registered. More specifically, if the inspector takes a photograph at a position and orientation different from that of the normal image registration, the imaging support device will indicate the direction of the same or a similar position and orientation as when the normal image was registered (see arrows 411 and 412 in Figure 2 below).

[0017] By using such a camera support device, inspectors can take images during inspection at the same or a similar position and orientation as when they took the images during registration. This allows the camera support device to generate high-quality images of the inspection target, enabling it to output defect candidate locations with high accuracy and streamline the visual inspection process. In the following, the position and orientation when taking images under normal conditions will be referred to as the first position and orientation, and the position and orientation during visual inspection will be referred to as the second position and orientation.

[0018] ≪Configuration of the Imaging Support Device≫ Figure 1 is a functional block diagram of the imaging support device 100 according to this embodiment. The imaging support device 100 is a computer and comprises a control unit 110, a storage unit 120, and an input / output unit 180. A user interface device 820, such as a touch panel display, is connected to the input / output unit 180. The input / output unit 180 may also be equipped with a communication device, enabling data transmission and reception with other devices. Furthermore, a media drive may be connected to the input / output unit 180, enabling data exchange using a recording medium.

[0019] A camera 810 for photographing the object to be inspected is also connected to the input / output unit 180. The camera 810 may be connected using wireless communication. The camera 810 may be equipped with, for example, a receiving antenna for a global navigation satellite system, an accelerometer, and a gyroscope, and its own position and attitude can be measured. In addition to the image of the object to be inspected, the camera 810 transmits its own position and attitude (position and attitude) at the time of shooting to the shooting support device 100. The camera 810 is not limited to still images; it may also shoot video as a series of still images. An example of such a camera 810 is a smartphone.

[0020] ≪Image Capture Support Device: Memory Unit≫ The memory unit 120 is composed of memory devices such as ROM (Read Only Memory), RAM (Random Access Memory), and SSD (Solid State Drive). The memory unit 120 stores a position and orientation database 130, an image database 140, an image generation model 121, and a program 128. The various contents of the memory unit 120 may be stored in an external storage device such as a cloud server and read as needed. The program 128 includes a description of the processing of the functional unit provided in the control unit 110, which will be described later. The memory unit 120 may be located outside the configuration of the image capture support device 100.

[0021] ≪Storage Unit: Position and Orientation Database and Image Database≫ The position and orientation database 130 stores the position and orientation (first position and orientation) of the camera 810 when it took an image of the inspection target under normal conditions (normal image). The position and orientation is a 5-dimensional numerical vector consisting of a 3D coordinate indicating position and a 2D orientation vector indicating orientation. The position and orientation can be considered as a point in the position and orientation space, which is a 5-dimensional numerical vector space. It is assumed that distances (e.g., geometric distances) are appropriately defined in the position and orientation space. The image database 140 stores normal images associated with the position and orientation at the time the normal image was taken.

[0022] <<Memory Unit: Image Generation Model>> The image generation model 121 is a machine learning model trained and generated using training data in which position and orientation (see position and orientation database 130) is the explanatory variable and a normal image (see image database 140) corresponding to that position and orientation is the target variable (ground truth). By using the image generation model 121, it is possible to generate an image of the inspection target taken at that position and orientation under normal conditions, based on the position and orientation.

[0023] In the position-orientation space, the more position-orientation data from normal images is available near a specified position-orientation, the more the generated image will reflect the normal image and result in a higher quality image. High quality means that the evaluation function constructed during the generation of the image generation model calculates a high evaluation score. This can be considered as a high degree of similarity between the generated image and an image actually taken of the inspection target from the specified position-orientation. Conversely, if a position-orientation outside of the position-orientation data from normal images is specified, the generated image will be a low-resolution image or an image with low similarity to an image actually taken of the inspection target from the specified position-orientation.

[0024] As described above, the imaging support device 100 (as an external inspection support device) stores a plurality of first position and orientation data, which are the position and orientation of the camera 810 that captured an image of the object, in the storage unit 120 (see position and orientation database 130). The storage unit 120 also stores an image generation model 121, which is a machine learning model generated using training data in which the first position and orientation data are explanatory variables and the image of the object captured at the said first position and orientation data are the target variable.

[0025] <<Image Capture Support Device: Control Unit>> The control unit 110 is configured to include a CPU (Central Processing Unit) and comprises a receiving unit 111, a model generation unit 112, a determination unit 113, a decision unit 114, an image generation unit 115, an image comparison unit 116, a defect determination unit 117, and a display control unit 118. The control unit 110 may also be configured to include a GPU (Graphics Processing Unit), an NPU (Neural (network) Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc.

[0026] <Control Unit: Receiving Unit> The receiving unit 111 receives the image of the object to be inspected transmitted by the camera 810, as well as the position and orientation (first position and second position) at the time the image was taken. When registering the normal image of the object to be inspected, the receiving unit 111 stores the received position and orientation in the position and orientation database 130, and the image (normal image) associated with the position and orientation (first position and orientation) in the image database 140.

[0027] As described above, the shooting support device 100 includes a receiving unit 111 that receives a second position and orientation, which is the position and orientation of the camera 810 that captured an image of the object. The receiving unit 111 receives the second position and orientation along with the captured image, which is an image of the object captured at the second position and orientation.

[0028] <Control Unit: Model Generation Unit> The model generation unit 112 trains and generates an image generation model 121 using training data in which position and orientation are explanatory variables and normal images associated with the position and orientation are the target variable (correct answer).

[0029] ≪Control Unit: Determination Unit≫ The determination unit 113 compares the position and orientation of the camera 810 during the visual inspection (second position and orientation) with the position and orientation of the camera 810 during normal image registration (first position and orientation) to determine whether the position and orientation (second position and orientation) is appropriate. For example, the determination unit 113 determines that it is appropriate if there are a predetermined number or more of the position and orientations during normal image registration within a predetermined distance of the position and orientation during the visual inspection. Here, the predetermined distance is the distance within the position and orientation space, and is set by the user of the shooting support device along with the predetermined number.

[0030] The determination unit 113 may determine compliance if the sum of the distances between the position and orientation at the time of visual inspection and a predetermined number of position and orientation at the time of normal image registration, starting from the closest position and orientation, is within a predetermined distance. Alternatively, the determination unit 113 may determine compliance based on the distance between the position and orientation at the time of visual inspection and the position and orientation at the time of normal image capture, and the number of position and orientation at the time of normal image capture that are within that distance. For example, compliance may be determined based on multiple distances and the number of position and orientation at the time of normal image capture, such as 3 or more position and orientation at the time of normal image capture within a distance of 5, and 8 or more within a distance of 10.

[0031] Thus, the determination unit 113 determines that the camera is suitable if the position and orientation of the camera 810 during normal image registration are distributed in the vicinity of the position and orientation of the camera 810 during the visual inspection in a manner that satisfies the above-described conditions. The setting of the predetermined distance and predetermined number used in the above determination will be described later.

[0032] As described above, the imaging support device 100 includes a determination unit 113 that determines whether the second positional posture is suitable based on the number of first positional postures in the space including the first and second positional postures that are within a predetermined distance from the second positional posture. The determination unit 113 determines whether the second positional posture is suitable based on the distance between the second positional posture and the first positional posture, and the number of first positional postures that are within that distance.

[0033] ≪Control Unit: Determination Unit≫ The determination unit 114, when the judgment unit 113 determines whether the camera 810 is suitable, finds a position and orientation that would make it suitable, and determines the direction in which to change the position and orientation to achieve that position and orientation. The display control unit 118, described later, outputs this direction to the display connected to the input / output unit 180. For example, the determination unit 114 finds a position and orientation that is closest to the camera 810 and that would result in a suitable judgment from the judgment unit 113 (also referred to as the appropriate position and orientation), and determines the direction in which to change the position and orientation to approach the appropriate position and orientation (see arrows 411 and 412 in Figure 2, described later).

[0034] The determination unit 114 may pre-determine a region of position and orientation in which the judgment unit 113 determines suitability after the registration of a normal image is complete. If the judgment unit 113 determines suitability is not suitable, the determination unit 114 may decide to change the position and orientation in a direction that approaches that region. The determination unit 114 may also decide to change the position and orientation in a direction that results in a position and orientation that increases the evaluation function.

[0035] The destination to which the display control unit 118 outputs the direction is not limited to the display connected to the input / output unit 180. If the camera 810 is equipped with a display, the display control unit 118 may transmit the direction for changing the position and orientation to the camera 810 so that the camera 810 displays it on its own display.

[0036] Figure 2 is a screen configuration diagram of the visual inspection support screen 410 according to this embodiment. The captured image is displayed in area 413 of the visual inspection support screen 410. Incidentally, the captured image shown in Figure 2 is an image taken during a visual inspection of the underside of a train car body. The visual inspection support screen 410 is the screen displayed when the judgment of the judgment unit 113 is negative, and arrows 411 and 412 are displayed at the top of the visual inspection support screen 410 to guide the inspector to the appropriate position and posture. Arrow 411 indicates the direction of the appropriate position and instructs the inspector to move the camera 810 to the right. Arrow 412 indicates the direction of the appropriate posture (orientation) and instructs the inspector to point the camera 810 to the right (clockwise).

[0037] As described above, the imaging support device 100 includes a determination unit 114 that, when the determination unit 113 determines that the second position and orientation is unsuitable, determines the direction in which to change the position and orientation so as to approach the position and orientation that the determination unit 113 determines to be suitable, starting from the second position and orientation.

[0038] ≪Control Unit: Image Generation Unit≫ Returning to Figure 1, we continue the explanation of the control unit 110. The image generation unit 115 generates images using the image generation model 121 based on the position and orientation of the camera 810 during the visual inspection. The more of the position and orientation during normal image registration that are close to the position and orientation during the visual inspection, the higher the quality of the generated image. Images are generated when the camera 810 is in an appropriate position and orientation. For this reason, predetermined distances and numbers used in the determination are set so that the determination unit 113 determines that a position and orientation that generates a high-quality image is appropriate.

[0039] <Control Unit: Image Comparison Unit> The image comparison unit 116 compares the image generated by the image generation unit 115 based on the position and orientation of the camera 810 during the visual inspection with the image captured by the camera 810 at that position and orientation, and calculates the difference in pixels. The image comparison unit 116 calculates the difference by comparing, for example, the RGB values ​​of the images. The image comparison unit 116 may also calculate the difference based on brightness, or it may calculate the difference by comparing images after image processing such as edge detection.

[0040] <<Control Unit: Defect Judgment Unit>> The defect judgment unit 117 judges the presence or absence of a defect based on the pixel difference calculated by the image comparison unit 116. For example, if the magnitude of consecutive differences (e.g., the number of pixels) is a predetermined number or more, the defect judgment unit 117 judges that there is a defect in the area where the difference exists. The display control unit 118 outputs the area as a defective location (defect candidate location) to the display (see area 421 in FIG. 3 described later).

[0041] FIG. 3 is a screen configuration diagram of the appearance inspection support screen 420 according to the present embodiment. The captured image is displayed in the area 422 of the appearance inspection support screen 420. The appearance inspection support screen 420 is a screen when it is judged that there is a defect, and the area 421 (defect candidate location) as the defective location is displayed superimposed on the captured image.

[0042] <<Control Unit: Display Control Unit>> The display control unit 118 (output control unit) outputs an appearance inspection support screen 410 including the direction (see arrows 411 and 412 shown in FIG. 2) for changing the position and orientation of the camera 810 determined by the determination unit 114. Further, the display control unit 118 outputs an appearance inspection support screen 420 including the location judged by the defect judgment unit 117 to have a defect (see area 421 shown in FIG. 3). The output destination is, for example, a display connected to the input / output unit 180.

[0043] As described above, when the suitability of the second position and orientation is appropriate, the imaging support device 100 includes an image generation unit 115 that generates a generated image, which is an image of the object when photographed in the second position and orientation, using the image generation model 121. Further, the imaging support device 100 includes an image comparison unit 116 that compares the captured image and the generated image and calculates the difference.

[0044] Further, when the difference satisfies a predetermined condition, the imaging support device 100 includes a defect judgment unit 117 that judges that there is a defect in the area where the difference exists (see area 421). The imaging support device 100 includes an output control unit (display control unit 118) that outputs the direction determined by the determination unit 114. The output control unit outputs the area where the difference exists.

[0045] <<Normal Image Registration Process>> Figure 4 is a flowchart of the normal image registration process according to this embodiment. During normal image registration, the inspector uses the camera 810 to capture a normal appearance inspection target (appearance inspection location) from one or more directions. Note that the appearance inspection location is not necessarily limited to one location. If there are multiple appearance inspection locations on one facility, the inspector may capture the appearance inspection locations one by one.

[0046] In step S11, the receiving unit 111 starts a process of repeating steps S12 to S13 while the inspector is capturing a normal image of the appearance inspection location with the camera 810. In step S12, the receiving unit 111 receives the captured image from the camera 810 and the position and orientation of the camera 810 at the time when the captured image is captured.

[0047] In step S13, the receiving unit 111 stores the position and orientation received in step S12 in the position and orientation database 130, and stores the captured image (normal image) associated with the position and orientation in the image database 140. In step S14, the model generation unit 112 trains and generates the image generation model 121 using the learning data with the position and orientation in the image database 140 as the explanatory variable and the normal image corresponding to the position and orientation as the objective variable (correct answer).

[0048] <<Appearance Inspection Support Process>> Figure 5 is a flowchart of the appearance inspection support process according to this embodiment. During appearance inspection, the inspector uses the camera 810 to capture the appearance inspection target (appearance inspection location) from one or more directions.

[0049] In step S21, the receiving unit 111 starts a process of repeating steps S22 to S29 while the inspector is capturing the appearance inspection location with the camera 810. In step S22, the receiving unit 111 receives the captured image from the camera 810 and the position and orientation of the camera 810 at the time when the captured image is captured.

[0050] In step S23, the determination unit 113 compares the position and orientation of the camera 810 with the position and orientation at the time of normal image registration in the position and orientation database 130 to determine whether it is appropriate or not. If the determination unit 113 determines it is appropriate (step S23 → appropriate), it proceeds to step S26; if it determines it is not appropriate (step S23 → not appropriate), it proceeds to step S24.

[0051] In step S24, the determination unit 114 determines an appropriate position and orientation and calculates the direction in which to change the position and orientation to approach that position and orientation. In step S25, the display control unit 118 outputs the direction calculated in step S24 (see Figure 2) to the display and returns the process to step S22.

[0052] In step S26, the image generation unit 115 generates an image using the image generation model 121 based on the position and orientation of the camera 810 received in step S22. In step S27, the image comparison unit 116 compares the image captured by the camera 810 (captured image) received in step S22 with the image generated in step S26 (generated image) and calculates the difference in pixels.

[0053] In step S28, the defect determination unit 117 determines whether or not there is a defect based on the difference calculated in step S27. If the defect determination unit 117 determines that there is a defect (step S28 → present), it proceeds to step S29; if it determines that there is no defect (step S28 → absent), it returns to step S22. In step S29, the display control unit 118 outputs the area on the captured image with the defect (see Figure 3) to the display and returns to step S22.

[0054] ≪Features of the Shooting Support Device≫ When an inspector takes a picture during inspection at a position (second position) that deviates from the position (first position) used when registering a normal image, the shooting support device 100 will indicate the direction of the position (second) that is the same as or close to the position (first) used when registering a normal image. By changing the position (second) of the camera 810 according to this instruction, the inspector will be able to take a picture at the same or close to the position (second) that was used when registering a normal image.

[0055] The imaging support device 100 generates an image of the object to be inspected based on the position and orientation of the camera 810, and by comparing this generated image with the image taken by the inspector, outputs the difference between the two images as a candidate for defect.

[0056] Inspectors are instructed to take photographs in the same or similar position and orientation as when registering normal images (see Figure 2). By taking photographs in this position and orientation, the photographing support device 100 can generate images similar to those of a normal inspection target taken in that position and orientation. Since the photographing support device 100 outputs defect candidates (see Figure 3) based on such images, the accuracy of the defect candidates is expected to be high, and the visual inspection work can be made more efficient. In addition, since the inspector can identify the location of the defect candidate (the area of ​​the photographed image in which the defect candidate is visible), they can directly visually check the location of the inspection target to confirm whether or not there is a defect.

[0057] <<Modification: Photographing support device>> The photographing support device 100 can be used not only for visual inspection, but also for detecting changes in the layout of a facility, or for monitoring / observing changes in the appearance of equipment, facilities, or manufactured goods. For such applications, a photographing support device comprising a receiving unit 111, a determination unit 113, and a decision unit 114 may be used.

[0058] <<Modification: Camera>> In the embodiment described above, the camera 810 is carried by the inspector to photograph the object to be visually inspected. Alternatively, a robot 500 (see Figure 6 below) may be equipped with a camera to photograph the object to be visually inspected instead of the inspector.

[0059] Figure 6 is an overall configuration diagram of a modified image capture system 50 according to this embodiment. The image capture system 50 comprises a robot 500 and an image capture support device 100. The robot 500 comprises a control unit 510, a communication unit 520, a sensor 530, and a camera 540. The robot 500 is equipped with wheels or tracks for movement, an arm to grasp the camera 540 and change its position and orientation, and actuators to drive the wheels / tracks and arm, but these are not shown in Figure 6.

[0060] The control unit 510 is configured, for example, to include a microprocessor and controls the robot 500. The communication unit 520 is equipped with communication equipment and can communicate data with the shooting support device 100. The data includes the image captured by the camera 540, the position and orientation of the camera 540 when the image was captured, and the direction in which the position and orientation will be changed to approach the appropriate position and orientation determined by the determination unit 114 of the shooting support device 100.

[0061] Sensor 530 is, for example, a receiving antenna for a global navigation satellite system, an acceleration sensor, or a gyroscope. Based on the measurement results of the sensors, the control unit 510 calculates the position and orientation of the camera 540 and transmits it to the shooting support device 100 along with the image of the object to be inspected taken by the camera 540. The display control unit 118 of the shooting support device 100 transmits the direction determined by the determination unit 114 to the robot 500.

[0062] In a visual inspection support system, which is a photographic system 50 that supports visual inspection and includes a photographic support device 100 and a robot 500, the arrows 411 and 412 do not need to be displayed on the visual inspection support screen 410 (see Figure 2). The robot 500 (control unit 510) changes its posture (orientation) by moving itself and the camera 540 so that they become the appropriate position and posture calculated by the determination unit 114 while changing the object to be visually inspected.

[0063] The system does not necessarily require just one robot; it could include multiple robots, such as robots that move on the ground / floor or drones that fly. Using such a visual inspection support system is expected to reduce the labor involved in visual inspections. It also enables visual inspection of areas that would be dangerous to inspect manually.

[0064] As described above, the shooting system 50 includes a camera 540 and a robot 500 that photographs an object while changing the position and orientation of the camera 540. The shooting system 50 also includes a receiving unit 111, a determination unit 113, and a decision unit 114. Furthermore, the shooting system 50 includes an output control unit (display control unit 118) that outputs the direction determined by the decision unit 114 to the robot 500. The robot 500 changes the position and orientation of the camera 540 according to the direction (which changes the position and orientation output by the output control unit).

[0065] ≪Modified Version: Judgment Unit / Decision Unit≫ In the above embodiment, the judgment unit 113 determines that a position and orientation during visual inspection (second position and orientation) is acceptable if a predetermined number of position and orientations during normal image registration (first position and orientation) are within a predetermined distance of the said position and orientation. The judgment unit 113 may also make a determination based on an evaluation score calculated based on the distance between the position and orientation during visual inspection and the position and orientation during normal image registration that is closest to the said position and orientation.

[0066] For example, a distance evaluation model, which is a machine learning model, is constructed using training data in which the distance to the closest normal image registration location and location for various positions and orientations is used as the explanatory variable, and the evaluation score of the position and orientation calculated by the evaluation function during image generation of the image generation model 121 is used as the target variable. The input (explanatory variable) of this distance evaluation model is distance, and the output (target variable) is the evaluation score.

[0067] The determination unit 113 may determine compliance based on an evaluation score calculated using a distance evaluation model from the distance between the position and orientation at the time of visual inspection and the position and orientation at the time of normal image registration that is closest to that position and orientation. For example, the appropriate distance is defined as the distance at which the evaluation score output by the distance evaluation model is equal to or greater than a predetermined evaluation score. The determination unit 113 may determine compliance if the distance between the position and orientation at the time of visual inspection and the position and orientation at the time of normal image registration that is closest to that position and orientation is within the appropriate distance. The distance may also be the average of the distances to a predetermined number of position and orientations at the time of normal image registration, starting from the closest one, instead of the distance to the position and orientation at the time of normal image registration that is closest.

[0068] The determination unit 113 may determine whether the position and orientation during the visual inspection are acceptable or not based on an evaluation score obtained from an evaluation function for the position and orientation. For example, if the evaluation score for the position and orientation is equal to or greater than a predetermined evaluation score, the position and orientation may be determined to be acceptable.

[0069] For example, if the position during visual inspection is shifted 20 cm to the left compared to the appropriate position, the determination unit 114 may decide to change the position so that the position during visual inspection is shifted 20 cm to the right. The display control unit 118 may also display the amount of change along with an arrow 411 (see Figure 2) indicating the direction of the appropriate position. The amount of change is the difference (distance) between the appropriate position and the position during visual inspection, for example, 20 cm.

[0070] For example, if the position and orientation during visual inspection is 45 degrees to the left compared to the appropriate position and orientation, the determination unit 114 may decide to change the orientation (direction) so that the position and orientation during visual inspection is 45 degrees to the right. The display control unit 118 may also display the amount of change along with the arrow 412 indicating the direction of the appropriate orientation. The amount of change is the difference (distance) between the appropriate position and orientation and the position and orientation during visual inspection, for example, 45 degrees.

[0071] As described above, the determination unit 113 determines suitability based on the evaluation score of the second position and orientation using an evaluation function constructed when the image generation model 121 is generated. The determination unit 113 also determines suitability based on the evaluation score calculated using a distance evaluation model, which is calculated based on the distance between the second position and orientation and the first position and orientation that is closest to the second position and orientation. The distance evaluation model is a machine learning model generated using training data in which the distance between the second position and orientation and the first position and orientation that is closest to the second position and orientation is the explanatory variable, and the evaluation score of the second position and orientation using an evaluation function constructed when the image generation model 121 is generated is the target variable.

[0072] The determination unit 114 determines the amount of change to the second position and attitude based on the distance between the second position and attitude and the position and attitude that the judgment unit 113 determines to be appropriate.

[0073] <Other Modifications> Although several embodiments and modifications of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. For example, the shooting support device 100 includes a model generation unit 112 to generate an image generation model 121, but the image generation unit 115 may generate an image using an image generation model 121 generated by another device.

[0074] The visual inspection support screen 410 (see Figure 2) displays arrows 411 and 412 indicating the direction to change the position and orientation of the camera 810 to the appropriate position and orientation. Alternatively, numerical values ​​(coordinates in the position and orientation space) indicating the position and orientation may be displayed instead of directions. Furthermore, although the shooting support device is configured to function as a "visual inspection support device" to assist in the inspection of the appearance of an object, its use is not limited to that of a visual inspection support device.

[0075] The present invention can take on various other embodiments, and furthermore, various modifications such as omissions and substitutions can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention as described herein, and are also included in the scope of the invention and its equivalents as described in the claims.

[0076] ≪Hardware Configuration≫ The imaging support device 100 according to the above embodiment is implemented by a computer 900 having a configuration such as that shown in Figure 7. Figure 7 is a hardware configuration diagram showing an example of a computer 900 that implements the functions of the imaging support device 100 according to the above embodiment. The computer 900 includes a CPU 901, ROM 902, RAM 903, SSD 904, input / output interface 905 (labeled as input / output I / F (Interface) in Figure 7), communication interface 906 (labeled as communication I / F in Figure 7), and media interface 907 (labeled as media I / F in Figure 7). The computer 900 may be equipped with an HDD (Hard Disc Drive) instead of the SSD 904, or it may be equipped with an HDD in addition to the SSD 904.

[0077] The CPU 901 operates based on programs stored in the ROM 902 or SSD 904 and is controlled by the control unit 110 in Figure 1. The ROM 902 stores boot programs executed by the CPU 901 when the computer 900 starts up, as well as programs related to the computer 900's hardware.

[0078] The CPU 901 controls input devices 910, such as a mouse and keyboard, and output devices 911, such as a display and printer, via the input / output interface 905. The CPU 901 acquires data from the input devices 910 and outputs the generated data to the output devices 911 via the input / output interface 905.

[0079] The SSD 904 stores programs executed by the CPU 901 and data used by those programs. The communication interface 906 receives data from other devices (e.g., camera 810) not shown via the communication network and outputs it to the CPU 901, and also transmits data generated by the CPU 901 to other devices via the communication network.

[0080] The media interface 907 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads the program from the recording medium 912 onto the RAM 903 via the media interface 907 and executes the loaded program. The recording medium 912 can be an optical recording medium such as a DVD (Digital Versatile Disk), a magneto-optical recording medium such as an MO (Magneto Optical Disk), a magnetic recording medium, a conductive memory tape medium, or a semiconductor memory.

[0081] For example, when the computer 900 functions as the imaging support device 100 according to the above embodiment, the CPU 901 of the computer 900 realizes the functions of the imaging support device 100 by executing the program 128 (see Figure 1) loaded on the RAM 903. The CPU 901 reads the program from the recording medium 912 and executes it. In addition, the CPU 901 may read the program from another device via a communication network, or it may install the program 128 from the recording medium 912 onto the SSD 904 and execute it.

[0082] 100 Shooting support device 111 Receiving unit 112 Model generation unit 113 Judgment unit 114 Determination unit 115 Image generation unit 116 Image comparison unit 117 Defect judgment unit 118 Display control unit (output control unit) 121 Image generation model 130 Position and orientation database 140 Image database 50 Shooting system 500 Robot 540 Camera 810 Camera

Claims

1. A shooting support device that stores a plurality of first position and orientations, which are the position and orientation of a camera that has captured an image of an object, in a storage unit, and uses the first position and orientations to support the shooting of the object, comprising: a receiving unit that receives a second position and orientation, which is the position and orientation of a camera that has captured an image of the object; a determination unit that determines whether the second position and orientation is appropriate according to the number of first position and orientations in a space including the first and second position and orientations that are within a predetermined distance from the second position and orientation; a determination unit that, when the determination unit determines that the second position and orientation is not appropriate, determines a direction in which the position and orientation will be changed to approach a position and orientation that the determination unit determines to be appropriate, starting from the second position and orientation; and an output control unit that outputs the direction determined by the determination unit.

2. The storage unit further stores an image generation model which is a machine learning model generated using training data in which the first position and orientation is an explanatory variable and an image of an object captured in the first position and orientation is an objective variable; the receiving unit receives a captured image which is an image of the object captured in the second position and orientation along with the second position and orientation; an image generation unit which, if the suitability of the second position and orientation is deemed appropriate, uses the image generation model to generate a generated image which is an image of the object captured in the second position and orientation; an image comparison unit which compares the captured image and the generated image and calculates the difference; and a defect determination unit which, if the difference satisfies predetermined conditions, determines that there is a defect in the region where the difference exists; and the output control unit outputs the region where the difference exists.

3. The shooting support device according to claim 2, wherein the determination unit determines suitability based on the second position and orientation evaluation points using an evaluation function constructed when the image generation model is generated.

4. The shooting support device according to claim 2, wherein the determination unit determines suitability based on an evaluation score calculated using a distance evaluation model based on the distance between the second position and orientation and the first position and orientation that is most recent to the second position and orientation, and the distance evaluation model is a machine learning model generated using training data in which the distance between the second position and orientation and the first position and orientation that is most recent to the second position and orientation is used as an explanatory variable and the evaluation score of the second position and orientation using an evaluation function constructed when the image generation model is generated is used as the objective variable.

5. The imaging support device according to claim 1, wherein the determination unit determines whether the second position and orientation is appropriate based on the distance between the second position and orientation and the first position and orientation, and the number of first position and orientations within that distance.

6. The imaging support device according to claim 4 or 5, wherein the determination unit determines the amount of change of the second position and attitude based on the distance between the second position and attitude and the position and attitude determined to be appropriate by the determination unit.

7. A shooting system that stores a plurality of first positional orientations, which are the position and orientation of a camera that has captured an image of an object, in a storage unit, and uses the first positional orientations to support the shooting of the object, comprising: a robot equipped with a camera that photographs the object while changing the position and orientation of the camera; a receiving unit that receives a second positional orientation, which is the position and orientation of a camera that has captured an image of the object; a determination unit that determines whether the second positional orientation is appropriate or not according to the number of first positional orientations in a space including the first positional orientation and the second positional orientation that are within a predetermined distance from the second positional orientation; a determination unit that, when the determination unit determines that the second positional orientation is not appropriate, determines a direction in which to change the position and orientation to approach a positional orientation that the determination unit determines to be appropriate, starting from the second positional orientation; and an output control unit that outputs the direction determined by the determination unit to the robot, wherein the robot changes the position and orientation of the camera according to the direction.

8. A shooting support device that assists in photographing an object, wherein a plurality of first positional orientations, which are the position and orientation of a camera that has captured an image of an object, are stored in a memory unit, and the shooting support device that assists in photographing the object using the first positional orientations, performs the steps of: receiving a second positional orientation, which is the position and orientation of a camera that has captured an image of an object; determining whether the second positional orientation is suitable or not according to the number of first positional orientations in a space including the first positional orientations and the second positional orientations that are within a predetermined distance from the second positional orientation; if the second positional orientation is determined to be unsuitable, determining a direction in which the position and orientation will be changed to approach a positional orientation that is determined to be suitable, starting from the second positional orientation; and outputting the direction.

Images