Image processing device, image processing method, and program

JPWO2023188510A5Active Publication Date: 2024-12-09FUJIFILM CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024511190
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-09-03
Publication Date
2024-12-09
Estimated Expiration
2042-11-09

AI Technical Summary

Technical Problem

Existing image processing technologies face challenges in visually grasping the correspondence between two-dimensional images and the regions they represent, especially when dealing with multiple images of the same object from different viewpoints, making it difficult to identify and compare relevant portions and imaging positions effectively.

Method used

An image processing device and method that generates three-dimensional images from multiple two-dimensional images, allowing for the display of two-dimensional and three-dimensional images side by side, with the ability to select and highlight images of interest, and display position specifying images to facilitate comparison and identification of specific portions and positions.

Benefits of technology

Enables effective visual comparison and identification of corresponding regions and positions between two-dimensional and three-dimensional images, improving the visibility and understanding of object states and damage by allowing for side-by-side comparison and highlighting of relevant features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2023188510000001
    Figure 2023188510000001
  • Figure 2023188510000002
    Figure 2023188510000002
  • Figure 2023188510000003
    Figure 2023188510000003
Patent Text Reader

Abstract

This image processing device comprises a processor. The processor: displays a three-dimensional image and a plurality of two-dimensional images which are used for generating the three-dimensional image that represents a subject in an actual space, and are associated with a plurality of portions of the three-dimensional image in a comparable state on a screen; selects an attention two-dimensional image from the plurality of two-dimensional images in accordance with a given selection instruction; and displays an attention portion corresponding to the attention two-dimensional image among the plurality of portions, in a visually identifiable state on the screen.
Need to check novelty before this filing date? Find Prior Art

Description

Image processing device, image processing method, and program

[0001] The technology of the present disclosure relates to an image processing device, an image processing method, and a program.

[0002] Japanese Patent Application Laid-Open Publication No. 2020-005186 discloses an image display system configured with a computer system. The computer system inputs an image group including multiple images captured at different dates, locations, and directions, displays a list of the image group on a list screen, and displays a first image selected from the image group based on a user's operation on an individual screen. The computer system also determines an adjacent image for the first image based on a spatial positional relationship between the first image and a pair of candidate images spatially surrounding the first image and a determination of an overlap state regarding the capture range. Based on the user's operation, the computer system selects the adjacent image for the first image as a second image on the individual screen and displays the second image as a new first image.

[0003] Japanese Patent Laid-Open Publication No. 2007-093661 discloses a navigation device that is mounted on an automobile and simultaneously displays a first map and a second map that has a different representation format from the first map. The navigation device includes a display device, a map display means, a current position calculation means, a current position display means, and a position designation receiving means. The map display means displays the first map and the second map in different display areas of the display device. The current position calculation means calculates the current position. The current position display means displays a current position mark representing the current position calculated by the current position calculation means on at least one of the first map and the second map displayed by the map display means. The position designation receiving means receives from a user a designation of a position on the display area where the first map is displayed. The map display means displays the second map in a form that enables identification of a position on the second map that represents the same point on the first map as the point corresponding to the position designated by the position designation means.

[0004] Japanese Patent Laid-Open Publication No. 2010-200024 discloses a stereoscopic image display device. The stereoscopic image display device includes a display means, an instruction input means, an alignment means, and a display control means. The display means displays a list of thumbnail images generated from captured images captured from multiple viewpoints before stereoscopically displaying the captured images. The instruction input means receives a selection instruction to select a thumbnail image from the list. When the selection instruction is received and the captured images are stereoscopically displayed, the alignment means aligns the captured images from multiple viewpoints corresponding to the selected thumbnail image with each other in a detection area of ​​a specific target in the captured images. The display control means adds detection area information indicating the detection area of ​​the specific target to the thumbnail images.

[0005] As an example, one embodiment of the technology disclosed herein provides an image processing device, an image processing method, and a program that can visually grasp the correspondence between each two-dimensional image and an area of ​​an object that corresponds to each two-dimensional image.

[0006] A first aspect of the technology of the present disclosure is an image processing device that includes a processor, and that displays on a screen a plurality of two-dimensional images used to generate a three-dimensional image showing an object in real space, the plurality of two-dimensional images corresponding to a plurality of portions of the three-dimensional image, in a manner that allows comparison with the three-dimensional image, selects a two-dimensional image of interest from the plurality of two-dimensional images in accordance with given selection instructions, and displays on the screen a portion of interest that corresponds to the two-dimensional image of interest from the plurality of portions in a manner that allows visual identification.

[0007] A second aspect of the technology of the present disclosure is an image processing device according to the first aspect, in which a state in which multiple two-dimensional images and a three-dimensional image can be compared is a state in which a first area including multiple two-dimensional images and a second area including a three-dimensional image are arranged side by side.

[0008] A third aspect of the technology of the present disclosure is an image processing device according to the first or second aspect, in which a state in which a portion of interest can be visually identified includes a state in which the portion of interest is distinguishable from the rest of the multiple portions.

[0009] A fourth aspect of the technology of the present disclosure is an image processing device according to any one of the first to third aspects, in which a state in which a portion of interest can be visually identified includes a state in which the two-dimensional image of interest is distinguishable from the remaining two-dimensional images among the plurality of two-dimensional images.

[0010] A fifth aspect of the technology of the present disclosure is an image processing device according to any one of the first to fourth aspects, wherein the processor displays on a screen a plurality of position-specific images that can identify a plurality of imaging positions where imaging was performed to obtain a plurality of two-dimensional images, in a state that allows comparison with the three-dimensional image, selects, in accordance with a selection instruction, an imaging position that corresponds to a position-specific image selected from the plurality of position-specific images as a target imaging position from the plurality of imaging positions, and selects, from the plurality of two-dimensional images, a two-dimensional image obtained by imaging from the target imaging position as a target two-dimensional image.

[0011] A sixth aspect of the technology of the present disclosure is an image processing device according to the fifth aspect, in which a state in which a plurality of position-specific images can be compared with a three-dimensional image includes a state in which a plurality of position-specific images are placed opposite a three-dimensional image.

[0012] A seventh aspect of the technology of the present disclosure is an image processing device according to the fifth or sixth aspect, in which a state in which a plurality of two-dimensional images and a three-dimensional image can be compared is a state in which a third area including a plurality of two-dimensional images and a fourth area including an image showing a state in which a plurality of position-specific images and a three-dimensional image are opposed to each other are lined up.

[0013] An eighth aspect of the technology of the present disclosure is an image processing device according to any one of the fifth to seventh aspects, wherein a state in which a portion of interest can be visually identified includes a state in which an image specifying position of interest is distinguishable from the remaining image specifying positions among a plurality of image specifying positions.

[0014] A ninth aspect of the technology of the present disclosure is an image processing device according to any one of the fifth to eighth aspects, wherein the image processing device has a first operating mode in which a plurality of two-dimensional images and a three-dimensional image are displayed on a screen in a state in which they can be compared, and a second operating mode in which a plurality of position-specific images are displayed on a screen in a state in which they can be compared with the three-dimensional image, and the processor sets the operating mode to either the first operating mode or the second operating mode in accordance with given setting instructions.

[0015] A tenth aspect of the technology of the present disclosure is an image processing device according to any one of the fifth to ninth aspects, in which a three-dimensional image is displayed on a screen from a viewpoint corresponding to a two-dimensional image of interest.

[0016] An eleventh aspect of the technology of the present disclosure is an image processing device that includes a processor, the processor displays on a screen a plurality of two-dimensional images used to generate a three-dimensional image showing an object in real space, the plurality of two-dimensional images corresponding to a plurality of portions of the three-dimensional image in a manner that allows comparison with the three-dimensional image, selects a portion of interest from the plurality of portions in accordance with given selection instructions, selects a two-dimensional image of interest that corresponds to the portion of interest from the plurality of two-dimensional images, and displays the two-dimensional image of interest on the screen in a manner that allows it to be distinguished from the remaining two-dimensional images from the plurality of two-dimensional images.

[0017] A twelfth aspect of the technology of the present disclosure is an image processing device according to the eleventh aspect, wherein the processor displays on a screen a plurality of position-specific images that can identify a plurality of imaging positions where imaging was performed to obtain a plurality of two-dimensional images, in a state that allows comparison with the three-dimensional image, selects a focus position-specific image from the plurality of position-specific images in accordance with a selection instruction, and selects a two-dimensional image obtained by imaging from an imaging position identified from the focus position-specific image from the plurality of two-dimensional images as the focus two-dimensional image.

[0018] A thirteenth aspect of the technology of the present disclosure is an image processing method that includes: displaying on a screen a plurality of two-dimensional images used to generate a three-dimensional image showing an object in real space, the plurality of two-dimensional images corresponding to a plurality of portions of the three-dimensional image, in a state in which the three-dimensional image can be compared with the three-dimensional image; selecting a two-dimensional image of interest from the plurality of two-dimensional images in accordance with given selection instructions; and displaying on the screen a portion of interest that corresponds to the two-dimensional image of interest from the plurality of portions in a state in which the portion of interest can be visually identified.

[0019] A fourteenth aspect of the technology of the present disclosure is a program for causing a computer to execute processing including: displaying on a screen a plurality of two-dimensional images used to generate a three-dimensional image showing an object in real space, the plurality of two-dimensional images corresponding to a plurality of portions of the three-dimensional image, in a state in which the three-dimensional image can be compared with the three-dimensional image; selecting a two-dimensional image of interest from the plurality of two-dimensional images in accordance with given selection instructions; and displaying on the screen a portion of interest that corresponds to the two-dimensional image of interest from the plurality of portions in a state in which the portion of interest can be visually identified.

[0020] 1 is a perspective view showing an example of an inspection system according to the first embodiment. FIG. 2 is a block diagram showing an example of an inspection support device according to the first embodiment. FIG. 3 is a block diagram showing an example of an imaging device according to the first embodiment. FIG. 4 is a block diagram showing an example of a functional configuration for realizing the inspection support information generation process according to the first embodiment. FIG. 5 is a block diagram showing an example of data transmitted from the imaging device according to the first embodiment to the inspection support device. FIG. 6 is a block diagram showing an example of the operation of the acquisition unit and the 3D image generation unit according to the first embodiment. FIG. 7 is a block diagram showing an example of the operation of the 3D image generation unit and the inspection support information generation unit according to the first embodiment. FIG. 8 is a block diagram showing an example of a functional configuration for realizing the inspection support process according to the first embodiment. FIG. 9 is a block diagram showing an example of the operation of the operation mode setting unit, the first mode processing unit, the second mode processing unit, and the third mode processing unit according to the first embodiment. FIG. 10 is a block diagram showing an example of the operation of the first display control unit according to the first embodiment. FIG. 11 is a block diagram showing an example of the operation of the first image selection unit according to the first embodiment. FIG. 12 is a block diagram showing an example of the operation of the first pixel extraction unit and the first image generation unit according to the first embodiment. FIG. 13 is a block diagram showing an example of the operation of the first image generation unit and the first display control unit according to the first embodiment. FIG. 14 is a block diagram showing an example of the operation of the second display control unit according to the first embodiment. FIG. 15 is a block diagram showing an example of the operation of the second image selection unit according to the first embodiment. FIG. 16 is a block diagram showing an example of the operation of the second pixel extraction unit and the second image generation unit according to the first embodiment. FIG. 17 is a block diagram showing an example of the operation of the second image generation unit and the second display control unit according to the first embodiment. FIG. 1 is a block diagram showing an example of the operation of a third display control unit according to the first embodiment. FIG. 2 is a block diagram showing an example of the operation of a third image selection unit according to the first embodiment. FIG. 3 is a block diagram showing an example of the operation of a third image generation unit according to the first embodiment. FIG. 4 is a block diagram showing an example of the operation of a third image generation unit and a third display control unit according to the first embodiment. FIG. 5 is a flowchart showing an example of the flow of inspection support information generation processing according to the first embodiment. FIG. 6 is a flowchart showing an example of the flow of mode setting processing in the inspection support processing according to the first embodiment. FIG. 7 is a flowchart showing an example of the flow of first mode processing in the inspection support processing according to the first embodiment.1 is a flowchart showing an example of the flow of second mode processing in the inspection support processing related to the first embodiment. FIG. 2 is a flowchart showing an example of the flow of third mode processing in the inspection support processing related to the first embodiment. FIG. 3 is a block diagram showing an example of the operation of a fourth display control unit related to the second embodiment. FIG. 4 is a block diagram showing an example of the operation of a fourth image selection unit related to the second embodiment. FIG. 5 is a block diagram showing an example of the operation of a fourth pixel extraction unit and a fourth image generation unit related to the second embodiment. FIG. 6 is a block diagram showing an example of the operation of a fourth image generation unit and a fourth display control unit related to the second embodiment. FIG. 7 is a flowchart showing an example of inspection support processing related to the second embodiment.

[0021] Hereinafter, an example of an embodiment of an image processing device, an image processing method, and a program according to the technique of the present disclosure will be described with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". HDD is an abbreviation for "Hard Disk Drive". SSD is an abbreviation for "Solid State Drive". RAM is an abbreviation for "Random Access Memory". SRAM is an abbreviation for "Static Random Access Memory". DRAM is an abbreviation for "Dynamic Random Access Memory". EL is an abbreviation for "Electro Luminescence". RAM is an abbreviation for "Random Access Memory". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". GNSS is an abbreviation for "Global Navigation Satellite System". GPS is an abbreviation for "Global Positioning System". SfM is an abbreviation for "Structure from Motion". MVS is an abbreviation for "Multi-View Stereo". TPU is an abbreviation for "Tensor Processing Unit". USB is an abbreviation for "Universal Serial Bus". ASIC is an abbreviation for "Application Specific Integrated Circuit". FPGA is an abbreviation for "Field-Programmable Gate Array". PLD is an abbreviation for "Programmable Logic Device". SoC is an abbreviation for "System-on-a-chip". IC is an abbreviation for "Integrated Circuit".

[0024] First Embodiment First, a first embodiment of the present disclosure will be described.

[0025] 1 , an inspection system S includes an inspection support device 10 and an imaging device 100. The inspection system S is a system for inspecting an object 4 in real space. The object 4 is an example of an "object" in the technology of the present disclosure.

[0026] As an example, the object 4 is a bridge pier made of reinforced concrete. Here, a bridge pier is given as an example of the object 4, but the object 4 may be road equipment other than a bridge pier. Examples of road equipment include a road surface, a tunnel, a guardrail, a traffic light, and / or a windbreak fence. The object 4 may be social infrastructure other than road equipment (e.g., airport facilities, port facilities, water storage facilities, gas facilities, medical facilities, firefighting facilities, and / or educational facilities), or may be private property. The object 4 may also be land (e.g., national land and / or private land). The bridge pier given as an example of the object 4 may be a bridge pier made of materials other than reinforced concrete. In the first embodiment, inspection refers to, for example, inspecting the condition of the object 4. For example, the inspection system S inspects the object 4 for damage and / or the extent of the damage.

[0027] The inspection support device 10 is an example of an "image processing device" according to the technology of the present disclosure. The inspection support device 10 is, for example, a desktop personal computer. Here, a desktop personal computer is illustrated as the inspection support device 10, but this is merely an example and the device may be a notebook personal computer. The device is not limited to a personal computer and may be a server. The server may be a mainframe used on-premises together with the inspection support device 10, or an external server realized by cloud computing. The server may also be an external server realized by network computing such as fog computing, edge computing, or grid computing. The inspection support device 10 is communicatively connected to the imaging device 100. The inspection support device 10 is used by an inspector 6. The inspection support device 10 may be used at the site where the object 4 is installed, or at a location separate from the site where the object 4 is installed.

[0028] The imaging device 100 is, for example, a digital camera with an interchangeable lens. Here, an interchangeable lens digital camera is exemplified as the imaging device 100, but this is merely an example, and the imaging device 100 may be a digital camera built into various electronic devices such as a smart device or a wearable terminal. The imaging device 100 may also be an eyewear terminal in the form of glasses, or a head-mounted display terminal worn on the head. The imaging device 100 is used by an imaging person 8.

[0029] As an example, as shown in FIG. 2 , the inspection support device 10 includes a computer 12 , a reception device 14 , a display 16 , and a communication device 18 .

[0030] The computer 12 is an example of a "computer" according to the technology of the present disclosure. The computer 12 includes a processor 20, a storage 22, and a RAM 24. The processor 20 is an example of a "processor" according to the technology of the present disclosure. The processor 20, the storage 22, the RAM 24, the reception device 14, the display 16, and the communication device 18 are connected to a bus 26.

[0031] The processor 20 has, for example, a CPU and controls the entire inspection support device 10. Here, an example in which the processor 20 has a CPU is given, but this is merely one example. For example, the processor 20 may have a CPU and a GPU. In this case, for example, the GPU operates under the control of the CPU and is responsible for executing image processing.

[0032] The storage 22 is a non-volatile storage device that stores various programs, various parameters, etc. Examples of the storage 22 include an HDD and an SSD. Note that the HDD and SSD are merely examples, and a flash memory, a magnetoresistive memory, and / or a ferroelectric memory may be used instead of or in addition to the HDD and / or SSD.

[0033] The RAM 24 is a memory that temporarily stores information and is used as a work memory by the processor 20. The RAM 24 may be, for example, a DRAM and / or an SRAM.

[0034] The reception device 14 has a keyboard, a mouse, a touch panel, etc. (all not shown), and receives various instructions from the inspector 6. The display 16 has a screen 16A. The screen 16A is an example of a "screen" according to the technology of the present disclosure. The display 16 displays various information (e.g., images, text, etc.) on the screen 16A under the control of the processor 20. The display 16 is, for example, an EL display (e.g., an organic EL display or an inorganic EL display). Note that the display is not limited to an EL display, and may be another type of display such as a liquid crystal display.

[0035] The communication device 18 is communicably connected to the imaging device 100. Here, the communication device 18 is wirelessly connected to the imaging device 100 using a default wireless communication standard. Examples of the default wireless communication standard include Wi-Fi (registered trademark) and Bluetooth (registered trademark). The communication device 18 controls the exchange of information with the inspection support device 10. For example, the communication device 18 transmits information to the imaging device 100 in response to a request from the processor 20. The communication device 18 also receives information transmitted from the imaging device 100 and outputs the received information to the processor 20 via the bus 26. Note that the communication device 18 may also be communicably connected to the imaging device 100 via a wired connection.

[0036] As an example, as shown in FIG. 3, an imaging device 100 includes a computer 102 , an image sensor 104 , a positioning unit 106 , an acceleration sensor 108 , an angular velocity sensor 110 , and a communication device 112 .

[0037] The computer 102 includes a processor 114, a storage 116, and a RAM 118. The processor 114, the storage 116, the RAM 118, the image sensor 104, the positioning unit 106, the acceleration sensor 108, the angular velocity sensor 110, and the communication device 112 are connected to a bus 120. The processor 114, the storage 116, and the RAM 118 are realized by, for example, hardware similar to the processor 20, the storage 22, and the RAM 24 provided in the inspection support device 10 described above.

[0038] The image sensor 104 is, for example, a CMOS image sensor. Note that, although a CMOS image sensor is used as the image sensor 104 here, the technology of the present disclosure is not limited to this, and other image sensors may be used. The image sensor 104 captures an image of a subject (for example, the target object 4) and outputs image data obtained by capturing the image.

[0039] The positioning unit 106 is a device that detects the position of the imaging device 100. The position of the imaging device 100 is detected using, for example, GNSS (e.g., GPS). The positioning unit 106 has a GNSS receiver (not shown). The GNSS receiver receives, for example, radio waves transmitted from multiple satellites. The positioning unit 106 detects the position of the imaging device 100 based on the radio waves received by the GNSS receiver and outputs positioning data (e.g., data indicating latitude, longitude, and altitude) corresponding to the detected position.

[0040] The acceleration sensor 108 detects acceleration in each of the axial directions of the pitch axis, yaw axis, and roll axis of the image capturing device 100. The acceleration sensor 108 outputs acceleration data corresponding to the acceleration in each axial direction of the image capturing device 100. The angular velocity sensor 110 detects angular velocities around each of the pitch axis, yaw axis, and roll axis of the image capturing device 100. The angular velocity sensor 110 outputs angular velocity data corresponding to the angular velocities of the image capturing device 100 around each axis.

[0041] The processor 114 acquires the position of the image capture device 100 based on the positioning data and / or acceleration data, and generates position data indicating the acquired position. The processor 114 also acquires the attitude of the image capture device 100 (i.e., the amount of change in attitude relative to a reference attitude defined in a relative coordinate system) based on the angular velocity data, and generates attitude data indicating the acquired attitude. Hereinafter, the position of the image capture device 100 will be referred to as the "image capture position," and the attitude of the image capture device 100 will be referred to as the "image capture attitude."

[0042] Note that when the processor 114 acquires the imaging position based only on the positioning data, the acceleration sensor 108 may be omitted. On the other hand, when the processor 114 acquires the imaging position based only on the acceleration data, the positioning unit 106 may be omitted. When the processor 114 acquires the imaging position based on the positioning data, the imaging position in the absolute coordinate system is derived based on the positioning data. On the other hand, when the processor 114 acquires the imaging position based on the acceleration data, the amount of change in the imaging position relative to a reference position defined in the relative coordinate system is derived based on the acceleration data.

[0043] The communication device 112 is communicably connected to the inspection support device 10. The communication device 112 is realized by, for example, hardware similar to the communication device 18 provided in the inspection support device 10 described above.

[0044] The imaging device 100 transmits image data, position data, and attitude data to the inspection support device 10. The image data is data representing a two-dimensional image 50 obtained by imaging the object 4 with the imaging device 100. The position data is data representing the imaging position when the imaging device 100 captures an image, and is associated with the image data. Similarly, the attitude data is data representing the imaging attitude when the imaging device 100 captures an image, and is associated with the image data. In other words, the position data and attitude data are associated data attached to the image data.

[0045] For example, if only multiple two-dimensional images 50 obtained by capturing images of the object 4 from multiple imaging positions using the imaging device 100 are displayed on the screen 16A of the display 16 provided in the inspection support device 10, it is time-consuming to determine the correspondence between each of the two-dimensional images 50 and the area of ​​the object 4 corresponding to each of the two-dimensional images 50 based on the multiple two-dimensional images 50 displayed on the screen 16A. Furthermore, the greater the number of frames of the multiple two-dimensional images 50, the more complicated the task of determining the correspondence becomes. In view of this, in the first embodiment, the inspection support device 10 performs an inspection support information generation process and an inspection support process. The inspection support information generation process and the inspection support process performed by the inspection support device 10 will be described in detail below.

[0046] 4 as an example, an inspection support information generation program 30 is stored in the storage 22 of the inspection support device 10. The processor 20 of the inspection support device 10 reads the inspection support information generation program 30 from the storage 22 and executes the read inspection support information generation program 30 on the RAM 24. The processor 20 performs an inspection support information generation process for generating inspection support information 56 in accordance with the inspection support information generation program 30 executed on the RAM 24.

[0047] The inspection support information generation process is realized by the processor 20 operating as an acquisition unit 32 , a three-dimensional image generation unit 34 , and an inspection support information generation unit 36 ​​in accordance with the inspection support information generation program 30 .

[0048] 5, multiple points P1 located in the circumferential direction of the object 4 indicate imaging positions taken by the imaging device 100. The photographer 8 moves around the object 4 and captures images of the object 4 using the imaging device 100 from multiple imaging positions in the circumferential direction of the object 4. As an example, the photographer 8 captures images of different regions of the object 4 using the imaging device 100 from each imaging position. As different regions of the object 4 are captured by the imaging device 100 from each imaging position, the entire object 4 including multiple regions is captured.

[0049] The imaging position (i.e., point P1) corresponding to each two-dimensional image 50 obtained by imaging with the imaging device 100 corresponds to the origin of the line of sight L directed at the object 4, and the imaging attitude corresponding to each two-dimensional image 50 corresponds to the direction of the line of sight L directed at the object 4. Point P2 where the object 4 intersects with the line of sight L corresponds to the viewpoint when the object 4 is viewed with the line of sight L. The object 4 is imaged by the imaging device 100 from each imaging position, thereby obtaining two-dimensional images 50 corresponding to each viewpoint. Each two-dimensional image 50 is an image corresponding to a respective region of the object 4.

[0050] Note that, although an example is given here in which the photographer 8 captures images of the object 4 from each imaging position using the imaging device 100 while moving around the object 4, the imaging device 100 may be mounted on a moving object, and images of the object 4 may be captured from each imaging position by the imaging device 100 while the moving object is moving around the object 4. Furthermore, the moving object may be, for example, a drone, a gondola, a cart, a high-altitude work vehicle, an automated guided vehicle, or any other vehicle.

[0051] The imaging device 100 associates image data indicating the two-dimensional image 50 obtained by imaging from each imaging position with position data indicating the imaging position when imaging was performed, and orientation data indicating the imaging orientation when imaging was performed. Then, the imaging device 100 transmits each piece of image data and the position data and orientation data associated with each piece of image data to the inspection support device 10.

[0052] 6 as an example, the acquisition unit 32 acquires a two-dimensional image 50 based on each piece of image data received by the inspection support device 10. The acquisition unit 32 also acquires an imaging position corresponding to each two-dimensional image 50 based on each piece of position data received by the inspection support device 10. The acquisition unit 32 also acquires an imaging orientation corresponding to each two-dimensional image 50 based on each piece of orientation data received by the inspection support device 10.

[0053] The three-dimensional image generating unit 34 generates a three-dimensional image 52 showing the object 4 based on the plurality of two-dimensional images 50 acquired by the acquiring unit 32. Image processing techniques for generating the three-dimensional image 52 based on the plurality of two-dimensional images 50 include SfM, MVS, epipolar geometry, and stereo matching processing. The positions of the plurality of pixels included in the three-dimensional image 52 are identified by the plurality of three-dimensional coordinates obtained from the plurality of two-dimensional images 50. The three-dimensional image 52 is a three-dimensional model defined by the plurality of three-dimensional coordinates.

[0054] 7 as an example, the three-dimensional image 52 generated by the three-dimensional image generation unit 34 has a plurality of portions 54 corresponding to each of the two-dimensional images 50. Each portion 54 is formed by a pixel group, which is a collection of pixels corresponding to each of the two-dimensional images 50. The inspection support information generation unit 36 ​​generates inspection support information 56, which is information that associates each of the two-dimensional images 50 acquired by the acquisition unit 32, the imaging position corresponding to each of the two-dimensional images 50, the imaging attitude corresponding to each of the two-dimensional images 50, and the portions 54 corresponding to each of the two-dimensional images 50. The inspection support information 56 is stored in the storage 22.

[0055] As an example, as shown in Figure 8, an inspection assistance program 40 is stored in the storage 22 of the inspection assistance device 10. The inspection assistance program 40 is an example of a "program" according to the technology of the present disclosure. The processor 20 reads the inspection assistance program 40 from the storage 22 and executes the read inspection assistance program 40 on the RAM 24. The processor 20 performs an inspection assistance process to assist an inspector 6 (see Figure 1) in an inspection in accordance with the inspection assistance program 40 executed on the RAM 24.

[0056] The inspection support process is realized by the processor 20 operating as an operation mode setting unit 42 , a first mode processing unit 44 , a second mode processing unit 46 , and a third mode processing unit 48 in accordance with the inspection support program 40 .

[0057] The inspection support device 10 has a first mode, a second mode, and a third mode as operation modes. The operation mode setting unit 42 performs a mode setting process to selectively set the first mode, the second mode, and the third mode as the operation mode of the inspection support device 10.

[0058] In the mode setting process, when the operation mode setting unit 42 sets the operation mode of the inspection support device 10 to the first mode, the processor 20 operates as a first mode processing unit 44. The first mode processing unit 44 performs first mode processing. The first mode processing is realized by the first mode processing unit 44 operating as a first display control unit 44A, a first image selection unit 44B, a first pixel extraction unit 44C, and a first image generation unit 44D.

[0059] In the mode setting process, when the operation mode setting unit 42 sets the operation mode of the inspection support device 10 to the second mode, the processor 20 operates as a second mode processing unit 46. The second mode processing unit 46 performs second mode processing. The second mode processing is realized by the second mode processing unit 46 operating as a second display control unit 46A, a second image selection unit 46B, a second pixel extraction unit 46C, and a second image generation unit 46D.

[0060] In the mode setting process, when the operation mode setting unit 42 sets the operation mode of the inspection support device 10 to the third mode, the processor 20 operates as a third mode processing unit 48. The third mode processing unit 48 performs third mode processing. The third mode processing is realized by the third mode processing unit 48 operating as a third display control unit 48A, a third image selection unit 48B, and a third image generation unit 48C.

[0061] 9 as an example, the operation mode setting unit 42 sets the first mode by default as the operation mode of the inspection support device 10. When the operation mode setting unit 42 sets the operation mode of the inspection support device 10 to the first mode, the first display control unit 44A displays a first image 61 on the screen 16A. Details of the first image 61 will be described later, but the first image 61 includes a second mode setting button 72 and a third mode setting button 73 as soft keys.

[0062] When the first image 61 is displayed on the screen 16A and the reception device 14 receives a setting instruction to press the second mode setting button 72, the reception device 14 outputs a second mode setting instruction signal to the processor 20. Similarly, when the first image 61 is displayed on the screen 16A and the reception device 14 receives a setting instruction to press the third mode setting button 73, the reception device 14 outputs a third mode setting instruction signal to the processor 20.

[0063] When the operation mode of the inspection support device 10 is set to the first mode, the operation mode setting unit 42 determines whether a second mode setting instruction signal or a third mode setting instruction signal has been input to the processor 20. When the second mode setting instruction signal has been input to the processor 20, the operation mode setting unit 42 sets the second mode as the operation mode of the inspection support device 10. On the other hand, when the third mode setting instruction signal has been input to the processor 20, the operation mode setting unit 42 sets the third mode as the operation mode of the inspection support device 10.

[0064] When the operation mode setting unit 42 sets the operation mode of the inspection support device 10 to the second mode, the second display control unit 46A displays a second image 62 on the screen 16A. Details of the second image 62 will be described later, but the second image 62 includes a first mode setting button 71 and a third mode setting button 73 as soft keys.

[0065] When the second image 62 is displayed on the screen 16A and the reception device 14 receives a setting instruction to press the first mode setting button 71, the reception device 14 outputs a first mode setting instruction signal to the processor 20. Similarly, when the second image 62 is displayed on the screen 16A and the reception device 14 receives a setting instruction to press the third mode setting button 73, the reception device 14 outputs a third mode setting instruction signal to the processor 20.

[0066] When the operation mode of the inspection support device 10 is set to the second mode, the operation mode setting unit 42 determines whether a first mode setting instruction signal or a third mode setting instruction signal has been input to the processor 20. When the first mode setting instruction signal has been input to the processor 20, the operation mode setting unit 42 sets the first mode as the operation mode of the inspection support device 10. On the other hand, when the third mode setting instruction signal has been input to the processor 20, the operation mode setting unit 42 sets the third mode as the operation mode of the inspection support device 10.

[0067] When the operation mode setting unit 42 sets the operation mode of the inspection support device 10 to the third mode, the third display control unit 48A displays a third image 63 on the screen 16A. Details of the third image 63 will be described later, but the third image 63 includes a first mode setting button 71 and a second mode setting button 72.

[0068] When the reception device 14 receives a setting instruction to press the first mode setting button 71 while the third image 63 is displayed on the screen 16A, the reception device 14 outputs a first mode setting instruction signal to the processor 20. Similarly, when the reception device 14 receives a setting instruction to press the third mode setting button 73 while the third image 63 is displayed on the screen 16A, the reception device 14 outputs a third mode setting instruction signal to the processor 20.

[0069] When the operation mode of the inspection support device 10 is set to the third mode, the operation mode setting unit 42 determines whether a first mode setting instruction signal or a second mode setting instruction signal has been input to the processor 20. When the first mode setting instruction signal has been input to the processor 20, the operation mode setting unit 42 sets the first mode as the operation mode of the inspection support device 10. On the other hand, when the second mode setting instruction signal has been input to the processor 20, the operation mode setting unit 42 sets the second mode as the operation mode of the inspection support device 10.

[0070] Hereinafter, when there is no need to distinguish between the first mode setting instruction signal, the second mode setting instruction signal, and the third mode setting instruction signal, the first mode setting instruction signal, the second mode setting instruction signal, and the third mode setting instruction signal will be referred to as the "mode setting instruction signal."

[0071] The second mode of the plurality of operation modes of the inspection support device 10 is an example of a “first operation mode” according to the technology of the present disclosure. The third mode of the plurality of operation modes of the inspection support device 10 is an example of a “second operation mode” according to the technology of the present disclosure.

[0072] 10 shows, as an example, a state in which a first image 61 is displayed on the screen 16A. The first image 61 includes a first image area 81 and a second image area 82. As an example, the first image area 81 and the second image area 82 are displayed on the screen 16A aligned in the left-right direction of the first image 61. The first image area 81 includes a plurality of two-dimensional images 50, and the second image area 82 includes a three-dimensional image 52.

[0073] The first display control unit 44A includes a plurality of two-dimensional images 50 in the first image area 81 based on the plurality of two-dimensional images 50 included in the inspection support information 56. The first display control unit 44A also includes a three-dimensional image 52 in the second image area 82 based on the three-dimensional image 52 included in the inspection support information 56.

[0074] The first image area 81 includes a predetermined number of two-dimensional images 50 out of the plurality of two-dimensional images 50. The predetermined number is set, for example, by the inspector 6 giving an instruction specifying the predetermined number to the reception device 14 (see FIG. 9 ). Also, for example, the inspector 6 giving an instruction to the reception device 14 to scroll the first image area 81 causes the first image area 81 to be scrolled, and thereby the two-dimensional images 50 included in the first image area 81 are changed.

[0075] The second image area 82 includes the three-dimensional image 52 in a two-dimensional state as a result of rendering. For example, the inspector 6 issues an instruction to the reception device 14 (see FIG. 9 ) to change the size of the three-dimensional image 52, thereby changing the size of the three-dimensional image 52. Furthermore, for example, the inspector 6 issues an instruction to the reception device 14 to rotate the three-dimensional image 52, thereby rotating the three-dimensional image 52.

[0076] A first image area 81 including a plurality of two-dimensional images 50 and a second image area 82 including a three-dimensional image 52 are displayed side by side on the screen 16A, thereby enabling comparison of the plurality of two-dimensional images 50 and the three-dimensional image 52. Note that, in the example shown in Fig. 10, the first image area 81 and the second image area 82 are displayed on the screen 16A side by side in the left-right direction of the first image 61, but, for example, the first image area 81 and the second image area 82 may be displayed on the screen 16A side by side in the up-down direction of the first image 61, or the first image area 81 and the second image area 82 may be displayed on the screen 16A with the first image area 81 incorporated into a part of the second image area 82.

[0077] The two-dimensional image 50 is an example of a "two-dimensional image" according to the technology of the present disclosure. The three-dimensional image 52 is an example of a "three-dimensional image" according to the technology of the present disclosure. The first image area 81 is an example of a "first area" according to the technology of the present disclosure. The second image area 82 is an example of a "second area" according to the technology of the present disclosure.

[0078] 11 , when a selection instruction that is an instruction to select one of the plurality of two-dimensional images 50 included in the first image area 81 is received by the reception device 14 while the first image 61 is displayed on the screen 16A, the reception device 14 outputs a selection instruction signal indicating the selection instruction to the processor 20. The selection instruction is an example of a “selection instruction” according to the technology of the present disclosure.

[0079] When a selection instruction signal is input to the processor 20, the first image selector 44B selects, in accordance with the selection instruction indicated by the selection instruction signal, a two-dimensional image 50 corresponding to the selection instruction (hereinafter referred to as a "two-dimensional image of interest 50A") from the plurality of two-dimensional images 50 included in the inspection support information 56. The two-dimensional image of interest 50A is an example of a "two-dimensional image of interest" according to the technology of the present disclosure.

[0080] 12 , the first pixel extractor 44C acquires an imaging position and imaging orientation corresponding to the two-dimensional image of interest 50A from the inspection support information 56. The first pixel extractor 44C then derives a viewpoint corresponding to the two-dimensional image of interest 50A based on the acquired imaging position and imaging orientation. The first pixel extractor 44C then extracts pixels from the three-dimensional image 52 included in the inspection support information 56 to include the three-dimensional image 52 in the second image region 82 at the derived viewpoint. When extracting pixels from the three-dimensional image 52, the first pixel extractor 44C also extracts pixels from the three-dimensional image 52 to include the entire three-dimensional image 52 in a size that fits the entire three-dimensional image 52 within the second image region 82.

[0081] The first image generation unit 44D generates a first image area 81 that includes a predetermined number of two-dimensional images 50 from the plurality of two-dimensional images 50 included in the inspection support information 56 and in which the two-dimensional image of interest 50A is surrounded by a frame 90. The first image generation unit 44D also generates a second image area 82 that includes a three-dimensional image 52 having a size that allows the entire image to fit within the second image area 82 from a viewpoint corresponding to the two-dimensional image of interest 50A, based on the pixels extracted by the first pixel extraction unit 44C. For example, the three-dimensional image 52 is included in the second image area 82 so that the viewpoint corresponding to the two-dimensional image of interest 50A is located at a center 82C of the second image area 82.

[0082] As an example, as shown in FIG. 13, the first image generating unit 44D generates a first image 61 by combining the generated first image area 81 and second image area 82.

[0083] The first display control unit 44A outputs first image data indicating the first image 61 generated by the first image generation unit 44D to the display 16. As a result, the first image 61 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of two-dimensional images 50 out of the multiple two-dimensional images 50 are displayed on the screen 16A in a state where they are included in the first image area 81, and the two-dimensional image of interest 50A is displayed on the screen 16A in a state where it is surrounded by a frame 90. By displaying the two-dimensional image of interest 50A on the screen 16A in a state where it is surrounded by the frame 90, the two-dimensional image of interest 50A becomes distinguishable from the remaining two-dimensional images 50 out of the multiple two-dimensional images 50.

[0084] Furthermore, the three-dimensional image 52 is displayed on the screen 16A from a viewpoint corresponding to the two-dimensional image of interest 50A and at a size that fits entirely within the second image area 82. By displaying the three-dimensional image 52 on the screen 16A from a viewpoint corresponding to the two-dimensional image of interest 50A, a portion 54 of the three-dimensional image 52 that corresponds to the two-dimensional image of interest 50A (hereinafter referred to as "portion of interest 54A") becomes visually identifiable. The "portion 54" is an example of a "portion" according to the technology of the present disclosure, and the portion of interest 54A of the three-dimensional image 52 is an example of a "portion of interest" according to the technology of the present disclosure.

[0085] 13 , the 2D image of interest 50A is displayed on the screen 16A surrounded by a frame 90. However, the 2D image of interest 50A may be displayed on the screen 16A in other ways so as to be distinguishable from the remaining 2D images 50. For example, the 2D image of interest 50A may be displayed on the screen 16A in a way that it is displayed in a color different from the remaining 2D images 50, a way that the 2D image of interest 50A is patterned, or a way that the 2D image of interest 50A is brighter than the remaining 2D images 50. Even in such examples, the 2D image of interest 50A becomes distinguishable from the remaining 2D images 50.

[0086] As an example, Figure 14 shows a state in which a second image 62 is displayed on the screen 16A. The second image 62 includes a first image area 81 and a third image area 83. As an example, the first image area 81 and the third image area 83 are displayed on the screen 16A aligned in the left-right direction of the second image 62. The first image area 81 is similar to the first image area 81 of the first image 61 (see Figure 10). The third image area 83 includes a three-dimensional image 52. The third image area 83 is an example of a "second area" according to the technology of the present disclosure.

[0087] The second display control unit 46A includes a plurality of two-dimensional images 50 in the first image area 81 based on the plurality of two-dimensional images 50 included in the inspection support information 56. The second display control unit 46A also includes a three-dimensional image 52 in the third image area 83 based on the three-dimensional image 52 included in the inspection support information 56.

[0088] The third image area 83 includes the three-dimensional image 52 in a two-dimensional state as a result of rendering. For example, the inspector 6 issues an instruction to the reception device 14 (see FIG. 9 ) to change the size of the three-dimensional image 52, thereby changing the size of the three-dimensional image 52. Furthermore, for example, the inspector 6 issues an instruction to the reception device 14 to rotate the three-dimensional image 52, thereby rotating the three-dimensional image 52.

[0089] A first image area 81 containing multiple two-dimensional images 50 and a third image area 83 containing a three-dimensional image 52 are displayed side by side on the screen 16A, making it possible to compare the multiple two-dimensional images 50 and the three-dimensional image 52.

[0090] Note that Figure 14 shows an example in which the first image area 81 and the third image area 83 are displayed on screen 16A lined up in the left-right direction of the second image 62, but for example, the first image area 81 and the third image area 83 may be displayed on screen 16A lined up in the top-bottom direction of the second image 62, or the first image area 81 and the third image area 83 may be displayed on screen 16A with the first image area 81 incorporated into part of the third image area 83.

[0091] As an example, as shown in Figure 15, when the second image 62 is displayed on the screen 16A and a selection instruction to select one of the multiple two-dimensional images 50 included in the first image area 81 is received by the reception device 14, the reception device 14 outputs a selection instruction signal indicating the selection instruction to the processor 20.

[0092] When a selection instruction signal is input to the processor 20, the second image selection unit 46B selects a target 2D image 50A corresponding to the selection instruction from the multiple 2D images 50 included in the inspection support information 56 in accordance with the selection instruction indicated by the selection instruction signal.

[0093] As an example, as shown in FIG. 16, the second pixel extracting unit 46C extracts a portion of interest 54A associated with a two-dimensional image of interest 50A from a three-dimensional image 52 included in the inspection support information 56.

[0094] The second image generation unit 46D generates a first image area 81 that includes a predetermined number of two-dimensional images 50 from the plurality of two-dimensional images 50 included in the inspection support information 56, and in which the two-dimensional image of interest 50A is surrounded by a frame 90. The second image generation unit 46D also generates a third image area 83 that includes the portion of interest 54A in the three-dimensional image 52, based on the portion of interest 54A extracted by the second pixel extraction unit 46C.

[0095] As an example, as shown in FIG. 17, the second image generating unit 46D generates a second image 62 by combining the generated first image area 81 and third image area 83.

[0096] The second display control unit 46A outputs second image data indicating the second image 62 generated by the second image generation unit 46D to the display 16. As a result, the second image 62 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of two-dimensional images 50 out of the plurality of two-dimensional images 50 are displayed on the screen 16A in a state where they are included in the first image area 81, and the two-dimensional image of interest 50A is displayed on the screen 16A in a state where it is surrounded by a frame 90.

[0097] Furthermore, the portion of interest 54A of the three-dimensional image 52 is displayed in an enlarged state on the screen 16A. By displaying the portion of interest 54A of the three-dimensional image 52 in an enlarged state on the screen 16A, the portion of interest 54A of the three-dimensional image 52 becomes visually identifiable. In other words, by displaying the portion of interest 54A of the three-dimensional image 52 in an enlarged state on the screen 16A, the portion of interest 54A becomes distinguishable from the remaining portions 54 of the multiple portions 54 that form the three-dimensional image 52. This makes the portion of interest 54A of the three-dimensional image 52 visually identifiable.

[0098] Furthermore, since the two-dimensional image of interest 50A is displayed on the screen 16A surrounded by the frame 90, the two-dimensional image of interest 50A becomes distinguishable from the remaining two-dimensional images 50 among the plurality of two-dimensional images 50. This makes it possible to visually identify the correspondence between the two-dimensional image of interest 50A and the portion of interest 54A of the three-dimensional image 52.

[0099] 17 , the portion of interest 54A of the three-dimensional image 52 is displayed on the screen 16A in an enlarged state. However, the entire three-dimensional image 52 may be displayed on the screen 16A at a size that fits within the third image area 83, and the portion of interest 54A of the three-dimensional image 52 may be displayed on the screen 16A in a manner that allows it to be distinguished from the remaining portion 54 in some other manner. For example, the portion of interest 54A may be displayed on the screen 16A in a different color from the remaining portion 54, with a pattern applied to the portion of interest 54A, with the portion of interest 54A surrounded by a frame, or with the pixels forming the outline of the portion of interest 54A having a higher brightness than the surrounding pixels. Even in such an example, the portion of interest 54A of the three-dimensional image 52 can be visually identified.

[0100] 18 shows, as an example, a state in which a third image 63 is displayed on the screen 16A. The third image 63 includes a first image area 81 and a fourth image area 84. As an example, the first image area 81 and the fourth image area 84 are displayed on the screen 16A aligned in the left-right direction of the third image 63. The first image area 81 is similar to the first image area 81 of the first image 61 (see FIG. 10 ). The fourth image area 84 includes a three-dimensional image 52 and a plurality of position-specifying images 92. Each position-specifying image 92 is an image for specifying a plurality of imaging positions where imaging was performed to obtain a plurality of two-dimensional images 50, and indicates the imaging positions corresponding to each two-dimensional image 50.

[0101] The third display control unit 48A includes a plurality of two-dimensional images 50 in the first image area 81 based on the plurality of two-dimensional images 50 included in the inspection support information 56. The third display control unit 48A also includes a three-dimensional image 52 in the fourth image area 84 based on the three-dimensional image 52 included in the inspection support information 56.

[0102] The fourth image area 84 includes the three-dimensional image 52 in a two-dimensional state as a result of rendering. For example, the inspector 6 issues an instruction to the reception device 14 (see FIG. 9 ) to change the size of the three-dimensional image 52, thereby changing the size of the three-dimensional image 52. Furthermore, for example, the inspector 6 issues an instruction to the reception device 14 to rotate the three-dimensional image 52, thereby rotating the three-dimensional image 52.

[0103] Furthermore, the third display control unit 48A includes a plurality of position specifying images 92 in the fourth image area 84 based on each imaging position included in the inspection support information 56. As an example, each position specifying image 92 is represented in the shape of a plate. The plurality of position specifying images 92 are included in the fourth image area 84 in a state where they can be compared with the three-dimensional image 52. Specifically, the plurality of position specifying images 92 are included in the fourth image area 84 in a state where they face the three-dimensional image 52 by being arranged around the three-dimensional image 52. In other words, the fourth image area 84 includes an image showing the plurality of position specifying images 92 and the three-dimensional image 52 facing each other.

[0104] By displaying the first image area 81 and the fourth image area 84 side by side on the screen 16A, the multiple two-dimensional images 50 and the three-dimensional image 52 can be compared, and the multiple two-dimensional images 50 and the multiple position-specific images 92 can be compared.

[0105] Note that Figure 18 shows an example in which the first image area 81 and the fourth image area 84 are displayed on screen 16A lined up in the left-right direction of the third image 63, but for example, the first image area 81 and the fourth image area 84 may also be displayed on screen 16A lined up in the top-bottom direction of the third image 63, or the first image area 81 and the fourth image area 84 may be displayed on screen 16A with the first image area 81 incorporated into part of the fourth image area 84.

[0106] The first image area 81 is an example of a "first area" and a "third area" according to the technology of the present disclosure. The fourth image area 84 is an example of a "second area" and a "fourth area" according to the technology of the present disclosure. The position identification image 92 is an example of a "position identification image" according to the technology of the present disclosure.

[0107] 19 as an example, when the third image 63 is displayed on the screen 16A and a selection instruction to select one of the plurality of position specifying images 92 included in the fourth image area 84 is received by the reception device 14, the reception device 14 outputs a selection instruction signal indicating the selection instruction to the processor 20. Note that, hereinafter, the position specifying image 92 selected from the plurality of position specifying images 92 will be referred to as the "interest position specifying image 92A."

[0108] When a selection instruction signal is input to the processor 20, the third image selection unit 48B selects an imaging position corresponding to the attention position specifying image 92A (hereinafter referred to as the "attention imaging position") from the plurality of imaging positions included in the inspection support information 56 in accordance with the selection instruction indicated by the selection instruction signal. Then, the third image selection unit 48B selects a attention two-dimensional image 50A corresponding to the attention imaging position from the plurality of two-dimensional images 50 included in the inspection support information 56. The attention two-dimensional image 50A is a two-dimensional image 50 obtained by imaging from the attention imaging position. The attention position specifying image 92A is an example of a "attention position specifying image" according to the technology of the present disclosure. The attention imaging position is an example of a "attention imaging position" according to the technology of the present disclosure.

[0109] 20 as an example, the third image generation unit 48C generates a fourth image area 84 including the three-dimensional image 52 based on the three-dimensional image 52 included in the inspection support information 56. The third image generation unit 48C also includes a plurality of position specifying images 92 in the fourth image area 84 based on the imaging positions and imaging orientations included in the inspection support information 56. The plurality of position specifying images 92 are arranged around the three-dimensional image 52, and are thereby included in the fourth image area 84 in a state facing the three-dimensional image 52. Each position specifying image 92 is arranged at a position corresponding to each imaging position and in an orientation corresponding to each imaging orientation.

[0110] Furthermore, the third image generating unit 48C includes, in the fourth image region 84, a focus position specifying image 92A that corresponds to the focus imaging position among the plurality of position specifying images 92. The focus position specifying image 92A is included in the fourth image region 84 in a state in which it is distinguishable from the remaining position specifying images 92 among the plurality of position specifying images 92. In the example shown in FIG. 20 , as an example of a state in which the focus position specifying image 92A is distinguishable from the remaining position specifying images 92, the focus position specifying image 92A is displayed in a color different from that of the remaining position specifying images 92. Note that the focus position specifying image 92A may be surrounded by a frame, or a pattern may be added to the focus position specifying image 92A. Furthermore, the brightness of the pixels forming the outline of the focus position specifying image 92A may be increased compared to the surrounding pixels.

[0111] Furthermore, the third image generation unit 48C generates a first image area 81 that includes a predetermined number of two-dimensional images 50 from the multiple two-dimensional images 50 included in the inspection support information 56, and in which the two-dimensional image of interest 50A is surrounded by a frame 90.

[0112] As an example, as shown in FIG. 21, the third image generating unit 48C generates a third image 63 by combining the generated first image area 81 and fourth image area 84.

[0113] The third display control unit 48A outputs third image data indicating the third image 63 generated by the third image generation unit 48C to the display 16. As a result, the third image 63 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of two-dimensional images 50 out of the multiple two-dimensional images 50 are displayed on the screen 16A while being included in the first image area 81, and the two-dimensional image of interest 50A is displayed on the screen 16A while being surrounded by a frame 90. In addition, the three-dimensional image 52 is displayed on the screen 16A.

[0114] Furthermore, the multiple position specifying images 92 are arranged around the three-dimensional image 52 and are displayed on the screen 16A facing the three-dimensional image 52, and the attention position specifying image 92A is displayed on the screen 16A in a state where it can be distinguished from the remaining position specifying images 92 among the multiple position specifying images 92. By displaying the attention position specifying image 92A on the screen 16A in a state where it can be distinguished from the remaining position specifying images 92, the attention portion 54A in the three-dimensional image 52 that corresponds to the attention position specifying image 92A becomes visually identifiable. In other words, the imaging position and imaging orientation are specified by the attention position specifying image 92A, and the correspondence between the specified imaging position and imaging orientation and the attention portion 54A in the three-dimensional image 52 becomes visually identifiable. As a result, the attention portion 54A in the three-dimensional image 52 becomes visually identifiable.

[0115] Furthermore, since the two-dimensional image of interest 50A is displayed on the screen 16A surrounded by the frame 90, the two-dimensional image of interest 50A becomes distinguishable from the remaining two-dimensional images 50 among the plurality of two-dimensional images 50. This makes it possible to visually identify the correspondence between the two-dimensional image of interest 50A and the portion of interest 54A of the three-dimensional image 52.

[0116] 21 , the entire three-dimensional image 52 is displayed on the screen 16A at a size that fits within the fourth image region 84. However, the portion of interest 54A of the three-dimensional image 52 may be enlarged and displayed on the screen 16A. Even in such an example, the portion of interest 54A of the three-dimensional image 52 becomes visually identifiable.

[0117] Next, the operation of the inspection support device 10 according to the first embodiment will be described with reference to FIGS. 22 to 26. FIG.

[0118] First, an example of the flow of the inspection support information generation process performed by the processor 20 of the inspection support device 10 will be described with reference to FIG.

[0119] In the inspection support information generation process shown in Fig. 22, first, in step ST10, the acquisition unit 32 (see Fig. 6) acquires a two-dimensional image 50 based on each image data received by the inspection support device 10. The acquisition unit 32 also acquires an imaging position corresponding to each two-dimensional image 50 based on each position data received by the inspection support device 10. Furthermore, the acquisition unit 32 acquires an imaging orientation corresponding to each two-dimensional image 50 based on each orientation data received by the inspection support device 10. After the process of step ST10 is executed, the inspection support information generation process proceeds to step ST12.

[0120] In step ST12, the three-dimensional image generator 34 (see FIG. 6 ) generates a three-dimensional image 52 showing the object 4 based on the plurality of two-dimensional images 50 acquired in step ST10. After the processing of step ST12 is executed, the inspection support information generation processing proceeds to step ST14.

[0121] In step ST14, the inspection support information generation unit 36 ​​(see FIG. 7 ) generates inspection support information 56, which is information that associates each of the two-dimensional images 50 acquired in step ST10 with the imaging positions corresponding to each of the two-dimensional images 50, the imaging attitudes corresponding to each of the two-dimensional images 50, and the parts 54 corresponding to each of the two-dimensional images 50. After the processing of step ST14 is executed, the inspection support information generation processing ends.

[0122] Next, an example of the flow of the inspection support process performed by the processor 20 of the inspection support device 10 will be described with reference to Fig. 23 to Fig. 26. First, an example of the flow of the mode setting process in the inspection support process will be described with reference to Fig. 23.

[0123] In the mode setting process shown in Fig. 23, first, in step ST20, the operation mode setting unit 42 (see Fig. 9) determines whether or not a mode setting instruction signal has been input to the processor 20. If the mode setting instruction signal has been input to the processor 20 in step ST20, the determination is affirmative, and the inspection support process proceeds to step ST22. If the mode setting instruction signal has not been input to the processor 20 in step ST20, the determination is negative, and the mode setting process proceeds to step ST32.

[0124] In step ST22, the operation mode setting unit 42 determines whether the mode setting instruction signal input to the processor 20 in step ST20 is the first mode setting signal. If the mode setting instruction signal is the first mode setting signal in step ST22, the determination is affirmative, and the inspection support process proceeds to step ST24. If the mode setting instruction signal is not the first mode setting signal in step ST22, the determination is negative, and the mode setting process proceeds to step ST26.

[0125] In step ST24, the operation mode setting unit 42 sets the first mode as the operation mode of the inspection support device 10. This executes the first mode process. After the process of step ST24 is executed, the mode setting process proceeds to step ST32.

[0126] In step ST26, the operation mode setting unit 42 determines whether the mode setting instruction signal input to the processor 20 in step ST20 is the second mode setting signal. If the mode setting instruction signal is the second mode setting signal in step ST26, the determination is affirmative, and the inspection support process proceeds to step ST28. If the mode setting instruction signal is not the second mode setting signal in step ST26, the determination is negative, and the mode setting process proceeds to step ST30.

[0127] In step ST28, the operation mode setting unit 42 sets the second mode as the operation mode of the inspection support device 10. This executes the second mode process. After the process of step ST28 is executed, the mode setting process proceeds to step ST32.

[0128] In step ST30, the operation mode setting unit 42 sets the third mode as the operation mode of the inspection support device 10. This executes the third mode process. After the process of step ST30 is executed, the mode setting process proceeds to step ST32.

[0129] In step ST32, the processor 20 determines whether a condition for terminating the mode setting process (hereinafter referred to as the "mode setting process termination condition") is met. One example of the mode setting process termination condition is a condition in which an end instruction from the inspector 6 is received by the reception device 14, and an end instruction signal from the reception device 14 is input to the processor 20. If the mode setting process termination condition is not met in step ST32, the determination is negative, and the mode setting process proceeds to step ST20. If the mode setting process termination condition is met in step ST32, the determination is positive, and the inspection support process including the mode setting process is terminated.

[0130] Next, an example of the flow of the first mode processing of the inspection support processing will be described with reference to FIG.

[0131] 24, first, in step ST40, the first display control unit 44A (see FIG. 10) displays the first image 61 on the screen 16A. After the processing of step ST40 is executed, the first mode processing proceeds to step ST42.

[0132] In step ST42, the first image selection unit 44B (see FIG. 11 ) determines whether or not a selection instruction signal indicating a selection instruction, which is an instruction to select any one of the multiple two-dimensional images 50, has been input to the processor 20. If a selection instruction signal indicating a selection instruction has been input to the processor 20 in step ST42, the determination is affirmative, and the first mode processing proceeds to step ST44. If a selection instruction signal indicating a selection instruction has not been input to the processor 20 in step ST42, the determination is negative, and the first mode processing proceeds to step ST52.

[0133] In step ST44, the first image selection unit 44B selects a two-dimensional image of interest 50A corresponding to the selection instruction indicated by the selection instruction signal from the plurality of two-dimensional images 50 included in the inspection support information 56. After the processing of step ST44 is executed, the first mode processing proceeds to step ST46.

[0134] In step ST46, the first pixel extractor 44C (see FIG. 12 ) acquires the imaging position and imaging orientation corresponding to the 2D image of interest 50A selected in step ST44 from the inspection support information 56. The first pixel extractor 44C also derives a viewpoint corresponding to the 2D image of interest 50A based on the acquired imaging position and imaging orientation. The first pixel extractor 44C then extracts pixels from the 3D image 52 included in the inspection support information 56 to include the 3D image 52 in the second image area 82 at the derived viewpoint. After the processing of step ST46 is executed, the first mode processing proceeds to step ST48.

[0135] In step ST48, the first image generation unit 44D (see FIG. 12) generates a first image region 81 that includes a predetermined number of two-dimensional images 50 from the plurality of two-dimensional images 50 included in the inspection support information 56 and in which the two-dimensional image of interest 50A is surrounded by a frame 90. Furthermore, the first image generation unit 44D generates a second image region 82 that includes a three-dimensional image 52 having a size that fits entirely within the second image region 82 from a viewpoint corresponding to the two-dimensional image of interest 50A, based on the pixels extracted in step ST46. The first image generation unit 44D (see FIG. 13) then generates a first image 61 by combining the generated first image region 81 and second image region 82. After the processing of step ST48 is executed, the first mode processing proceeds to step ST50.

[0136] In step ST50, the first display control unit 44A (see FIG. 13) outputs the first image data representing the first image 61 generated in step ST48 to the display 16. As a result, the first image 61 is displayed on the screen 16A of the display 16. After the processing of step ST50 is executed, the first mode processing proceeds to step ST52.

[0137] In step ST52, the processor 20 determines whether a condition for terminating the first mode processing (hereinafter referred to as the "first mode processing termination condition") is met. Examples of the first mode processing termination condition include a condition in which an end instruction signal from the reception device 14 is input to the processor 20 as a result of the reception device 14 receiving an end instruction from the inspector 6, or a condition in which a mode setting instruction signal indicating an instruction to set an operating mode different from the first mode is input to the processor 20. If the first mode processing termination condition is not met in step ST52, the determination is negative, and the first mode processing proceeds to step ST42. If the first mode processing termination condition is met in step ST52, the determination is positive, and the first mode processing ends.

[0138] Next, an example of the flow of the second mode processing of the inspection support processing will be described with reference to FIG.

[0139] 25, first, in step ST60, the second display control unit 46A (see FIG. 14) displays a second image 62 on the screen 16A. After the processing of step ST60 is executed, the second mode processing proceeds to step ST62.

[0140] In step ST62, the second image selection unit 46B (see FIG. 15 ) determines whether or not a selection instruction signal, which is an instruction to select any one of the multiple two-dimensional images 50, has been input to the processor 20. If a selection instruction signal indicating a selection instruction has been input to the processor 20 in step ST62, the determination is affirmative, and the second mode processing proceeds to step ST64. If a selection instruction signal indicating a selection instruction has not been input to the processor 20 in step ST62, the determination is negative, and the second mode processing proceeds to step ST72.

[0141] In step ST64, the second image selection unit 46B selects a two-dimensional image of interest 50A corresponding to the selection instruction indicated by the selection instruction signal from the plurality of two-dimensional images 50 included in the inspection support information 56. After the processing of step ST64 is executed, the second mode processing proceeds to step ST66.

[0142] In step ST66, the second pixel extraction unit 46C (see FIG. 16 ) extracts the portion of interest 54A associated with the two-dimensional image of interest 50A from the three-dimensional image 52 included in the inspection support information 56. After the processing of step ST66 is executed, the second mode processing proceeds to step ST68.

[0143] In step ST68, the second image generation unit 46D (see FIG. 16 ) generates a first image region 81 that includes a predetermined number of two-dimensional images 50 from the multiple two-dimensional images 50 included in the inspection support information 56 and in which the two-dimensional image of interest 50A is surrounded by a frame 90. The second image generation unit 46D also generates a third image region 83 that includes the portion of interest 54A in the three-dimensional image 52, based on the portion of interest 54A extracted in step ST66. The second image generation unit 46D (see FIG. 17 ) then generates the second image 62 by combining the generated first image region 81 and third image region 83. After the processing of step ST68 is executed, the second mode processing proceeds to step ST70.

[0144] In step ST70, the second display control unit 46A (see FIG. 17) outputs the second image data representing the second image 62 generated in step ST68 to the display 16. As a result, the second image 62 is displayed on the screen 16A of the display 16. After the processing of step ST70 is executed, the second mode processing proceeds to step ST72.

[0145] In step ST72, the processor 20 determines whether a condition for terminating the second mode processing (hereinafter referred to as the "second mode processing termination condition") is met. Examples of the second mode processing termination condition include a condition in which an end instruction signal from the reception device 14 is input to the processor 20 as a result of the reception device 14 receiving an end instruction from the inspector 6, or a condition in which a mode setting instruction signal indicating an instruction to set an operating mode different from the second mode is input to the processor 20. If the second mode processing termination condition is not met in step ST72, the determination is negative, and the second mode processing proceeds to step ST62. If the second mode processing termination condition is met in step ST72, the determination is positive, and the second mode processing ends.

[0146] Next, an example of the flow of the third mode processing of the inspection support processing will be described with reference to FIG.

[0147] 26, first, in step ST80, the third display control unit 48A (see FIG. 18) displays the third image 63 on the screen 16A. After the processing of step ST80 is executed, the third mode processing proceeds to step ST82.

[0148] In step ST82, the third image selection unit 48B (see FIG. 19 ) determines whether or not a selection instruction signal indicating a selection instruction, which is an instruction to select the interest position specifying image 92A from the plurality of position specifying images 92, has been input to the processor 20. If a selection instruction signal indicating a selection instruction has been input to the processor 20 in step ST82, the determination is affirmative, and the third mode processing proceeds to step ST84. If a selection instruction signal indicating a selection instruction has not been input to the processor 20 in step ST82, the determination is negative, and the third mode processing proceeds to step ST90.

[0149] In step ST84, the third image selection unit 48B selects, in accordance with the selection instruction indicated by the selection instruction signal, a target imaging position corresponding to the target imaging position identification image 92A from the plurality of imaging positions included in the inspection support information 56. Then, the third image selection unit 48B selects a target two-dimensional image 50A corresponding to the target imaging position from the plurality of two-dimensional images 50 included in the inspection support information 56. After the processing of step ST84 is executed, the third mode processing proceeds to step ST86.

[0150] In step ST86, the third image generation unit 48C (see FIG. 20) generates a fourth image area 84 including the three-dimensional image 52 based on the three-dimensional image 52 included in the inspection support information 56. The third image generation unit 48C also includes multiple position specifying images 92 in the fourth image area 84 based on the imaging position and imaging posture included in the inspection support information 56. The third image generation unit 48C also includes a focus position specifying image 92A corresponding to the focus imaging position selected in step ST84 in the fourth image area 84. The third image generation unit 48C (see FIG. 21) then generates the third image 63 by combining the generated first image area 81 and fourth image area 84. After the processing of step ST86 is executed, the third mode processing proceeds to step ST88.

[0151] In step ST88, the third display control unit 48A (see FIG. 21) outputs the third image data indicating the third image 63 generated in step ST86 to the display 16. As a result, the third image 63 is displayed on the screen 16A of the display 16. After the processing of step ST88 is executed, the third mode processing proceeds to step ST90.

[0152] In step ST90, the processor 20 determines whether a condition for terminating the third mode processing (hereinafter referred to as the "third mode processing termination condition") is met. Examples of the third mode processing termination condition include a condition in which an end instruction from the inspector 6 is received by the reception device 14, and an end instruction signal from the reception device 14 is input to the processor 20, or a condition in which a mode setting instruction signal indicating an instruction to set an operating mode different from the third mode is input to the processor 20. In step ST90, if the third mode processing termination condition is not met, the determination is negative, and the third mode processing proceeds to step ST82. In step ST90, if the third mode processing termination condition is met, the determination is positive, and the third mode processing ends. Note that the inspection support method described above as the operation of the inspection support device 10 is an example of an "image processing method" according to the technology disclosed herein.

[0153] As described above, in the inspection support device 10 according to the first embodiment, the processor 20 displays on the screen 16A a plurality of 2D images 50 used to generate a 3D image 52 representing the object 4 in real space, the plurality of 2D images 50 corresponding to a plurality of portions 54 of the 3D image 52, in a manner that allows comparison with the 3D image 52 (see FIGS. 10, 14, and 18). The processor 20 also selects a 2D image of interest 50A from the plurality of 2D images 50 in accordance with a given selection instruction (see FIGS. 11, 15, and 19). The processor 20 then displays on the screen 16A a visually identifiable portion 54A of the plurality of portions 54 corresponding to the 2D image of interest 50A (see FIGS. 13, 17, and 21). Therefore, the correspondence between each 2D image 50 and the region of the object 4 corresponding to each 2D image 50 can be visually grasped.

[0154] Furthermore, the state in which the plurality of two-dimensional images 50 and the three-dimensional image 52 can be compared means that a first image area 81 including the plurality of two-dimensional images 50 and an image area including the three-dimensional image 52 (i.e., the second image area 82, the third image area 83, or the fourth image area 84) are arranged side by side (see FIGS. 10, 14, and 18). Therefore, the plurality of two-dimensional images 50 and the three-dimensional image 52 can be visually compared.

[0155] Furthermore, a state in which the attention portion 54A is visually identifiable includes a state in which the attention portion 54A is distinguishable from the remaining portions 54 of the plurality of portions 54 (see FIGS. 17 and 21). Therefore, for example, the visibility of the attention portion 54A can be improved compared to a case in which the attention portion 54A is not distinguishable from the remaining portions 54.

[0156] Furthermore, a state in which the portion of interest 54A is visually identifiable includes a state in which the two-dimensional image of interest 50A is distinguishable from the remaining two-dimensional images 50 of the plurality of two-dimensional images 50 (see FIGS. 13 , 17 , and 21 ). Therefore, for example, the visibility of the portion of interest 54A can be improved compared to a case in which the two-dimensional image of interest 50A is not distinguishable from the remaining two-dimensional images 50. Furthermore, when the plurality of two-dimensional images 50 and the three-dimensional image 52 are displayed on the screen 16A in a state in which they can be compared and the portion of interest 54A is distinguishable from the remaining portions 50 (see FIGS. 17 and 21 ), the correspondence between the two-dimensional image of interest 50A and the portion of interest 54A can be visually identified.

[0157] The processor 20 also displays on the screen 16A a plurality of position specifying images 92 that can identify a plurality of imaging positions where imaging was performed to obtain the plurality of two-dimensional images 50, in a manner that allows comparison with the three-dimensional image 52 (see FIG. 18 ). The processor 20 also selects, from the plurality of imaging positions, an imaging position corresponding to an attention-position specifying image 92A selected from the plurality of position specifying images 92 in accordance with a selection instruction (see FIG. 19 ). The processor 20 then selects, from the plurality of two-dimensional images 50, a two-dimensional image 50 obtained by imaging from the attention imaging position as an attention two-dimensional image 50A (see FIG. 19 ). Therefore, by selecting the attention-position specifying image 92A from the plurality of position specifying images 92, the attention two-dimensional image 50A can be selected from the plurality of two-dimensional images 50.

[0158] Furthermore, the state in which the plurality of position specifying images 92 and the three-dimensional image 52 can be compared includes a state in which the plurality of position specifying images 92 and the three-dimensional image 52 are opposed to each other (see FIG. 19 ). Therefore, based on the state in which the plurality of position specifying images 92 and the three-dimensional image 52 are opposed to each other, it is possible to select an attention position specifying image 92A corresponding to the attention portion 54A of the three-dimensional image 52 from the plurality of position specifying images 92.

[0159] Furthermore, the state in which the plurality of two-dimensional images 50 and the three-dimensional image 52 can be compared is a state in which the first image area 81 including the plurality of two-dimensional images 50 and the fourth image area 84 (i.e., an image area including an image showing the plurality of position specifying images 92 and the three-dimensional image 52 facing each other) are arranged side by side (see FIGS. 18 and 19 ). Therefore, the plurality of two-dimensional images 50, the plurality of portions 54 of the three-dimensional image 52, and the plurality of position specifying images 92 can be visually compared.

[0160] Furthermore, the state in which the attention portion 54A is visually identifiable includes a state in which the attention position specifying image 92A is distinguishable from the remaining position specifying images 92 among the plurality of position specifying images 92. Therefore, for example, the visibility of the attention portion 54A can be improved compared to a case in which the attention position specifying image 92A is not distinguishable from the remaining position specifying images 92.

[0161] The inspection support device 10 (see FIG. 9 ) also has an operation mode (e.g., a first mode and a second mode) in which a plurality of two-dimensional images 50 and a three-dimensional image 52 are displayed on the screen 16A in a state in which they can be compared, and an operation mode (i.e., a third mode) in which a plurality of position-specific images 92 are displayed on the screen 16A in a state in which they can be compared with the three-dimensional image 52. The processor 20 sets the operation mode according to a given setting instruction. Therefore, for example, depending on the portion of interest 54A, the screen 16A can be selectively switched between a state in which the plurality of two-dimensional images 50 and a three-dimensional image 52 can be compared and a state in which the plurality of position-specific images 92 can be compared with the three-dimensional image 52.

[0162] In the first mode, the three-dimensional image 52 is displayed on the screen 16A from a viewpoint corresponding to the two-dimensional image of interest 50A (see FIG. 13). Therefore, based on the viewpoint corresponding to the two-dimensional image of interest 50A, it is possible to visually identify the portion of interest 54A among the multiple portions 54 that corresponds to the two-dimensional image of interest 50A.

[0163] The inspection support device 10 according to the first embodiment has the first mode, the second mode, and the third mode, but any one of the first mode, the second mode, and the third mode may be omitted. Also, the inspection support device 10 according to the first embodiment may have only one of the first mode, the second mode, and the third mode.

[0164] Second Embodiment Next, a second embodiment of the present disclosure will be described.

[0165] In the second embodiment, the inspection support device 10 has the following configuration changes from that of the first embodiment.

[0166] As an example, as shown in FIG. 27 , the processor 20 operates as a fourth display control unit 94A. The fourth display control unit 94A displays a fourth image 64 on the screen 16A. The fourth image 64 includes a first image area 81 and a fifth image area 85. As an example, the first image area 81 and the fifth image area 85 are displayed on the screen 16A aligned in the left-right direction of the fifth image. The first image area 81 is similar to the first image area 81 of the first image 61 (see FIG. 10 ). The fifth image area 85 includes a three-dimensional image 52.

[0167] The fourth display control unit 94A includes a plurality of two-dimensional images 50 in the first image area 81 based on the plurality of two-dimensional images 50 included in the inspection support information 56. The fourth display control unit 94A also includes a three-dimensional image 52 in the fifth image area 85 based on the three-dimensional image 52 included in the inspection support information 56.

[0168] The fifth image area 85 includes the three-dimensional image 52 in a two-dimensional state as a result of rendering. For example, the inspector 6 issues an instruction to the reception device 14 (see FIG. 9 ) to change the size of the three-dimensional image 52, thereby changing the size of the three-dimensional image 52. Furthermore, for example, the inspector 6 issues an instruction to the reception device 14 to rotate the three-dimensional image 52, thereby rotating the three-dimensional image 52.

[0169] By displaying a first image area 81 containing multiple two-dimensional images 50 and a fifth image area 85 containing a three-dimensional image 52 side by side on the screen 16A, the multiple two-dimensional images 50 and the three-dimensional image 52 can be compared.

[0170] Note that Figure 27 shows an example in which the first image area 81 and the fifth image area 85 are displayed on the screen 16A lined up in the left-right direction of the fourth image 64, but for example, the first image area 81 and the fifth image area 85 may also be displayed on the screen 16A lined up in the top-bottom direction of the fourth image 64, or the first image area 81 and the fifth image area 85 may be displayed on the screen 16A with the first image area 81 incorporated into part of the fifth image area 85.

[0171] The first image area 81 is an example of a "first area" according to the technology of the present disclosure. The fifth image area 85 is an example of a "second area" according to the technology of the present disclosure.

[0172] As an example, as shown in Figure 28, when the fourth image 64 is displayed on the screen 16A and a selection instruction to select any part 54 of the three-dimensional image 52 included in the fifth image area 85 is received by the reception device 14, the reception device 14 outputs a selection instruction signal indicating the selection instruction to the processor 20.

[0173] The processor 20 operates as a fourth image selection unit 94B. When a selection instruction signal is input to the processor 20, the fourth image selection unit 94B selects a portion 54 corresponding to the selection instruction (i.e., a portion of interest 54A) from the plurality of portions 54 included in the inspection support information 56 in accordance with the selection instruction indicated by the selection instruction signal. The fourth image selection unit 94B also selects a two-dimensional image 50 corresponding to the portion of interest 54A (i.e., a two-dimensional image of interest 50A) from the plurality of two-dimensional images 50 included in the inspection support information 56.

[0174] 29 , the processor 20 operates as a fourth pixel extractor 94C and a fourth image generator 94D. The fourth pixel extractor 94C extracts a portion of interest 54A corresponding to the two-dimensional image of interest 50A from the three-dimensional image 52 included in the inspection support information 56.

[0175] The fourth image generation unit 94D generates a first image region 81 that includes a predetermined number of two-dimensional images 50 out of the multiple two-dimensional images 50 included in the inspection support information 56, and in which a two-dimensional image of interest 50A is surrounded by a frame 90. The fourth image generation unit 94D also generates a fifth image region 85 that includes a three-dimensional image 52 based on a three-dimensional image 52 included in the inspection support information 56. The fourth image generation unit 94D further includes a portion of interest 54A extracted by the fourth pixel extraction unit 94C in the fifth image region 85 in a state where the portion of interest 54A is distinguishable from the remaining portion 54. For example, the fourth image generation unit 94D includes the portion of interest 54A in the fifth image region 85 in a state where the portion of interest 54A is represented in a color different from the remaining portion 54.

[0176] As an example, as shown in FIG. 30, the fourth image generating unit 94D generates a fourth image 64 by combining the generated first image area 81 and fifth image area 85.

[0177] The fourth display control unit 94A outputs fourth image data indicating the fourth image 64 generated by the fourth image generation unit 94D to the display 16. As a result, the fourth image 64 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of two-dimensional images 50 out of the plurality of two-dimensional images 50 are displayed on the screen 16A while being included in the first image area 81, and the two-dimensional image of interest 50A is displayed on the screen 16A while being surrounded by a frame 90. By displaying the two-dimensional image of interest 50A on the screen 16A while being surrounded by the frame 90, the two-dimensional image of interest 50A becomes distinguishable from the remaining two-dimensional images 50 out of the plurality of two-dimensional images 50.

[0178] Furthermore, the portion of interest 54A included in the three-dimensional image 52 is displayed on the screen 16A in a state where it can be distinguished from the remaining portion 54. By displaying the portion of interest 54A on the screen 16A in a state where it can be distinguished from the remaining portion 54, the portion of interest 54A in the three-dimensional image 52 becomes visually identifiable.

[0179] 30, the attention portion 54A is displayed in a color different from the remaining portion 54. However, the attention portion 54A may be displayed on the screen 16A in another manner so as to be distinguishable from the remaining portion 54. For example, the attention portion 54A may be displayed on the screen 16A in a manner in which the attention portion 54A is surrounded by a frame, in a manner in which a pattern is added to the attention portion 54A, or in a manner in which the attention portion 54A is brighter than the remaining portion 54. Even in such an example, the attention portion 54A becomes distinguishable from the remaining portion 54.

[0180] Next, the operation of the inspection support device 10 according to the second embodiment will be described with reference to Fig. 31. Fig. 31 shows an example of the flow of the inspection support process according to the second embodiment.

[0181] 31 , first, in step ST100, the fourth display control unit 94A (see FIG. 27 ) displays the fourth image 64 on the screen 16A. After the processing of step ST100 is executed, the inspection support processing proceeds to step ST102.

[0182] In step ST102, the fourth image selection unit 94B (see FIG. 28 ) determines whether or not a selection instruction signal indicating a selection instruction, which is an instruction to select any portion 54 of the three-dimensional image 52, has been input to the processor 20. If a selection instruction signal indicating a selection instruction has been input to the processor 20 in step ST102, the determination is affirmative, and the inspection support processing proceeds to step ST104. If a selection instruction signal indicating a selection instruction has not been input to the processor 20 in step ST102, the determination is negative, and the inspection support processing proceeds to step ST112.

[0183] In step ST104, the fourth image selection unit 94B selects a portion of interest 54A corresponding to the selection instruction indicated by the selection instruction signal from the plurality of portions 54 included in the inspection support information 56. The fourth image selection unit 94B also selects a two-dimensional image of interest 50A corresponding to the portion of interest 54A from the plurality of two-dimensional images 50 included in the inspection support information 56. After the processing of step ST104 is executed, the inspection support processing proceeds to step ST106.

[0184] In step ST106, the fourth pixel extraction unit 94C (see FIG. 29 ) extracts the portion of interest 54A corresponding to the two-dimensional image of interest 50A from the three-dimensional image 52 included in the inspection support information 56. After the processing of step ST106 is executed, the inspection support processing proceeds to step ST108.

[0185] In step ST108, the fourth image generation unit 94D (see FIG. 30 ) generates a first image region 81 including a predetermined number of two-dimensional images 50 from the plurality of two-dimensional images 50 included in the inspection support information 56, and in which a two-dimensional image of interest 50A is surrounded by a frame 90. The fourth image generation unit 94D also generates a fifth image region 85 including the three-dimensional image 52 based on the three-dimensional image 52 included in the inspection support information 56. Furthermore, the fourth image generation unit 94D includes the portion of interest 54A extracted in step ST106 in the fifth image region 85 in a state where it is distinguishable from the remaining portion 54. The fourth image generation unit 94D then generates a fourth image 64 by combining the generated first image region 81 and fifth image region 85. After the processing of step ST108 is executed, the inspection support processing proceeds to step ST110.

[0186] In step ST110, the fourth display control unit 94A (see FIG. 30) outputs fourth image data indicating the fourth image 64 generated in step ST108 to the display 16. As a result, the fourth image 64 is displayed on the screen 16A of the display 16. After the processing of step ST110 is executed, the inspection support processing proceeds to step ST112.

[0187] In step ST112, the processor 20 determines whether a condition for terminating the inspection support process (hereinafter referred to as the "termination condition") is met. One example of the termination condition is a condition in which an end instruction signal from the reception device 14 is input to the processor 20 as a result of the reception device 14 receiving an end instruction from the inspector 6. If the termination condition is not met in step ST112, the determination is negative, and the inspection support process proceeds to step ST102. If the termination condition is met in step ST112, the determination is positive, and the inspection support process ends.

[0188] As described above, in the inspection support device 10 according to the second embodiment, the processor 20 displays on the screen 16A a plurality of two-dimensional images 50 used to generate a three-dimensional image 52 showing the object 4 in real space, the plurality of two-dimensional images 50 corresponding to a plurality of portions 54 of the three-dimensional image 52, in a manner that allows comparison with the three-dimensional image 52 (see FIG. 27 ). Furthermore, the processor 20 selects a portion of interest 54A from the plurality of portions 54 in accordance with a given selection instruction, and selects a two-dimensional image of interest 50A corresponding to the portion of interest 54A from the plurality of two-dimensional images 50 (see FIG. 28 ). The processor 20 then displays the two-dimensional image of interest 50A on the screen 16A in a manner that allows it to be distinguished from the remaining two-dimensional images 50 of the plurality of two-dimensional images 50. Therefore, the correspondence between each two-dimensional image 50 and the region of the object 4 corresponding to each two-dimensional image 50 can be visually grasped.

[0189] Furthermore, the processor 20 displays the portion of interest 54A on the screen 16A in a visually identifiable state (see FIG. 30). Therefore, the correspondence between the two-dimensional image of interest 50A and the portion of interest 54A can be visually identified.

[0190] Furthermore, the state in which the attention portion 54A is visually identifiable includes a state in which the attention portion 54A is distinguishable from the remaining portions 54 of the plurality of portions 54 (see FIG. 30 ). Therefore, for example, the visibility of the attention portion 54A can be improved compared to when the attention portion 54A is not distinguishable from the remaining portions 54.

[0191] The operation mode of the inspection support device 10 according to the second embodiment may be added as a fourth mode to the operation modes of the inspection support device 10 according to the first embodiment.

[0192] Furthermore, in the above embodiment, processor 20 is exemplified, but instead of processor 20, or together with processor 20, at least one other CPU, at least one GPU, and / or at least one TPU may be used.

[0193] Furthermore, in the above embodiment, an example has been described in which the inspection support information generation program 30 and the inspection support program 40 are stored in the storage 22, but the technology of the present disclosure is not limited to this. For example, the inspection support information generation program 30 and / or the inspection support program 40 may be stored in a portable, non-transitory, computer-readable storage medium (hereinafter simply referred to as a "non-transitory storage medium") such as an SSD or a USB memory. The inspection support information generation program 30 and / or the inspection support program 40 stored in the non-transitory storage medium may be installed in the computer 12 of the inspection support device 10.

[0194] In addition, the inspection support information generation program 30 and / or the inspection support program 40 may be stored in a storage device such as another computer or server device connected to the inspection support device 10 via a network, and the inspection support information generation program 30 and / or the inspection support program 40 may be downloaded and installed on the computer 12 in response to a request from the inspection support device 10.

[0195] Furthermore, it is not necessary to store all of the inspection support information generation program 30 and / or the inspection support program 40 in a storage device such as another computer or server device connected to the inspection support device 10, or in the storage 22; only part of the inspection support information generation program 30 and / or the inspection support program 40 may be stored.

[0196] Furthermore, although the inspection support device 10 has a built-in computer 12, the technology of the present disclosure is not limited to this, and for example, the computer 12 may be provided outside the inspection support device 10.

[0197] In addition, in the above embodiment, the computer 12 including the processor 20, the storage 22, and the RAM 24 is exemplified, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 12. Furthermore, instead of the computer 12, a combination of a hardware configuration and a software configuration may be used.

[0198] Furthermore, the following various processors can be used as hardware resources for executing the various processes described in the above embodiments. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing various processes by executing software, i.e., a program. Examples of processors include dedicated electronic circuits, such as FPGAs, PLDs, and ASICs, which are processors with a circuit configuration designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor uses the memory to execute various processes.

[0199] The hardware resources that execute various processes may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resources that execute various processes may be a single processor.

[0200] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes various processes. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute various processes, on a single IC chip, as typified by SoC. In this way, various processes are realized using one or more of the above-mentioned various processors as hardware resources.

[0201] Furthermore, the hardware structure of these various processors can be, more specifically, electronic circuits that combine circuit elements such as semiconductor devices. The above-described gaze detection process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0202] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0203] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0204] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A processor is provided. The processor, displaying on a screen a plurality of two-dimensional images used in generating a three-dimensional image showing an object in real space, the two-dimensional images corresponding to a plurality of portions of the three-dimensional image, in a state in which the three-dimensional image can be compared with the two-dimensional images; selecting a two-dimensional image of interest from the plurality of two-dimensional images according to a given selection instruction; A portion of interest corresponding to the two-dimensional image of interest among the plurality of portions is displayed on the screen in a visually identifiable state. Image processing device.

2. The state in which the plurality of two-dimensional images and the three-dimensional image can be compared is a state in which a first area including the plurality of two-dimensional images and a second area including the three-dimensional image are arranged side by side. The image processing device according to claim 1 .

3. The state in which the portion of interest is visually identifiable includes a state in which the portion of interest is distinguishable from the remaining portions of the plurality of portions. The image processing device according to claim 1 .

4. The state in which the portion of interest can be visually identified includes a state in which the two-dimensional image of interest can be distinguished from the remaining two-dimensional images of the plurality of two-dimensional images. The image processing device according to claim 1 .

5. The processor, displaying a plurality of position-specific images, which can specify a plurality of imaging positions where imaging was performed to obtain the plurality of two-dimensional images, on the screen in a state in which they can be compared with the three-dimensional image; selecting, from the plurality of imaging positions, an imaging position corresponding to the attention position specifying image selected from the plurality of position specifying images in accordance with the selection instruction, as an attention imaging position; A two-dimensional image obtained by performing the imaging from the imaging position of interest is selected as the two-dimensional image of interest from the plurality of two-dimensional images. The image processing device according to claim 1 .

6. The state in which the plurality of position-specific images and the three-dimensional image can be compared includes a state in which the plurality of position-specific images and the three-dimensional image are opposed to each other. The image processing device according to claim 5 .

7. The state in which the plurality of two-dimensional images and the three-dimensional image can be compared is a state in which a third area including the plurality of two-dimensional images and a fourth area including an image showing a state in which the plurality of position-specifying images and the three-dimensional image are opposed to each other are arranged side by side. The image processing device according to claim 5 .

8. The state in which the attention portion is visually identifiable includes a state in which the attention position specifying image is distinguishable from the remaining position specifying images of the plurality of position specifying images. The image processing device according to claim 5 .

9. the image processing device has a first operation mode in which the plurality of two-dimensional images and the three-dimensional image are displayed on the screen in a state in which they can be compared, and a second operation mode in which the plurality of position specifying images are displayed on the screen in a state in which they can be compared with the three-dimensional image, The processor sets the operation mode to either the first operation mode or the second operation mode in accordance with a given setting instruction. The image processing device according to claim 5 .

10. The three-dimensional image is displayed on the screen at a viewpoint corresponding to the two-dimensional image of interest. The image processing device according to claim 5 .

11. A processor is provided. The processor, displaying on a screen a plurality of two-dimensional images used in generating a three-dimensional image showing an object in real space, the two-dimensional images corresponding to a plurality of portions of the three-dimensional image, in a state in which the three-dimensional image can be compared with the two-dimensional images; Selecting a portion of interest from the plurality of portions in accordance with a given selection instruction; selecting a two-dimensional image of interest corresponding to the portion of interest from among the plurality of two-dimensional images; The plurality of two-dimensional images are displayed on the screen in a state in which the two-dimensional image of interest can be distinguished from the remaining two-dimensional images among the plurality of two-dimensional images. Image processing device.

12. The processor, displaying a plurality of position-specific images, which can specify a plurality of imaging positions where imaging was performed to obtain the plurality of two-dimensional images, on the screen in a state in which they can be compared with the three-dimensional image; selecting an interest position specifying image from the plurality of position specifying images in accordance with the selection instruction; The two-dimensional image obtained by performing the imaging from the imaging position specified from the image of the position of interest is selected as the two-dimensional image of interest from among the plurality of two-dimensional images. The image processing device according to claim 11.

13. Displaying on a screen a plurality of two-dimensional images used in generating a three-dimensional image showing an object in real space, the plurality of two-dimensional images corresponding to a plurality of portions of the three-dimensional image, in a state in which the three-dimensional image can be compared with the two-dimensional images; selecting a two-dimensional image of interest from the plurality of two-dimensional images in accordance with a given selection instruction; and a target portion corresponding to the target two-dimensional image among the plurality of portions is displayed on the screen in a visually identifiable state; An image processing method comprising:

14. Displaying on a screen a plurality of two-dimensional images used in generating a three-dimensional image showing an object in real space, the plurality of two-dimensional images corresponding to a plurality of portions of the three-dimensional image, in a state in which the three-dimensional image can be compared with the two-dimensional images; selecting a two-dimensional image of interest from the plurality of two-dimensional images in accordance with a given selection instruction; and a target portion corresponding to the target two-dimensional image among the plurality of portions is displayed on the screen in a visually identifiable state; A program for causing a computer to execute a process including the steps of: