Image processing device, image processing method, and program

Abstract

This image processing device comprises a processor. The processor: displays a three-dimensional image and a plurality of two-dimensional images which are used for generating the three-dimensional image that represents a subject in an actual space, and are associated with a plurality of portions of the three-dimensional image in a comparable state on a screen; selects an attention two-dimensional image from the plurality of two-dimensional images in accordance with a given selection instruction; and displays an attention portion corresponding to the attention two-dimensional image among the plurality of portions, in a visually identifiable state on the screen.

Description

Technical Field

[0001] The technology of the present disclosure relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Japanese Unexamined Patent Application Publication No. 2020-005186 discloses an image display system constituted by a computer system. The computer system inputs an image group including a plurality of images having different shooting dates, positions, and directions, displays a list of the image group on a list screen, and displays a first image selected from the image group on an individual screen based on a user operation. Further, the computer system determines an adjacent image for the first image based on a determination of a spatial positional relationship in a pair of the first image and a candidate image spatially adjacent to the first image, and a determination of an overlapping state regarding a shooting range, and selects, on the individual screen, an adjacent image for the first image as a second image based on a user operation, and displays the second image as a new first image.

[0003] Japanese Unexamined Patent Application Publication No. 2007-093661 discloses a navigation device mounted on an automobile that simultaneously displays a first map and a second map having a different expression form from the first map. The navigation device includes a display device, a map display unit, a current position calculation unit, a current position display unit, and a position designation reception unit. The map display unit displays the first map and the second map in different display areas of the display device. The current position calculation unit calculates the current position. The current position display unit displays a current position mark representing the current position calculated by the current position calculation unit on at least one of the first map and the second map displayed by the map display unit. The position designation reception unit receives, from the user, a designation of a position on the display area where the first map is displayed. The map display unit displays the second map in a distinguishable form at a position on the second map representing the same point as the point on the first map corresponding to the position designated by the position designation unit.

[0004] Japanese Patent Publication No. 2010-200024 discloses a stereoscopic image display device. The stereoscopic image display device comprises a display means, an instruction input means, an alignment means, and a display control means. The display means displays a list of thumbnail images generated from captured images taken from multiple viewpoints before displaying them stereoscopically. The instruction input means receives a selection instruction to select a thumbnail image from the list. When a selection instruction is received and the captured images are displayed stereoscopically, the alignment means aligns the captured images from multiple viewpoints corresponding to the selected thumbnail images with each other in the detection area of ​​a specific object in the captured images. The display control means adds detection area information indicating the detection area of ​​the specific object to the thumbnail image. [Overview of the project]

[0005] One embodiment of the technology of this disclosure provides, as an example, an image processing device, an image processing method, and a program that can visually grasp the correspondence between each two-dimensional image and the region of an object that corresponds to each two-dimensional image. [Means for solving the problem]

[0006] A first aspect of the technology of this disclosure is an image processing device comprising a processor, the processor displays on a screen in a manner that allows comparison between a plurality of two-dimensional images used to generate a three-dimensional image representing an object in real space, the plurality of two-dimensional images associated with a plurality of parts of the three-dimensional image and the three-dimensional image, the processor selects a two-dimensional image of interest from the plurality of two-dimensional images according to a given selection instruction, and displays on the screen in a manner that allows visual identification of the portion of interest corresponding to the two-dimensional image of interest among the plurality of parts.

[0007] A second aspect of the technology of this disclosure is an image processing apparatus according to the first aspect, wherein a state in which a plurality of two-dimensional images and a three-dimensional image can be compared is a state in which a first region containing a plurality of two-dimensional images and a second region containing a three-dimensional image are arranged side by side.

[0008] A third aspect of the technology of this disclosure is an image processing apparatus according to the first or second aspect, wherein the state in which the portion of interest can be visually identified includes a state in which the portion of interest can be distinguished from the remaining parts of a plurality of parts.

[0009] A fourth aspect of the technology of this disclosure is an image processing apparatus according to any one of the first to third aspects, wherein the state in which the portion of interest can be visually identified includes a state in which the two-dimensional image of interest can be distinguished from the remaining two-dimensional images among a plurality of two-dimensional images.

[0010] A fifth aspect of the technology of this disclosure is an image processing device according to any one of the first to fourth aspects, wherein the processor displays a plurality of position-identifying images on a screen in a manner that allows comparison with a three-dimensional image, which can identify a plurality of imaging positions where imaging has been performed to obtain a plurality of two-dimensional images, and, in accordance with a selection instruction, selects an imaging position corresponding to a selected position-identifying image of interest from the plurality of position-identifying images as a position-identifying position of interest, and selects a two-dimensional image obtained by imaging from the position-identifying position of interest from the plurality of two-dimensional images as a position-identifying two-dimensional image of interest.

[0011] A sixth aspect of the technology of this disclosure is an image processing apparatus according to the fifth aspect, wherein the state in which a plurality of location-identified images and a three-dimensional image can be compared includes a state in which a plurality of location-identified images and a three-dimensional image are placed opposite each other.

[0012] A seventh aspect of the technology of this disclosure is an image processing apparatus according to the fifth or sixth aspect, wherein the state in which a plurality of two-dimensional images and a three-dimensional image can be compared is a state in which a third region containing a plurality of two-dimensional images and a fourth region containing an image showing a configuration in which a plurality of location-identifying images and a three-dimensional image are facing each other are arranged side by side.

[0013] An eighth aspect of the technology of this disclosure is an image processing apparatus according to any one of the fifth to seventh aspects, wherein the state in which a portion of interest can be visually identified includes a state in which a portion of interest location identification image can be distinguished from the remaining location identification images among a plurality of location identification images.

[0014] A ninth aspect of the technology of this disclosure is an image processing apparatus according to any one of the fifth to eighth aspects, wherein the image processing apparatus has a first operating mode for displaying a plurality of two-dimensional images and a three-dimensional image on a screen in a manner that allows comparison, and a second operating mode for displaying a plurality of location-specific images on a screen in a manner that allows comparison with a three-dimensional image, and the processor sets either the first operating mode or the second operating mode according to a given setting instruction.

[0015] A tenth aspect of the technology of this disclosure is an image processing device relating to any one of the fifth to ninth aspects, wherein the three-dimensional image is displayed on the screen from a viewpoint corresponding to the two-dimensional image of interest.

[0016] An eleventh aspect of the technology of the present disclosure is an image processing device comprising a processor, the processor displays on a screen a plurality of two-dimensional images used to generate a three-dimensional image representing an object in real space, wherein the plurality of two-dimensional images associated with a plurality of parts of the three-dimensional image are displayed in a manner that allows comparison between the three-dimensional image and the two-dimensional image, and, according to a given selection instruction, selects a portion of interest from the plurality of parts, selects a two-dimensional image of interest that corresponds to the portion of interest from the plurality of two-dimensional images, and displays the two-dimensional image of interest on the screen in a manner that allows distinction between the two-dimensional image of interest and the remaining two-dimensional images from the plurality of two-dimensional images.

[0017] A twelfth aspect of the technology of this disclosure is an image processing apparatus according to the eleventh aspect, wherein the processor displays a plurality of position-identifying images on a screen in a manner that allows comparison with a three-dimensional image, which can identify a plurality of imaging positions where imaging has been performed to obtain a plurality of two-dimensional images, selects a position-identifying image of interest from the plurality of position-identifying images according to a selection instruction, and selects a two-dimensional image obtained by imaging performed from the imaging position identified from the position-identifying image of interest from the plurality of two-dimensional images as the two-dimensional image of interest.

[0018] A thirteenth aspect of the technology of this disclosure is an image processing method comprising: displaying a plurality of two-dimensional images used to generate a three-dimensional image of an object in real space, wherein the plurality of two-dimensional images associated with a plurality of parts of the three-dimensional image are displayed on a screen in a manner that allows for comparison with the three-dimensional image; selecting a two-dimensional image of interest from the plurality of two-dimensional images according to a given selection instruction; and displaying on a screen a portion of the plurality of parts that corresponds to the two-dimensional image of interest in a manner that allows for visual identification.

[0019] A fourteenth aspect of the technology of this disclosure is a program for causing a computer to perform a process that includes displaying a plurality of two-dimensional images used to generate a three-dimensional image of an object in real space, where the plurality of two-dimensional images associated with a plurality of parts of the three-dimensional image are displayed on a screen in a manner that allows for comparison with the three-dimensional image; selecting a two-dimensional image of interest from the plurality of two-dimensional images according to a given selection instruction; and displaying on the screen a portion of the plurality of parts that corresponds to the two-dimensional image of interest in a manner that allows for visual identification. [Brief explanation of the drawing]

[0020] [Figure 1] This is a perspective view showing an example of an inspection system according to the first embodiment. [Figure 2] This is a block diagram showing an example of an inspection support device according to the first embodiment. [Figure 3] This is a block diagram showing an example of an imaging device according to the first embodiment. [Figure 4]It is a block diagram showing an example of a functional configuration for realizing the inspection support information generation process according to the first embodiment. [Figure 5] It is a block diagram showing an example of data transmitted from the imaging device to the inspection support device according to the first embodiment. [Figure 6] It is a block diagram showing an example of the operations of the acquisition unit and the 3D image generation unit according to the first embodiment. [Figure 7] It is a block diagram showing an example of the operations of the 3D image generation unit and the inspection support information generation unit according to the first embodiment. [Figure 8] It is a block diagram showing an example of a functional configuration for realizing the inspection support process according to the first embodiment. [Figure 9] It is a block diagram showing an example of the operations of the operation mode setting unit, the first mode processing unit, the second mode processing unit, and the third mode processing unit according to the first embodiment. [Figure 10] It is a block diagram showing an example of the operation of the first display control unit according to the first embodiment. [Figure 11] It is a block diagram showing an example of the operation of the first image selection unit according to the first embodiment. [Figure 12] It is a block diagram showing an example of the operations of the first pixel extraction unit and the first image generation unit according to the first embodiment. [Figure 13] It is a block diagram showing an example of the operations of the first image generation unit and the first display control unit according to the first embodiment. [Figure 14] It is a block diagram showing an example of the operation of the second display control unit according to the first embodiment. [Figure 15] It is a block diagram showing an example of the operation of the second image selection unit according to the first embodiment. [Figure 16] It is a block diagram showing an example of the operations of the second pixel extraction unit and the second image generation unit according to the first embodiment. [Figure 17] It is a block diagram showing an example of the operations of the second image generation unit and the second display control unit according to the first embodiment. [Figure 18] It is a block diagram showing an example of the operation of the third display control unit according to the first embodiment. [Figure 19] This is a block diagram showing an example of the operation of the third image selection unit according to the first embodiment. [Figure 20] This is a block diagram showing an example of the operation of the third image generation unit according to the first embodiment. [Figure 21] This is a block diagram showing an example of the operation of the third image generation unit and the third display control unit according to the first embodiment. [Figure 22] This flowchart shows an example of the flow of the inspection support information generation process according to the first embodiment. [Figure 23] This flowchart shows an example of the flow of the mode setting process, which is part of the inspection support process according to the first embodiment. [Figure 24] This flowchart shows an example of the flow of the first mode processing among the inspection support processing according to the first embodiment. [Figure 25] This flowchart shows an example of the flow of the second mode processing among the inspection support processing according to the first embodiment. [Figure 26] This flowchart shows an example of the flow of the third mode processing among the inspection support processing according to the first embodiment. [Figure 27] This is a block diagram showing an example of the operation of the fourth display control unit according to the second embodiment. [Figure 28] This is a block diagram showing an example of the operation of the fourth image selection unit according to the second embodiment. [Figure 29] This is a block diagram showing an example of the operation of the fourth pixel extraction unit and the fourth image generation unit according to the second embodiment. [Figure 30] This is a block diagram showing an example of the operation of the fourth image generation unit and the fourth display control unit according to the second embodiment. [Figure 31] This flowchart shows an example of inspection support processing according to the second embodiment. [Modes for carrying out the invention]

[0021] Hereinafter, an example of an embodiment of the image processing apparatus, image processing method, and program relating to the technology of this disclosure will be described with reference to the attached drawings.

[0022] First, let's explain the terminology used in the following explanation.

[0023] CPU stands for "Central Processing Unit". GPU stands for "Graphics Processing Unit". HDD stands for "Hard Disk Drive". SSD stands for "Solid State Drive". RAM stands for "Random Access Memory". SRAM stands for "Static Random Access Memory". DRAM stands for "Dynamic Random Access Memory". EL stands for "Electro Luminescence". RAM stands for "Random Access Memory". CMOS stands for "Complementary Metal Oxide Semiconductor". GNSS stands for "Global Navigation Satellite System". GPS stands for "Global Positioning System". SfM stands for "Structure from Motion". MVS stands for "Multi-View Stereo". TPU stands for "Tensor Processing Unit". USB stands for "Universal Serial Bus". ASIC stands for "Application Specific Integrated Circuit". FPGA stands for "Field-Programmable Gate Array". PLD stands for "Programmable Logic Device". SoC stands for "System-on-a-chip". IC stands for "Integrated Circuit".

[0024] [First Embodiment] First, a first embodiment of this disclosure will be described.

[0025] As an example, as shown in Figure 1, the inspection system S includes an inspection support device 10 and an imaging device 100. The inspection system S is a system for inspecting an object 4 in real space. The object 4 is an example of an "object" in the technology of this disclosure.

[0026] As an example, Object 4 is a reinforced concrete bridge pier. Here, a bridge pier is given as an example of Object 4, but Object 4 may be other road facilities besides bridge piers. Examples of road facilities include road surfaces, tunnels, guardrails, traffic lights, and / or windbreak fences. Object 4 may also be social infrastructure other than road facilities (e.g., airport facilities, port facilities, water storage facilities, gas facilities, medical facilities, fire-fighting facilities, and / or educational facilities), or it may be privately owned property. Furthermore, Object 4 may be land (e.g., state-owned land and / or privately owned land). The bridge pier exemplified as Object 4 may be a bridge pier made of a material other than reinforced concrete. In the first embodiment, inspection refers to, for example, an inspection of the condition of Object 4. For example, the presence or absence of damage to Object 4 and / or the extent of damage are inspected by the inspection system S.

[0027] The inspection support device 10 is an example of an "image processing device" related to the technology of this disclosure. The inspection support device 10 is, for example, a desktop personal computer. Here, a desktop personal computer is given as an example of the inspection support device 10, but this is merely an example, and it may also be a notebook personal computer. Furthermore, it is not limited to a personal computer, but may also be a server. The server may be a mainframe used on-premises with the inspection support device 10, or it may be an external server realized by cloud computing. Alternatively, the server may be an external server realized by network computing such as fog computing, edge computing, or grid computing. The inspection support device 10 is communicated to the imaging device 100. The inspection support device 10 is used by the inspector 6. The inspection support device 10 may be used at the site where the object 4 is installed, or it may be used at a location other than the site where the object 4 is installed.

[0028] The imaging device 100 is, for example, a digital camera with interchangeable lenses. Here, a digital camera with interchangeable lenses is given as an example of the imaging device 100, but this is merely one example, and it may also be a digital camera built into various electronic devices such as smart devices or wearable terminals. Furthermore, the imaging device 100 may also be a pair of glasses-type eyewear terminal or a head-mounted display terminal worn on the head. The imaging device 100 is used by the imager 8.

[0029] As an example, as shown in Figure 2, the inspection support device 10 includes a computer 12, a reception device 14, a display 16, and a communication device 18.

[0030] Computer 12 is an example of a “computer” relating to the technology of this disclosure. Computer 12 comprises a processor 20, storage 22, and RAM 24. Processor 20 is an example of a “processor” relating to the technology of this disclosure. The processor 20, storage 22, RAM 24, receiving device 14, display 16, and communication device 18 are connected to a bus 26.

[0031] The processor 20, for example, has a CPU and controls the entire inspection support device 10. While this example shows the processor 20 having a CPU, this is merely one example. For instance, the processor 20 may have both a CPU and a GPU. In this case, for example, the GPU operates under the control of the CPU and is responsible for performing image processing.

[0032] Storage 22 is a non-volatile memory device that stores various programs and parameters. Examples of storage 22 include HDDs and SSDs. Note that HDDs and SSDs are merely examples, and flash memory, magnetoresistive memory, and / or ferroelectric memory may be used instead of, or in conjunction with, HDDs and / or SSDs.

[0033] RAM24 is memory that temporarily stores information and is used as work memory by the processor 20. Examples of RAM24 include DRAM and / or SRAM.

[0034] The reception device 14 has a keyboard, mouse, and touch panel, etc. (not shown in the illustration), and receives various instructions from the inspector 6. The display 16 has a screen 16A. Screen 16A is an example of a "screen" related to the technology of this disclosure. The display 16 displays various information (e.g., images and characters) on screen 16A under the control of the processor 20. Examples of the display 16 include an EL display (e.g., an organic EL display or an inorganic EL display). However, it is not limited to EL displays; other types of displays such as liquid crystal displays may also be used.

[0035] The communication device 18 is connected to the imaging device 100 in a communication-enabled manner. Here, the communication device 18 is connected to the imaging device 100 wirelessly using a predetermined wireless communication standard. Examples of predetermined wireless communication standards include Wi-Fi® or Bluetooth®. The communication device 18 is responsible for the exchange of information with the inspection support device 10. For example, the communication device 18 transmits information to the imaging device 100 in response to a request from the processor 20. The communication device 18 also receives information transmitted from the imaging device 100 and outputs the received information to the processor 20 via the bus 26. The communication device 18 may also be connected to the imaging device 100 via a wired connection.

[0036] As an example, as shown in Figure 3, the imaging device 100 includes a computer 102, an image sensor 104, a positioning unit 106, an acceleration sensor 108, an angular velocity sensor 110, and a communication device 112.

[0037] Computer 102 comprises a processor 114, storage 116, and RAM 118. The processor 114, storage 116, RAM 118, image sensor 104, positioning unit 106, acceleration sensor 108, angular velocity sensor 110, and communication device 112 are connected to bus 120. The processor 114, storage 116, and RAM 118 are implemented by hardware similar to, for example, the processor 20, storage 22, and RAM 24 provided in the inspection support device 10 described above.

[0038] The image sensor 104 is, for example, a CMOS image sensor. Although a CMOS image sensor is used as an example for the image sensor 104 here, the technology of this disclosure is not limited to this, and other image sensors may be used. The image sensor 104 captures an image of a subject (for example, object 4) and outputs the image data obtained by capturing the image.

[0039] The positioning unit 106 is a device that detects the position of the imaging device 100. The position of the imaging device 100 is detected, for example, using GNSS (e.g., GPS). The positioning unit 106 has a GNSS receiver (not shown). The GNSS receiver receives, for example, radio waves transmitted from multiple satellites. The positioning unit 106 detects the position of the imaging device 100 based on the radio waves received by the GNSS receiver and outputs positioning data (e.g., data indicating latitude, longitude, and altitude) corresponding to the detected position.

[0040] The acceleration sensor 108 detects the acceleration in the pitch axis, yaw axis, and roll axis of the imaging device 100. The acceleration sensor 108 outputs acceleration data corresponding to the acceleration in each axis of the imaging device 100. The angular velocity sensor 110 detects the angular velocity around the pitch axis, yaw axis, and roll axis of the imaging device 100. The angular velocity sensor 110 outputs angular velocity data corresponding to the angular velocity around each axis of the imaging device 100.

[0041] The processor 114 acquires the position of the imaging device 100 based on positioning data and / or acceleration data, and generates position data indicating the acquired position. The processor 114 also acquires the attitude of the imaging device 100 (i.e., the amount of change in attitude relative to a reference attitude defined in the relative coordinate system) based on angular velocity data, and generates attitude data indicating the acquired attitude. Hereinafter, the position of the imaging device 100 will be referred to as the "imaging position," and the attitude of the imaging device 100 will be referred to as the "imaging attitude."

[0042] Furthermore, if the processor 114 acquires the imaging position based solely on positioning data, the acceleration sensor 108 may be omitted. On the other hand, if the processor 114 acquires the imaging position based solely on acceleration data, the positioning unit 106 may be omitted. When the processor 114 acquires the imaging position based on positioning data, the imaging position in the absolute coordinate system is derived based on the positioning data. On the other hand, when the processor 114 acquires the imaging position based on acceleration data, the amount of change in the imaging position relative to a reference position defined in the relative coordinate system is derived based on the acceleration data.

[0043] The communication device 112 is connected to the inspection support device 10 in a communicative manner. The communication device 112 is implemented, for example, by hardware similar to the communication device 18 provided in the inspection support device 10 described above.

[0044] The imaging device 100 transmits image data, position data, and orientation data to the inspection support device 10. The image data is data representing a two-dimensional image 50 obtained when the object 4 is imaged by the imaging device 100. The position data is data indicating the imaging position when the imaging device 100 performs imaging, and is associated with the image data. Similarly, the orientation data is data indicating the imaging orientation when the imaging device 100 performs imaging, and is associated with the image data. In other words, the position data and orientation data are supplementary data attached to the image data.

[0045] For example, if only a plurality of two-dimensional images 50 obtained by imaging the object 4 from multiple imaging positions by the imaging device 100 are displayed on the screen 16A of the display 16 provided in the inspection support device 10, then the task of understanding the correspondence between each two-dimensional image 50 and the area of ​​the object 4 corresponding to each two-dimensional image 50, based on the plurality of two-dimensional images 50 displayed on the screen 16A, is time-consuming. Furthermore, the more frames the plurality of two-dimensional images 50 have, the more complicated the task of understanding the correspondence becomes. In view of these circumstances, in the first embodiment, the inspection support device 10 performs inspection support information generation processing and inspection support processing. The inspection support information generation processing and inspection support processing performed by the inspection support device 10 will be described in detail below.

[0046] As an example, as shown in Figure 4, the storage 22 of the inspection support device 10 stores an inspection support information generation program 30. The processor 20 of the inspection support device 10 reads the inspection support information generation program 30 from the storage 22 and executes the read inspection support information generation program 30 on the RAM 24. The processor 20 performs inspection support information generation processing to generate inspection support information 56 according to the inspection support information generation program 30 executed on the RAM 24.

[0047] The inspection support information generation process is realized when the processor 20 operates as an acquisition unit 32, a 3D image generation unit 34, and an inspection support information generation unit 36 ​​according to the inspection support information generation program 30.

[0048] As an example, as shown in Figure 5, multiple points P1 located in the circumferential direction of the object 4 indicate the imaging positions of the imaging device 100. The imager 8 moves around the object 4 and images the object 4 from multiple imaging positions in the circumferential direction using the imaging device 100. As an example, the imager 8 images different areas of the object 4 from each imaging position using the imaging device 100. By imaging different areas of the object 4 from each imaging position using the imaging device 100, the entire object 4, including multiple areas, is imaged.

[0049] The imaging position (i.e., point P1) corresponding to each two-dimensional image 50 obtained by imaging by the imaging device 100 corresponds to the starting point of the line of sight L directed at the object 4, and the imaging orientation corresponding to each two-dimensional image 50 corresponds to the direction of the line of sight L directed at the object 4. Point P2, where the object 4 and the line of sight L intersect, corresponds to the viewpoint when viewing the object 4 with the line of sight L. By imaging the object 4 from each imaging position using the imaging device 100, two-dimensional images 50 corresponding to each viewpoint are obtained. Each two-dimensional image 50 is an image corresponding to each region of the object 4.

[0050] In this example, the imager 8 moves around the object 4 and images the object 4 from each imaging position using the imaging device 100. However, the imaging device 100 may be mounted on a mobile body, and the object 4 may be imaged from each imaging position by the imaging device 100 when the mobile body is moving around the object 4. The mobile body may be, for example, a drone, gondola, trolley, aerial work platform, automated guided vehicle, or other vehicle.

[0051] The imaging device 100 associates image data representing the two-dimensional image 50 obtained by capturing images from each imaging position with position data indicating the imaging position when the image was captured and orientation data indicating the imaging orientation when the image was captured. The imaging device 100 then transmits each image data, along with the position data and orientation data associated with each image data, to the inspection support device 10.

[0052] As an example, as shown in Figure 6, the acquisition unit 32 acquires a two-dimensional image 50 based on the image data received by the inspection support device 10. The acquisition unit 32 also acquires the imaging position corresponding to each two-dimensional image 50 based on the position data received by the inspection support device 10. Furthermore, the acquisition unit 32 acquires the imaging posture corresponding to each two-dimensional image 50 based on the posture data received by the inspection support device 10.

[0053] The 3D image generation unit 34 generates a 3D image 52 representing the object 4 based on a plurality of 2D images 50 acquired by the acquisition unit 32. Image processing techniques used to generate the 3D image 52 based on the plurality of 2D images 50 include SfM, MVS, epipolar geometry, and stereo matching. The positions of multiple pixels included in the 3D image 52 are determined by multiple 3D coordinates obtained from the plurality of 2D images 50. The 3D image 52 is a 3D model defined by multiple 3D coordinates.

[0054] As an example, as shown in Figure 7, the 3D image 52 generated by the 3D image generation unit 34 has multiple parts 54 corresponding to each 2D image 50. Each part 54 is formed by a pixel group, which is a collection of pixels corresponding to each 2D image 50. The inspection support information generation unit 36 ​​generates inspection support information 56, which is information that associates each 2D image 50 acquired by the acquisition unit 32 with the imaging position corresponding to each 2D image 50, the imaging orientation corresponding to each 2D image 50, and the parts 54 corresponding to each 2D image 50. The inspection support information 56 is stored in the storage 22.

[0055] As an example, as shown in Figure 8, the storage 22 of the inspection support device 10 stores an inspection support program 40. The inspection support program 40 is an example of a "program" related to the technology of this disclosure. The processor 20 reads the inspection support program 40 from the storage 22 and executes the read inspection support program 40 on the RAM 24. The processor 20 performs inspection support processing to support the inspection by the inspector 6 (see Figure 1) according to the inspection support program 40 executed on the RAM 24.

[0056] The inspection support process is realized when the processor 20 operates as an operation mode setting unit 42, a first mode processing unit 44, a second mode processing unit 46, and a third mode processing unit 48, according to the inspection support program 40.

[0057] The inspection support device 10 has three operating modes: a first mode, a second mode, and a third mode. The operating mode setting unit 42 performs a mode setting process to selectively set the first mode, the second mode, and the third mode as the operating modes of the inspection support device 10.

[0058] In the mode setting process, if the operating mode of the inspection support device 10 is set to the first mode by the operating mode setting unit 42, the processor 20 operates as the first mode processing unit 44. The first mode processing unit 44 performs the first mode processing. The first mode processing is realized by the first mode processing unit 44 operating as the first display control unit 44A, the first image selection unit 44B, the first pixel extraction unit 44C, and the first image generation unit 44D.

[0059] In the mode setting process, if the operating mode of the inspection support device 10 is set to the second mode by the operating mode setting unit 42, the processor 20 operates as the second mode processing unit 46. The second mode processing unit 46 performs the second mode processing. The second mode processing is realized by the second mode processing unit 46 operating as the second display control unit 46A, the second image selection unit 46B, the second pixel extraction unit 46C, and the second image generation unit 46D.

[0060] In the mode setting process, if the operating mode of the inspection support device 10 is set to the third mode by the operating mode setting unit 42, the processor 20 operates as the third mode processing unit 48. The third mode processing unit 48 performs the third mode processing. The third mode processing is realized by the third mode processing unit 48 operating as the third display control unit 48A, the third image selection unit 48B, and the third image generation unit 48C.

[0061] As an example, as shown in Figure 9, the operation mode setting unit 42 sets the first mode as the operating mode of the inspection support device 10 by default. When the operation mode setting unit 42 sets the operating mode of the inspection support device 10 to the first mode, the first display control unit 44A displays the first image 61 on the screen 16A. Details of the first image 61 will be described later, but the first image 61 includes the second mode setting button 72 and the third mode setting button 73 as soft keys.

[0062] When the first image 61 is displayed on screen 16A and the receiving device 14 receives a setting instruction to press the second mode setting button 72, the receiving device 14 outputs a second mode setting instruction signal to the processor 20. Similarly, when the first image 61 is displayed on screen 16A and the receiving device 14 receives a setting instruction to press the third mode setting button 73, the receiving device 14 outputs a third mode setting instruction signal to the processor 20.

[0063] The operation mode setting unit 42 determines whether a second mode setting instruction signal or a third mode setting instruction signal has been input to the processor 20 when the operation mode of the inspection support device 10 is set to the first mode. If a second mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the second mode as the operation mode of the inspection support device 10. On the other hand, if a third mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the third mode as the operation mode of the inspection support device 10.

[0064] When the operating mode of the inspection support device 10 is set to the second mode by the operating mode setting unit 42, the second display control unit 46A displays the second image 62 on the screen 16A. Details of the second image 62 will be described later, but the second image 62 includes the first mode setting button 71 and the third mode setting button 73 as soft keys.

[0065] When the second image 62 is displayed on screen 16A and the receiving device 14 receives a setting instruction to press the first mode setting button 71, the receiving device 14 outputs a first mode setting instruction signal to the processor 20. Similarly, when the second image 62 is displayed on screen 16A and the receiving device 14 receives a setting instruction to press the third mode setting button 73, the receiving device 14 outputs a third mode setting instruction signal to the processor 20.

[0066] The operation mode setting unit 42 determines whether a first mode setting instruction signal or a third mode setting instruction signal has been input to the processor 20 when the operation mode of the inspection support device 10 is set to the second mode. If a first mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the first mode as the operation mode of the inspection support device 10. On the other hand, if a third mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the third mode as the operation mode of the inspection support device 10.

[0067] When the operating mode of the inspection support device 10 is set to the third mode by the operating mode setting unit 42, the third display control unit 48A displays the third image 63 on the screen 16A. Details of the third image 63 will be described later, but the third image 63 includes the first mode setting button 71 and the second mode setting button 72.

[0068] When the third image 63 is displayed on screen 16A and the receiving device 14 receives a setting instruction, which is an instruction to press the first mode setting button 71, the receiving device 14 outputs a first mode setting instruction signal to the processor 20. Similarly, when the third image 63 is displayed on screen 16A and the receiving device 14 receives a setting instruction, which is an instruction to press the third mode setting button 73, the receiving device 14 outputs a third mode setting instruction signal to the processor 20.

[0069] The operation mode setting unit 42 determines whether a first mode setting instruction signal or a second mode setting instruction signal has been input to the processor 20 when the operation mode of the inspection support device 10 is set to the third mode. If a first mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the first mode as the operation mode of the inspection support device 10. On the other hand, if a second mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the second mode as the operation mode of the inspection support device 10.

[0070] Hereafter, unless it is necessary to distinguish between the first mode setting instruction signal, the second mode setting instruction signal, and the third mode setting instruction signal, the first mode setting instruction signal, the second mode setting instruction signal, and the third mode setting instruction signal will be referred to as the "mode setting instruction signal."

[0071] The second mode among the multiple operating modes of the inspection support device 10 is an example of the "first operating mode" relating to the technology of this disclosure. The third mode among the multiple operating modes of the inspection support device 10 is an example of the "second operating mode" relating to the technology of this disclosure.

[0072] As an example, Figure 10 shows the state in which the first image 61 is displayed on screen 16A. The first image 61 includes a first image region 81 and a second image region 82. For example, the first image region 81 and the second image region 82 are displayed on screen 16A side by side in the left-right direction of the first image 61. The first image region 81 includes multiple two-dimensional images 50, and the second image region 82 includes a three-dimensional image 52.

[0073] The first display control unit 44A includes a plurality of two-dimensional images 50 in the first image area 81 based on the plurality of two-dimensional images 50 included in the inspection support information 56. The first display control unit 44A also includes a three-dimensional image 52 in the second image area 82 based on the three-dimensional image 52 included in the inspection support information 56.

[0074] The first image area 81 contains a predetermined number of two-dimensional images 50 from among a plurality of two-dimensional images 50. The predetermined number is set, for example, by the inspector 6 giving an instruction to the receiving device 14 (see Figure 9) to specify the predetermined number. Alternatively, for example, the inspector 6 gives an instruction to the receiving device 14 to scroll the first image area 81, causing the first image area 81 to be scrolled, thereby changing the two-dimensional images 50 contained in the first image area 81.

[0075] The second image region 82 contains the 3D image 52 in a 2D image state after rendering. For example, the size of the 3D image 52 is changed when the inspector 6 gives an instruction to the receiving device 14 (see Figure 9) to change the size of the 3D image 52. Also, for example, the 3D image 52 is rotated when the inspector 6 gives an instruction to the receiving device 14 to rotate the 3D image 52.

[0076] The first image region 81 containing multiple two-dimensional images 50 and the second image region 82 containing three-dimensional images 52 are displayed side by side on screen 16A, making it possible to compare the multiple two-dimensional images 50 and the three-dimensional images 52. In the example shown in Figure 10, the first image region 81 and the second image region 82 are displayed side by side on screen 16A in the left-right direction of the first image 61. However, for example, the first image region 81 and the second image region 82 may be displayed side by side on screen 16A in the vertical direction of the first image 61, or the first image region 81 and the second image region 82 may be displayed on screen 16A with the first image region 81 incorporated into a part of the second image region 82.

[0077] The two-dimensional image 50 is an example of a "two-dimensional image" relating to the technology of this disclosure. The three-dimensional image 52 is an example of a "three-dimensional image" relating to the technology of this disclosure. The first image region 81 is an example of a "first region" relating to the technology of this disclosure. The second image region 82 is an example of a "second region" relating to the technology of this disclosure.

[0078] As an example, as shown in Figure 11, when the first image 61 is displayed on the screen 16A, and the receiving device 14 receives a selection instruction, which is an instruction to select one of the multiple two-dimensional images 50 included in the first image region 81, the receiving device 14 outputs a selection instruction signal to the processor 20 indicating the selection instruction. The selection instruction is an example of a "selection instruction" related to the technology of this disclosure.

[0079] When a selection instruction signal is input to the processor 20, the first image selection unit 44B selects a two-dimensional image 50 corresponding to the selection instruction (hereinafter referred to as "attention two-dimensional image 50A") from among a plurality of two-dimensional images 50 included in the inspection support information 56, in accordance with the selection instruction indicated by the selection instruction signal. The attention two-dimensional image 50A is an example of the "attention two-dimensional image" related to the technology of this disclosure.

[0080] As an example, as shown in Figure 12, the first pixel extraction unit 44C acquires the imaging position and orientation corresponding to the two-dimensional image 50A of interest from the inspection support information 56. The first pixel extraction unit 44C also derives a viewpoint corresponding to the two-dimensional image 50A of interest based on the acquired imaging position and orientation. The first pixel extraction unit 44C then extracts pixels from the three-dimensional image 52 contained in the inspection support information 56 to include the three-dimensional image 52 in the second image region 82 at the derived viewpoint. Furthermore, when extracting pixels from the three-dimensional image 52, the first pixel extraction unit 44C extracts pixels from the three-dimensional image 52 to include the entire three-dimensional image 52 in the second image region 82.

[0081] The first image generation unit 44D generates a first image region 81 that includes a predetermined number of two-dimensional images 50 from among the multiple two-dimensional images 50 included in the inspection support information 56, and in which the two-dimensional image of interest 50A is enclosed by a frame 90. The first image generation unit 44D also generates a second image region 82 that includes a three-dimensional image 52 that is large enough to fit entirely within the second image region 82 from a viewpoint corresponding to the two-dimensional image of interest 50A, based on the pixels extracted by the first pixel extraction unit 44C. For example, the three-dimensional image 52 is included in the second image region 82 such that the viewpoint corresponding to the two-dimensional image of interest 50A is located at the center 82C of the second image region 82.

[0082] As an example, as shown in Figure 13, the first image generation unit 44D generates the first image 61 by combining the generated first image region 81 and second image region 82.

[0083] The first display control unit 44A outputs first image data showing the first image 61 generated by the first image generation unit 44D to the display 16. As a result, the first image 61 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of two-dimensional images 50 out of a plurality of two-dimensional images 50 are included in the first image area 81 and displayed on the screen 16A, and the two-dimensional image of interest 50A is displayed on the screen 16A surrounded by a frame 90. By displaying the two-dimensional image of interest 50A surrounded by a frame 90 on the screen 16A, the two-dimensional image of interest 50A becomes distinguishable from the remaining two-dimensional images 50 out of the plurality of two-dimensional images 50.

[0084] Furthermore, the 3D image 52 is displayed on screen 16A from a viewpoint corresponding to the 2D image 50A of interest, and in a size that fits entirely within the second image region 82. By displaying the 3D image 52 on screen 16A from a viewpoint corresponding to the 2D image 50A of interest, the portion 54 of the 3D image 52 corresponding to the 2D image 50A of interest (hereinafter referred to as "portion 54A of interest") becomes visually identifiable. "Portion 54" is an example of a "portion" relating to the technology of this disclosure, and the portion 54A of interest of the 3D image 52 is an example of a "portion of interest" relating to the technology of this disclosure.

[0085] In the example shown in Figure 13, the two-dimensional image of interest 50A is displayed on screen 16A surrounded by a frame 90. However, the two-dimensional image of interest 50A may also be displayed on screen 16A in a manner that makes it distinguishable from the remaining two-dimensional images 50 by other means. For example, the two-dimensional image of interest 50A may be displayed on screen 16A in a manner in which it is represented in a different color from the remaining two-dimensional images 50, in a manner in which it has a pattern, or in a manner in which it has a higher brightness than the remaining two-dimensional images 50. Even in such examples, the two-dimensional image of interest 50A will be distinguishable from the remaining two-dimensional images 50.

[0086] As an example, Figure 14 shows the state in which the second image 62 is displayed on screen 16A. The second image 62 includes a first image region 81 and a third image region 83. As an example, the first image region 81 and the third image region 83 are displayed on screen 16A side by side in the left-right direction of the second image 62. The first image region 81 is the same as the first image region 81 of the first image 61 (see Figure 10). The third image region 83 includes a three-dimensional image 52. The third image region 83 is an example of the "second region" relating to the technology of this disclosure.

[0087] The second display control unit 46A includes a plurality of two-dimensional images 50 in the first image area 81 based on the plurality of two-dimensional images 50 included in the inspection support information 56. The second display control unit 46A also includes a three-dimensional image 52 in the third image area 83 based on the three-dimensional image 52 included in the inspection support information 56.

[0088] The third image region 83 contains the 3D image 52 in a 2D image state after rendering. For example, the size of the 3D image 52 is changed when the inspector 6 gives an instruction to the receiving device 14 (see Figure 9) to change the size of the 3D image 52. Also, for example, the 3D image 52 is rotated when the inspector 6 gives an instruction to the receiving device 14 to rotate the 3D image 52.

[0089] The first image region 81, which contains multiple 2D images 50, and the third image region 83, which contains a 3D image 52, are displayed side by side on screen 16A, making it possible to compare the multiple 2D images 50 and the 3D image 52.

[0090] In Figure 14, an example is shown in which the first image region 81 and the third image region 83 are displayed on screen 16A with the second image 62 aligned horizontally. However, for example, the first image region 81 and the third image region 83 may be displayed on screen 16A with the second image 62 aligned vertically, or the first image region 81 and the third image region 83 may be displayed on screen 16A with the first image region 81 incorporated into a part of the third image region 83.

[0091] As an example, as shown in Figure 15, when the second image 62 is displayed on the screen 16A, and the receiving device 14 receives a selection instruction, which is an instruction to select one of the multiple two-dimensional images 50 included in the first image region 81, the receiving device 14 outputs a selection instruction signal to the processor 20 indicating the selection instruction.

[0092] When a selection instruction signal is input to the processor 20, the second image selection unit 46B selects a two-dimensional image 50A corresponding to the selection instruction from among the multiple two-dimensional images 50 included in the inspection support information 56, according to the selection instruction indicated by the selection instruction signal.

[0093] As an example, as shown in Figure 16, the second pixel extraction unit 46C extracts a portion of interest 54A that is associated with the two-dimensional image of interest 50A from the three-dimensional image 52 included in the inspection support information 56.

[0094] The second image generation unit 46D generates a first image region 81 that includes a predetermined number of two-dimensional images 50 from among the multiple two-dimensional images 50 included in the inspection support information 56, and in which the two-dimensional image of interest 50A is enclosed by a frame 90. The second image generation unit 46D also generates a third image region 83 that includes the part of interest 54A of the three-dimensional image 52, based on the part of interest 54A extracted by the second pixel extraction unit 46C.

[0095] As an example, as shown in Figure 17, the second image generation unit 46D generates a second image 62 by combining the generated first image region 81 and third image region 83.

[0096] The second display control unit 46A outputs second image data showing the second image 62 generated by the second image generation unit 46D to the display 16. As a result, the second image 62 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of two-dimensional images 50 from among the multiple two-dimensional images 50 are included in the first image area 81 and displayed on the screen 16A, and the two-dimensional image of interest 50A is displayed on the screen 16A surrounded by a frame 90.

[0097] Furthermore, the portion 54A of interest in the 3D image 52 is displayed on screen 16A in an enlarged state. By displaying the portion 54A of interest in the 3D image 52 on screen 16A in an enlarged state, the portion 54A of interest in the 3D image 52 becomes visually identifiable. In other words, by displaying the portion 54A of interest in the 3D image 52 on screen 16A in an enlarged state, the portion 54A of interest becomes distinguishable from the remaining portion 54 of the multiple portions 54 that make up the 3D image 52. As a result, the portion 54A of interest in the 3D image 52 becomes visually identifiable.

[0098] Furthermore, by displaying the 2D image of interest 50A within a frame 90 on screen 16A, the 2D image of interest 50A becomes distinguishable from the other 2D images 50 among the multiple 2D images 50. This makes it possible to visually identify the correspondence between the 2D image of interest 50A and the 3D image 52, specifically the portion of interest 54A.

[0099] In the example shown in Figure 17, the portion of interest 54A in the 3D image 52 is displayed on screen 16A in an enlarged state. However, the entire 3D image 52 may be displayed on screen 16A at a size that fits within the third image region 83, and the portion of interest 54A in the 3D image 52 may be displayed on screen 16A in a manner that makes it distinguishable from the rest of the portion 54 by other means. For example, the portion of interest 54A may be represented in a different color from the rest of the portion 54, the portion of interest 54A may have a pattern applied to it, the portion of interest 54A may be surrounded by a frame, or the pixels forming the outline of the portion of interest 54A may have higher brightness than the surrounding pixels. Even in such examples, the portion of interest 54A in the 3D image 52 can be visually identified.

[0100] As an example, Figure 18 shows the state in which the third image 63 is displayed on screen 16A. The third image 63 includes a first image region 81 and a fourth image region 84. As an example, the first image region 81 and the fourth image region 84 are displayed on screen 16A side by side with respect to the third image 63. The first image region 81 is the same as the first image region 81 of the first image 61 (see Figure 10). The fourth image region 84 includes a three-dimensional image 52 and multiple location-identifying images 92. Each location-identifying image 92 is an image used to identify multiple imaging locations where imaging was performed to obtain multiple two-dimensional images 50, and indicates the imaging location corresponding to each two-dimensional image 50.

[0101] The third display control unit 48A includes a plurality of two-dimensional images 50 in the first image region 81 based on the plurality of two-dimensional images 50 included in the inspection support information 56. The third display control unit 48A also includes a three-dimensional image 52 in the fourth image region 84 based on the three-dimensional image 52 included in the inspection support information 56.

[0102] The fourth image region 84 contains the 3D image 52 in a 2D image state after rendering. For example, the size of the 3D image 52 is changed when the inspector 6 gives an instruction to the receiving device 14 (see Figure 9) to change the size of the 3D image 52. Also, for example, the 3D image 52 is rotated when the inspector 6 gives an instruction to the receiving device 14 to rotate the 3D image 52.

[0103] Furthermore, the third display control unit 48A includes a plurality of position-identifying images 92 in the fourth image region 84 based on each imaging position included in the inspection support information 56. Each position-identifying image 92 is represented, for example, in a plate shape. The plurality of position-identifying images 92 are included in the fourth image region 84 in a manner that allows them to be compared with the three-dimensional image 52. Specifically, the plurality of position-identifying images 92 are included in the fourth image region 84 in a manner that faces the three-dimensional image 52 by being arranged around the three-dimensional image 52. In other words, the fourth image region 84 includes an image that shows the plurality of position-identifying images 92 and the three-dimensional image 52 facing each other.

[0104] When the first image region 81 and the fourth image region 84 are displayed side by side on screen 16A, multiple two-dimensional images 50 and three-dimensional images 52 can be compared, and multiple two-dimensional images 50 and multiple location-specific images 92 can be compared.

[0105] In Figure 18, an example is shown in which the first image region 81 and the fourth image region 84 are displayed on screen 16A with the third image 63 aligned horizontally. However, for example, the first image region 81 and the fourth image region 84 may be displayed on screen 16A with the third image 63 aligned vertically, or the first image region 81 and the fourth image region 84 may be displayed on screen 16A with the first image region 81 incorporated into a part of the fourth image region 84.

[0106] The first image region 81 is an example of the "first region" and "third region" relating to the technology of this disclosure. The fourth image region 84 is an example of the "second region" and "fourth region" relating to the technology of this disclosure. The location identification image 92 is an example of the "location identification image" relating to the technology of this disclosure.

[0107] As an example, as shown in Figure 19, when the third image 63 is displayed on screen 16A, and the receiving device 14 receives a selection instruction, which is an instruction to select one of the multiple location identification images 92 contained in the fourth image region 84, the receiving device 14 outputs a selection instruction signal to the processor 20 indicating the selection instruction. Hereinafter, the selected location identification image 92 from the multiple location identification images 92 will be referred to as the "location identification image of interest 92A".

[0108] When a selection instruction signal is input to the processor 20, the third image selection unit 48B selects an imaging position corresponding to the focus position identification image 92A (hereinafter referred to as the "focus imaging position") from a plurality of imaging positions included in the inspection support information 56, in accordance with the selection instruction indicated by the selection instruction signal. The third image selection unit 48B then selects a focus 2D image 50A corresponding to the focus imaging position from a plurality of 2D images 50 included in the inspection support information 56. The focus 2D image 50A is a 2D image 50 obtained by imaging from the focus imaging position. The focus position identification image 92A is an example of a "focus position identification image" related to the technology of this disclosure. The focus imaging position is an example of a "focus imaging position" related to the technology of this disclosure.

[0109] As an example, as shown in Figure 20, the third image generation unit 48C generates a fourth image region 84 that includes the three-dimensional image 52, based on the three-dimensional image 52 included in the inspection support information 56. The third image generation unit 48C also includes a plurality of position identification images 92 in the fourth image region 84 based on the imaging position and imaging orientation included in the inspection support information 56. The plurality of position identification images 92 are included in the fourth image region 84 in a state facing the three-dimensional image 52 by being arranged around the three-dimensional image 52. Each position identification image 92 is arranged at a position corresponding to each imaging position and in an orientation corresponding to each imaging orientation.

[0110] Furthermore, the third image generation unit 48C includes the focus location identification image 92A, which corresponds to the target imaging location among the multiple location identification images 92, in the fourth image region 84. The focus location identification image 92A is included in the fourth image region 84 in a state that makes it distinguishable from the remaining location identification images 92. In the example shown in Figure 20, as an example of a state in which the focus location identification image 92A is distinguishable from the remaining location identification images 92, the focus location identification image 92A is represented in a different color from the remaining location identification images 92. The focus location identification image 92A may be surrounded by a frame, or a pattern may be applied to the focus location identification image 92A. In addition, the pixels that form the outline of the focus location identification image 92A may have higher brightness than the surrounding pixels.

[0111] Furthermore, the third image generation unit 48C generates a first image region 81 that includes a predetermined number of two-dimensional images 50 from among the multiple two-dimensional images 50 included in the inspection support information 56, and in which the two-dimensional image of interest 50A is surrounded by a frame 90.

[0112] As an example, as shown in Figure 21, the third image generation unit 48C generates a third image 63 by combining the generated first image region 81 and fourth image region 84.

[0113] The third display control unit 48A outputs third image data showing the third image 63 generated by the third image generation unit 48C to the display 16. As a result, the third image 63 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of two-dimensional images 50 from among the multiple two-dimensional images 50 are displayed on the screen 16A in a state where they are included in the first image area 81, and the two-dimensional image of interest 50A is displayed on the screen 16A surrounded by a frame 90. In addition, the three-dimensional image 52 is displayed on the screen 16A.

[0114] Furthermore, the multiple location-identifying images 92 are arranged around the 3D image 52 and displayed on the screen 16A facing the 3D image 52, and the location-identifying image of interest 92A is displayed on the screen 16A in a state that is distinguishable from the other location-identifying images 92. Because the location-identifying image of interest 92A is displayed on the screen 16A in a state that is distinguishable from the other location-identifying images 92, the area of ​​interest 54A in the 3D image 52 that corresponds to the location-identifying image of interest 92A becomes visually identifiable. In other words, the imaging position and imaging orientation are identified by the location-identifying image of interest 92A, and the correspondence between the identified imaging position and orientation and the area of ​​interest 54A in the 3D image 52 becomes visually identifiable. As a result, the area of ​​interest 54A in the 3D image 52 becomes visually identifiable.

[0115] Furthermore, by displaying the 2D image of interest 50A within a frame 90 on screen 16A, the 2D image of interest 50A becomes distinguishable from the other 2D images 50 among the multiple 2D images 50. This makes it possible to visually identify the correspondence between the 2D image of interest 50A and the 3D image 52, specifically the portion of interest 54A.

[0116] In the example shown in Figure 21, the entire 3D image 52 is displayed on screen 16A in a size that fits within the fourth image region 84. However, the portion of interest 54A of the 3D image 52 may also be displayed on screen 16A in an enlarged state. Even in such an example, the portion of interest 54A of the 3D image 52 becomes visually identifiable.

[0117] Next, the operation of the inspection support device 10 according to the first embodiment will be explained with reference to Figures 22 to 26.

[0118] First, with reference to Figure 22, we will explain an example of the flow of the inspection support information generation process performed by the processor 20 of the inspection support device 10.

[0119] In the inspection support information generation process shown in Figure 22, first, in step ST10, the acquisition unit 32 (see Figure 6) acquires a two-dimensional image 50 based on the image data received by the inspection support device 10. The acquisition unit 32 also acquires the imaging position corresponding to each two-dimensional image 50 based on the position data received by the inspection support device 10. Furthermore, the acquisition unit 32 acquires the imaging orientation corresponding to each two-dimensional image 50 based on the orientation data received by the inspection support device 10. After the processing in step ST10 is executed, the inspection support information generation process proceeds to step ST12.

[0120] In step ST12, the 3D image generation unit 34 (see Figure 6) generates a 3D image 52 representing the object 4 based on the multiple 2D images 50 acquired in step ST10. After the processing in step ST12 is completed, the inspection support information generation process proceeds to step ST14.

[0121] In step ST14, the inspection support information generation unit 36 ​​(see Figure 7) generates inspection support information 56, which is information that associates each 2D image 50 acquired in step ST10 with the imaging position corresponding to each 2D image 50, the imaging orientation corresponding to each 2D image 50, and the portion 54 corresponding to each 2D image 50. After the processing in step ST14 is executed, the inspection support information generation process ends.

[0122] Next, with reference to Figures 23 to 26, we will describe an example of the flow of inspection support processing performed by the processor 20 of the inspection support device 10. First, with reference to Figure 23, we will describe an example of the flow of the mode setting process, which is part of the inspection support processing.

[0123] In the mode setting process shown in Figure 23, first, in step ST20, the operation mode setting unit 42 (see Figure 9) determines whether or not a mode setting instruction signal has been input to the processor 20. If a mode setting instruction signal has been input to the processor 20 in step ST20, the determination is affirmed, and the inspection support process proceeds to step ST22. If a mode setting instruction signal has not been input to the processor 20 in step ST20, the determination is denied, and the mode setting process proceeds to step ST32.

[0124] In step ST22, the operation mode setting unit 42 determines that the mode setting instruction signal input to the processor 20 in step ST20 corresponds to the first mode setting. instructions Determine whether it is a signal or not. In step ST22, if the mode setting instruction signal is present, the first mode setting instructions If it is a signal, the determination is affirmed and the inspection support process proceeds to step ST24. In step ST22, the mode setting instruction signal is the first mode setting. instructions If no signal is detected, the determination is rejected, and the mode setting process proceeds to step ST26.

[0125] In step ST24, the operation mode setting unit 42 sets the first mode as the operation mode of the inspection support device 10. As a result, the first mode processing is executed. After the processing in step ST24 is completed, the mode setting process moves on to step ST32.

[0126] In step ST26, the operation mode setting unit 42 determines whether the mode setting instruction signal input to the processor 20 in step ST20 is a second mode setting signal. If the mode setting instruction signal in step ST26 is a second mode setting signal, the determination is affirmed, and the inspection support process proceeds to step ST28. If the mode setting instruction signal in step ST26 is not a second mode setting signal, the determination is denied, and the mode setting process proceeds to step ST30.

[0127] In step ST28, the operation mode setting unit 42 sets the second mode as the operation mode of the inspection support device 10. This executes the second mode processing. After the processing in step ST28 is completed, the mode setting process moves to step ST32.

[0128] In step ST30, the operation mode setting unit 42 sets the third mode as the operation mode of the inspection support device 10. As a result, the third mode processing is executed. After the processing in step ST30 is completed, the mode setting process moves on to step ST32.

[0129] In step ST32, the processor 20 determines whether the conditions for terminating the mode setting process (hereinafter referred to as "mode setting process termination conditions") have been met. An example of a mode setting process termination condition is that a termination instruction signal from the receiving device 14 has been received by the receiving device 14 and input to the processor 20. If the mode setting process termination conditions are not met in step ST32, the determination is denied and the mode setting process proceeds to step ST20. If the mode setting process termination conditions are met in step ST32, the determination is affirmed and the inspection support process, including the mode setting process, is terminated.

[0130] Next, with reference to Figure 24, we will explain an example of the flow of the first mode processing among the inspection support processes.

[0131] In the first mode processing shown in Figure 24, first, in step ST40, the first display control unit 44A (see Figure 10) displays the first image 61 on the screen 16A. After the processing in step ST40 is completed, the first mode processing proceeds to step ST42.

[0132] In step ST42, the first image selection unit 44B (see Figure 11) determines whether a selection instruction signal, which is an instruction to select one of the multiple two-dimensional images 50, has been input to the processor 20. If a selection instruction signal indicating a selection instruction has been input to the processor 20 in step ST42, the determination is affirmed, and the first mode processing proceeds to step ST44. If a selection instruction signal indicating a selection instruction has not been input to the processor 20 in step ST42, the determination is denied, and the first mode processing proceeds to step ST52.

[0133] In step ST44, the first image selection unit 44B selects a two-dimensional image 50A of interest from among a plurality of two-dimensional images 50 included in the inspection support information 56, corresponding to the selection instruction indicated by the selection instruction signal. After the processing in step ST44 is completed, the first mode processing proceeds to step ST46.

[0134] In step ST46, the first pixel extraction unit 44C (see Figure 12) obtains the imaging position and orientation corresponding to the two-dimensional image 50A of interest selected in step ST44 from the inspection support information 56. The first pixel extraction unit 44C also derives a viewpoint corresponding to the two-dimensional image 50A of interest based on the acquired imaging position and orientation. Then, the first pixel extraction unit 44C extracts pixels from the three-dimensional image 52 contained in the inspection support information 56 to include the three-dimensional image 52 in the second image region 82 at the derived viewpoint. After the processing in step ST46 is executed, the first mode processing proceeds to step ST48.

[0135] In step ST48, the first image generation unit 44D (see Figure 12) generates a first image region 81 that includes a predetermined number of two-dimensional images 50 from among the multiple two-dimensional images 50 included in the inspection support information 56, and in which the two-dimensional image of interest 50A is surrounded by a frame 90. The first image generation unit 44D also generates a second image region 82 that includes a three-dimensional image 52 that is large enough to fit entirely within the second image region 82 from a viewpoint corresponding to the two-dimensional image of interest 50A, based on the pixels extracted in step ST46. Then, the first image generation unit 44D (see Figure 13) generates a first image 61 by combining the generated first image region 81 and second image region 82. After the processing in step ST48 is executed, the first mode processing proceeds to step ST50.

[0136] In step ST50, the first display control unit 44A (see Figure 13) outputs first image data showing the first image 61 generated in step ST48 to the display 16. As a result, the first image 61 is displayed on screen 16A of the display 16. After the processing in step ST50 is completed, the first mode processing proceeds to step ST52.

[0137] In step ST52, the processor 20 determines whether the conditions for the termination of the first mode processing (hereinafter referred to as the "first mode processing termination conditions") have been met. Examples of first mode processing termination conditions include the condition that a termination instruction from the inspector 6 is received by the receiving device 14 and a termination instruction signal from the receiving device 14 is input to the processor 20, or the condition that a mode setting instruction signal indicating an instruction to set an operating mode different from the first mode is input to the processor 20. If the first mode processing termination conditions are not met in step ST52, the determination is denied and the first mode processing proceeds to step ST42. If the first mode processing termination conditions are met in step ST52, the determination is affirmed and the first mode processing terminates.

[0138] Next, with reference to Figure 25, we will explain an example of the flow of the second mode processing among the inspection support processes.

[0139] In the second mode processing shown in Figure 25, first, in step ST60, the second display control unit 46A (see Figure 14) displays the second image 62 on the screen 16A. After the processing in step ST60 is completed, the second mode processing proceeds to step ST62.

[0140] In step ST62, the second image selection unit 46B (see Figure 15) determines whether a selection instruction signal, which is an instruction to select one of the multiple two-dimensional images 50, has been input to the processor 20. If a selection instruction signal indicating a selection instruction has been input to the processor 20 in step ST62, the determination is affirmed, and the second mode processing proceeds to step ST64. If a selection instruction signal indicating a selection instruction has not been input to the processor 20 in step ST62, the determination is denied, and the second mode processing proceeds to step ST72.

[0141] In step ST64, the second image selection unit 46B selects a two-dimensional image 50A of interest from among a plurality of two-dimensional images 50 included in the inspection support information 56, corresponding to the selection instruction indicated by the selection instruction signal. After the processing in step ST64 is completed, the second mode processing proceeds to step ST66.

[0142] In step ST66, the second pixel extraction unit 46C (see Figure 16) extracts the area of ​​interest 54A, which is associated with the two-dimensional image of interest 50A, from the three-dimensional image 52 included in the inspection support information 56. After the processing in step ST66 is completed, the second mode processing proceeds to step ST68.

[0143] In step ST68, the second image generation unit 46D (see Figure 16) generates a first image region 81 that includes a predetermined number of two-dimensional images 50 from among the multiple two-dimensional images 50 included in the inspection support information 56, and in which the two-dimensional image of interest 50A is enclosed by a frame 90. The second image generation unit 46D also generates a third image region 83 that includes the part of interest 54A of the three-dimensional image 52, based on the part of interest 54A extracted in step ST66. Then, the second image generation unit 46D (see Figure 17) generates a second image 62 by combining the generated first image region 81 and third image region 83. After the processing in step ST68 is executed, the second mode processing proceeds to step ST70.

[0144] In step ST70, the second display control unit 46A (see Figure 17) outputs second image data showing the second image 62 generated in step ST68 to the display 16. As a result, the second image 62 is displayed on screen 16A of the display 16. After the processing in step ST70 is completed, the second mode processing proceeds to step ST72.

[0145] In step ST72, the processor 20 determines whether the conditions for the termination of the second mode processing (hereinafter referred to as the "second mode processing termination conditions") have been met. Examples of second mode processing termination conditions include the condition that a termination instruction from the inspector 6 is received by the receiving device 14 and a termination instruction signal from the receiving device 14 is input to the processor 20, or the condition that a mode setting instruction signal indicating an instruction to set an operating mode different from the second mode is input to the processor 20. If the second mode processing termination conditions are not met in step ST72, the determination is denied and the second mode processing proceeds to step ST62. If the second mode processing termination conditions are met in step ST72, the determination is affirmed and the second mode processing terminates.

[0146] Next, with reference to Figure 26, we will explain an example of the flow of the third mode processing among the inspection support processes.

[0147] In the third mode processing shown in Figure 26, first, in step ST80, the third display control unit 48A (see Figure 18) displays the third image 63 on the screen 16A. After the processing in step ST80 is completed, the third mode processing proceeds to step ST82.

[0148] In step ST82, the third image selection unit 48B (see Figure 19) determines whether a selection instruction signal, which is an instruction to select the target location identification image 92A from among the multiple location identification images 92, has been input to the processor 20. If a selection instruction signal indicating a selection instruction has been input to the processor 20 in step ST82, the determination is affirmed, and the third mode processing proceeds to step ST84. If a selection instruction signal indicating a selection instruction has not been input to the processor 20 in step ST82, the determination is denied, and the third mode processing proceeds to step ST90.

[0149] In step ST84, the third image selection unit 48B selects a target imaging position corresponding to the target position identification image 92A from a plurality of imaging positions included in the inspection support information 56, according to the selection instruction indicated by the selection instruction signal. Then, the third image selection unit 48B selects a target two-dimensional image 50A corresponding to the target imaging position from a plurality of two-dimensional images 50 included in the inspection support information 56. After the processing in step ST84 is executed, the third mode processing proceeds to step ST86.

[0150] In step ST86, the third image generation unit 48C (see Figure 20) generates a fourth image region 84 containing the three-dimensional image 52 based on the three-dimensional image 52 included in the inspection support information 56. The third image generation unit 48C also includes a plurality of position identification images 92 in the fourth image region 84 based on the imaging position and imaging orientation included in the inspection support information 56. Furthermore, the third image generation unit 48C includes a point-of-interest position identification image 92A corresponding to the point-of-interest imaging position selected in step ST84 in the fourth image region 84. Then, the third image generation unit 48C (see Figure 21) generates a third image 63 by combining the generated first image region 81 and fourth image region 84. After the processing in step ST86 is executed, the third mode processing proceeds to step ST88.

[0151] In step ST88, the third display control unit 48A (see Figure 21) outputs third image data showing the third image 63 generated in step ST86 to the display 16. As a result, the third image 63 is displayed on screen 16A of the display 16. After the processing in step ST88 is completed, the third mode processing proceeds to step ST90.

[0152] In step ST90, the processor 20 determines whether the conditions for the termination of the third mode processing (hereinafter referred to as the "third mode processing termination conditions") have been met. Examples of third mode processing termination conditions include the condition that a termination instruction from the inspector 6 is received by the receiving device 14 and a termination instruction signal from the receiving device 14 is input to the processor 20, or the condition that a mode setting instruction signal indicating an instruction to set an operating mode different from the third mode is input to the processor 20. If the third mode processing termination conditions are not met in step ST90, the determination is denied and the third mode processing proceeds to step ST82. If the third mode processing termination conditions are met in step ST90, the determination is affirmed and the third mode processing is terminated. The inspection support method described above as the operation of the inspection support device 10 is an example of the "image processing method" related to the technology of this disclosure.

[0153] As described above, in the inspection support device 10 according to the first embodiment, the processor 20 displays on the screen 16A a plurality of two-dimensional images 50 used to generate a three-dimensional image 52 representing an object 4 in real space, and which are associated with a plurality of parts 54 of the three-dimensional image 52, in a state that allows comparison between the two-dimensional images 50 and the three-dimensional image 52 (see Figures 10, 14, and 18). The processor 20 also selects a two-dimensional image of interest 50A from the plurality of two-dimensional images 50 according to a given selection instruction (see Figures 11, 15, and 19). The processor 20 then displays on the screen 16A a state that allows visual identification of the part of interest 54A that corresponds to the two-dimensional image of interest 50A among the plurality of parts 54 (see Figures 13, 17, and 21). Therefore, the correspondence between each two-dimensional image 50 and the region of the object 4 corresponding to each two-dimensional image 50 can be visually grasped.

[0154] Furthermore, a state in which multiple 2D images 50 and 3D images 52 can be compared is a state in which a first image region 81 containing multiple 2D images 50 and an image region containing the 3D images 52 (i.e., a second image region 82, a third image region 83, or a fourth image region 84) are arranged side by side (see Figures 10, 14, and 18). Therefore, multiple 2D images 50 and 3D images 52 can be visually compared.

[0155] Furthermore, the state in which the portion of interest 54A can be visually identified includes the state in which the portion of interest 54A can be distinguished from the remaining portions 54 of the multiple portions 54 (see Figures 17 and 21). Therefore, for example, the visibility of the portion of interest 54A can be improved compared to the case in which the portion of interest 54A cannot be distinguished from the remaining portions 54.

[0156] Furthermore, the state in which the area of ​​interest 54A can be visually identified includes the state in which the 2D image of interest 50A can be distinguished from the remaining 2D images 50 among the multiple 2D images 50 (see Figures 13, 17, and 21). Therefore, for example, the visibility of the area of ​​interest 54A can be improved compared to the case in which the 2D image of interest 50A cannot be distinguished from the remaining 2D images 50. Also, when the multiple 2D images 50 and the 3D image 52 are displayed on the screen 16A in a state in which the area of ​​interest 54A can be compared and distinguished from the remaining areas 54 (see Figures 17 and 21), the correspondence between the 2D image of interest 50A and the area of ​​interest 54A can be visually identified.

[0157] Furthermore, the processor 20 displays multiple location identification images 92, which can identify multiple imaging locations where imaging was performed to obtain multiple two-dimensional images 50, on the screen 16A in a state that can be compared with the three-dimensional image 52 (see Figure 18). The processor 20 also selects an imaging location corresponding to the selected location identification image 92A from the multiple location identification images 92 as the location identification location (see Figure 19) according to the selection instruction. Then, the processor 20 selects the two-dimensional image 50 obtained by imaging from the location identification location as the location identification image 50A from the multiple two-dimensional images 50 (see Figure 19). Therefore, by selecting the location identification image 92A from the multiple location identification images 92, the location identification image 50A can be selected from the multiple two-dimensional images 50.

[0158] Furthermore, a state in which multiple location-identified images 92 and a 3D image 52 can be compared includes a state in which multiple location-identified images 92 and a 3D image 52 are facing each other (see Figure 19). Therefore, based on the state in which multiple location-identified images 92 and a 3D image 52 are facing each other, a location-identified image 92A of interest corresponding to the portion of interest 54A in the 3D image 52 can be selected from the multiple location-identified images 92.

[0159] Furthermore, a state in which multiple two-dimensional images 50 and three-dimensional images 52 can be compared is a state in which a first image region 81 containing multiple two-dimensional images 50 and a fourth image region 84 (i.e., an image region containing an image showing a configuration in which multiple location-identifying images 92 and three-dimensional images 52 are placed opposite each other) are arranged side by side (see Figures 18 and 19). Therefore, multiple two-dimensional images 50, multiple parts 54 of the three-dimensional image 52, and multiple location-identifying images 92 can be visually compared.

[0160] Furthermore, the state in which the area of ​​interest 54A can be visually identified includes the state in which the area of ​​interest location identification image 92A can be distinguished from the other location identification images 92 among the multiple location identification images 92. Therefore, for example, the visibility of the area of ​​interest 54A can be improved compared to the case in which the area of ​​interest location identification image 92A cannot be distinguished from the other location identification images 92.

[0161] Furthermore, the inspection support device 10 (see Figure 9) has an operating mode (for example, a first mode and a second mode) that displays a plurality of two-dimensional images 50 and a three-dimensional image 52 on the screen 16A in a manner that allows for comparison, and an operating mode (i.e., a third mode) that displays a plurality of location-identifying images 92 on the screen 16A in a manner that allows for comparison with the three-dimensional image 52. The processor 20 sets the operating mode according to a given setting instruction. Therefore, for example, depending on the area of ​​interest 54A, the screen 16A can be selectively switched between a state in which a plurality of two-dimensional images 50 and a three-dimensional image 52 can be compared, and a state in which a plurality of location-identifying images 92 can be compared with the three-dimensional image 52.

[0162] Furthermore, in the first mode, the 3D image 52 is displayed on screen 16A from a viewpoint corresponding to the 2D image 50A of interest (see Figure 13). Therefore, based on the viewpoint corresponding to the 2D image 50A of interest, the portion 54A of interest that corresponds to the 2D image 50A of interest can be visually identified from among the multiple portions 54.

[0163] The inspection support device 10 according to the first embodiment has a first mode, a second mode, and a third mode, but any one of the first mode, second mode, and third mode may be omitted. Alternatively, the inspection support device 10 according to the first embodiment may have only one of the first mode, second mode, and third mode.

[0164] [Second Embodiment] Next, a second embodiment of this disclosure will be described.

[0165] In the second embodiment, the inspection support device 10 has the following configuration changes compared to the first embodiment.

[0166] As an example, as shown in Figure 27, the processor 20 operates as the fourth display control unit 94A. The fourth display control unit 94A displays the fourth image 64 on the screen 16A. The fourth image 64 includes a first image area 81 and a fifth image area 85. As an example, the first image area 81 and the fifth image area 85 are, 4 image 64 They are displayed on screen 16A, arranged horizontally. The first image region 81 is the same as the first image region 81 of the first image 61 (see Figure 10). The fifth image region 85 contains the three-dimensional image 52.

[0167] The fourth display control unit 94A includes a plurality of two-dimensional images 50 in the first image region 81 based on the plurality of two-dimensional images 50 included in the inspection support information 56. The fourth display control unit 94A also includes a three-dimensional image 52 in the fifth image region 85 based on the three-dimensional image 52 included in the inspection support information 56.

[0168] The fifth image region 85 contains the 3D image 52 in a 2D image state after rendering. For example, the size of the 3D image 52 is changed when the inspector 6 gives an instruction to the receiving device 14 (see Figure 9) to change the size of the 3D image 52. Also, for example, the 3D image 52 is rotated when the inspector 6 gives an instruction to the receiving device 14 to rotate the 3D image 52.

[0169] The first image region 81, which contains multiple two-dimensional images 50, and the fifth image region 85, which contains a three-dimensional image 52, are displayed side by side on screen 16A, making it possible to compare the multiple two-dimensional images 50 and the three-dimensional image 52.

[0170] In Figure 27, an example is shown in which the first image region 81 and the fifth image region 85 are displayed on screen 16A with the fourth image 64 aligned horizontally. However, for example, the first image region 81 and the fifth image region 85 may be displayed on screen 16A with the fourth image 64 aligned vertically, or the first image region 81 and the fifth image region 85 may be displayed on screen 16A with the first image region 81 incorporated into a part of the fifth image region 85.

[0171] Image region 81 is an example of the "first region" relating to the technology of this disclosure. Image region 85 is an example of the "second region" relating to the technology of this disclosure.

[0172] As an example, as shown in Figure 28, when the fourth image 64 is displayed on the screen 16A, and the receiving device 14 receives a selection instruction, which is an instruction to select any portion 54 of the three-dimensional image 52 included in the fifth image region 85, the receiving device 14 outputs a selection instruction signal to the processor 20 indicating the selection instruction.

[0173] The processor 20 operates as a fourth image selection unit 94B. When a selection instruction signal is input to the processor 20, the fourth image selection unit 94B selects a portion 54 corresponding to the selection instruction (i.e., the portion of interest 54A) from among a plurality of portions 54 included in the inspection support information 56, according to the selection instruction indicated by the selection instruction signal. The fourth image selection unit 94B also selects a two-dimensional image 50 corresponding to the portion of interest 54A (i.e., the two-dimensional image of interest 50A) from among a plurality of two-dimensional images 50 included in the inspection support information 56.

[0174] As an example, as shown in Figure 29, the processor 20 operates as a fourth pixel extraction unit 94C and a fourth image generation unit 94D. The fourth pixel extraction unit 94C extracts a portion of interest 54A corresponding to the two-dimensional image of interest 50A from the three-dimensional image 52 included in the inspection support information 56.

[0175] The fourth image generation unit 94D generates a first image region 81 that includes a predetermined number of two-dimensional images 50 from among the multiple two-dimensional images 50 included in the inspection support information 56, and in which the two-dimensional image of interest 50A is enclosed by a frame 90. The fourth image generation unit 94D also generates a fifth image region 85 that includes the three-dimensional image 52 based on the three-dimensional image 52 included in the inspection support information 56. Furthermore, the fourth image generation unit 94D includes the portion of interest 54A extracted by the fourth pixel extraction unit 94C in the fifth image region 85 in a state that is distinguishable from the remaining portion 54. For example, the fourth image generation unit 94D includes the portion of interest 54A in the fifth image region 85 in a state in which it is represented in a different color from the remaining portion 54.

[0176] As an example, as shown in Figure 30, the fourth image generation unit 94D generates the fourth image 64 by combining the generated first image region 81 and fifth image region 85.

[0177] The fourth display control unit 94A outputs fourth image data, which represents the fourth image 64 generated by the fourth image generation unit 94D, to the display 16. As a result, the fourth image 64 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of two-dimensional images 50 out of the multiple two-dimensional images 50 are included in the first image area 81 and displayed on the screen 16A, and the two-dimensional image of interest 50A is displayed on the screen 16A surrounded by a frame 90. By displaying the two-dimensional image of interest 50A surrounded by a frame 90 on the screen 16A, the two-dimensional image of interest 50A becomes distinguishable from the remaining two-dimensional images 50 out of the multiple two-dimensional images 50.

[0178] Furthermore, the area of ​​interest 54A included in the 3D image 52 is displayed on screen 16A in a state that makes it distinguishable from the rest of the area 54. By displaying the area of ​​interest 54A on screen 16A in a state that makes it distinguishable from the rest of the area 54, the area of ​​interest 54A in the 3D image 52 becomes visually identifiable.

[0179] In the example shown in Figure 30, the area of ​​interest 54A is displayed in a different color from the rest of the area 54. However, the area of ​​interest 54A may be displayed on the screen 16A in a manner that makes it distinguishable from the rest of the area 54 by other means. For example, the area of ​​interest 54A may be displayed on the screen 16A in a manner that surrounds it with a frame, has a pattern applied to it, or has a higher brightness than the rest of the area 54. In such examples, the area of ​​interest 54A can still be distinguished from the rest of the area 54.

[0180] Next, the operation of the inspection support device 10 according to the second embodiment will be explained with reference to Figure 31. Figure 31 shows an example of the flow of the inspection support process according to the second embodiment.

[0181] In the inspection support process shown in Figure 31, first, in step ST100, the fourth display control unit 94A (see Figure 27) displays the fourth image 64 on screen 16A. After the process in step ST100 is executed, the inspection support process proceeds to step ST102.

[0182] In step ST102, the fourth image selection unit 94B (see Figure 28) determines whether a selection instruction signal, which is an instruction to select any part 54 of the three-dimensional image 52, has been input to the processor 20. If a selection instruction signal indicating a selection instruction has been input to the processor 20 in step ST102, the determination is affirmed, and the inspection support process proceeds to step ST104. If a selection instruction signal indicating a selection instruction has not been input to the processor 20 in step ST102, the determination is denied, and the inspection support process proceeds to step ST112.

[0183] In step ST104, the fourth image selection unit 94B selects a portion 54A of interest from among multiple portions 54 included in the inspection support information 56, corresponding to the selection instruction indicated by the selection instruction signal. The fourth image selection unit 94B also selects a 2D image 50A of interest corresponding to the portion 54A of interest from among multiple 2D images 50 included in the inspection support information 56. After the processing in step ST104 is executed, the inspection support process proceeds to step ST106.

[0184] In step ST106, the fourth pixel extraction unit 94C (see Figure 29) extracts the portion of interest 54A corresponding to the two-dimensional image of interest 50A from the three-dimensional image 52 included in the inspection support information 56. After the processing in step ST106 is completed, the inspection support process proceeds to step ST108.

[0185] In step ST108, the fourth image generation unit 94D (see Figure 30) generates a first image region 81 that includes a predetermined number of two-dimensional images 50 from among the multiple two-dimensional images 50 included in the inspection support information 56, and in which the two-dimensional image of interest 50A is enclosed by a frame 90. The fourth image generation unit 94D also generates a fifth image region 85 that includes the three-dimensional image 52 based on the three-dimensional image 52 included in the inspection support information 56. Furthermore, the fourth image generation unit 94D includes the portion of interest 54A extracted in step ST106 in the fifth image region 85 in a state that is distinguishable from the remaining portion 54. Then, the fourth image generation unit 94D generates the fourth image 64 by combining the generated first image region 81 and fifth image region 85. After the processing in step ST108 is executed, the inspection support process proceeds to step ST110.

[0186] In step ST110, the fourth display control unit 94A (see Figure 30) outputs the fourth image data, which represents the fourth image 64 generated in step ST108, to the display 16. As a result, the fourth image 64 is displayed on screen 16A of the display 16. After the processing in step ST110 is completed, the inspection support process proceeds to step ST112.

[0187] In step ST112, the processor 20 determines whether the conditions for terminating the inspection support process (hereinafter referred to as "termination conditions") have been met. An example of a termination condition is that a termination instruction from the inspector 6 is received by the receiving device 14, and a termination instruction signal from the receiving device 14 is input to the processor 20. If the termination conditions are not met in step ST112, the determination is denied, and the inspection support process proceeds to step ST102. If the termination conditions are met in step ST112, the determination is affirmed, and the inspection support process terminates.

[0188] As described above, in the inspection support device 10 according to the second embodiment, the processor 20 displays on the screen 16A a plurality of two-dimensional images 50 that were used to generate the three-dimensional image 52 representing the object 4 in real space, and which are associated with a plurality of parts 54 of the three-dimensional image 52, in a state that allows comparison between the three-dimensional image 52 and the two-dimensional image 50 (see Figure 27). The processor 20 also selects a part of interest 54A from the plurality of parts 54 according to a given selection instruction, and selects a two-dimensional image of interest 50A that corresponds to the part of interest 54A from the plurality of two-dimensional images 50 (see Figure 28). The processor 20 then displays the two-dimensional image of interest 50A on the screen 16A in a state that allows distinction from the remaining two-dimensional images 50 of the plurality of two-dimensional images 50. Therefore, the correspondence between each two-dimensional image 50 and the region of the object 4 corresponding to each two-dimensional image 50 can be visually grasped.

[0189] Furthermore, the processor 20 displays the area of ​​interest 54A on the screen 16A in a visually identifiable state (see Figure 30). Therefore, the correspondence between the two-dimensional image of interest 50A and the area of ​​interest 54A can be visually identified.

[0190] Furthermore, the state in which the area of ​​interest 54A can be visually identified includes the state in which the area of ​​interest 54A can be distinguished from the remaining parts 54 of the multiple parts 54 (see Figure 30). Therefore, for example, the visibility of the area of ​​interest 54A can be improved compared to the case in which the area of ​​interest 54A cannot be distinguished from the remaining parts 54.

[0191] Furthermore, the operating modes of the inspection support device 10 according to the second embodiment may be added as a fourth mode to the operating modes of the inspection support device 10 according to the first embodiment.

[0192] Furthermore, although the above embodiment illustrates a processor 20, at least one other CPU, at least one GPU, and / or at least one TPU may be used instead of, or in conjunction with, the processor 20.

[0193] Furthermore, although the above embodiment describes an example in which the inspection support information generation program 30 and the inspection support program 40 are stored in the storage 22, the technology of this disclosure is not limited thereto. For example, the inspection support information generation program 30 and / or the inspection support program 40 may be stored in a portable, non-temporary, computer-readable storage medium such as an SSD or USB memory (hereinafter simply referred to as "non-temporary storage medium"). The inspection support information generation program 30 and / or the inspection support program 40 stored in the non-temporary storage medium may be installed in the computer 12 of the inspection support device 10.

[0194] Alternatively, the inspection support information generation program 30 and / or the inspection support program 40 may be stored in a storage device such as another computer or server connected to the inspection support device 10 via a network, and the inspection support information generation program 30 and / or the inspection support program 40 may be downloaded and installed on the computer 12 in response to a request from the inspection support device 10.

[0195] Furthermore, it is not necessary to store all of the inspection support information generation program 30 and / or the inspection support program 40 in a storage device such as another computer or server connected to the inspection support device 10, or in storage device 22; it is acceptable to store only a portion of the inspection support information generation program 30 and / or the inspection support program 40.

[0196] Furthermore, although the inspection support device 10 has a built-in computer 12, the technology of this disclosure is not limited to this, and for example, the computer 12 may be provided outside the inspection support device 10.

[0197] Furthermore, although the above embodiment illustrates a computer 12 including a processor 20, storage 22, and RAM 24, the technology of this disclosure is not limited thereto, and devices including ASICs, FPGAs, and / or PLDs may be used instead of the computer 12. Alternatively, a combination of hardware and software configurations may be used instead of the computer 12.

[0198] Furthermore, the following types of processors can be used as hardware resources to perform the various processes described in the above embodiments. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for performing various processes by executing software, i.e., a program. Other examples of processors include dedicated electronic circuits, which are processors with circuit configurations specifically designed to perform particular processes, such as FPGAs, PLDs, or ASICs. Each processor has built-in or connected memory, and each processor performs various processes by using this memory.

[0199] The hardware resources that perform various processes may consist of one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources that perform various processes may consist of a single processor.

[0200] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs various processes. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform various processes, on a single IC chip, as exemplified by SoCs. In this way, various processes are realized using one or more of the above types of processors as hardware resources.

[0201] Furthermore, the hardware structure of these various processors can more specifically utilize electronic circuits that combine circuit elements such as semiconductor elements. The above process This is merely one example. Therefore, it goes without saying that you may remove unnecessary steps, add new steps, or change the processing order, as long as you do not deviate from the main purpose.

[0202] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0203] In this specification, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0204] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. Equipped with a processor, The aforementioned processor, A plurality of two-dimensional images used to generate a three-dimensional image representing an object in real space, wherein the plurality of two-dimensional images, which are associated with a plurality of parts of the three-dimensional image, are displayed on a screen in a manner that allows for comparison with the three-dimensional image. In accordance with the given selection instructions, select the two-dimensional image of interest from the plurality of two-dimensional images, The portion of the aforementioned plurality of parts that corresponds to the two-dimensional image of interest is displayed on the screen in a way that allows for visual identification. Image processing device.

2. The state in which the plurality of two-dimensional images and the three-dimensional image can be compared is a state in which a first region containing the plurality of two-dimensional images and a second region containing the three-dimensional image are placed side by side. The image processing apparatus according to claim 1.

3. The state in which the portion of interest can be visually identified includes a state in which the portion of interest can be distinguished from the remaining portions of the plurality of parts. The image processing apparatus according to claim 1.

4. The state in which the portion of interest can be visually identified includes a state in which the two-dimensional image of interest can be distinguished from the remaining two-dimensional images among the plurality of two-dimensional images. The image processing apparatus according to claim 1.

5. The aforementioned processor, Multiple position-identifying images, which can identify multiple imaging locations where images were taken to obtain the multiple two-dimensional images, are displayed on the screen in a manner that allows them to be compared with the three-dimensional image. In accordance with the selection instruction, the imaging position corresponding to the selected object of focus image from the plurality of object-finding images is selected from the plurality of imaging positions as the object of focus imaging position. The two-dimensional image obtained by performing the imaging from the aforementioned target imaging position is selected from the plurality of two-dimensional images as the target two-dimensional image. The image processing apparatus according to claim 1.

6. The state in which the plurality of position-identifying images and the three-dimensional image can be compared includes the state in which the plurality of position-identifying images and the three-dimensional image are facing each other. The image processing apparatus according to claim 5.

7. The state in which the plurality of two-dimensional images and the three-dimensional image can be compared is a state in which a third region containing the plurality of two-dimensional images and a fourth region containing an image showing the plurality of position-specific images and the three-dimensional image facing each other are arranged side by side. The image processing apparatus according to claim 5.

8. The state in which the portion of interest can be visually identified includes a state in which the image of the location of interest can be distinguished from the remaining location-identifying images among the plurality of location-identifying images. The image processing apparatus according to claim 5.

9. The image processing device has a first operating mode in which the plurality of two-dimensional images and the three-dimensional image are displayed on the screen in a manner that allows for comparison, and a second operating mode in which the plurality of location-specific images are displayed on the screen in a manner that allows for comparison with the three-dimensional image. The processor sets either the first operating mode or the second operating mode according to a given setting instruction. The image processing apparatus according to claim 5.

10. The three-dimensional image is displayed on the screen from a viewpoint corresponding to the two-dimensional image of interest. The image processing apparatus according to claim 5.

11. Equipped with a processor, The aforementioned processor, A plurality of two-dimensional images used to generate a three-dimensional image representing an object in real space, wherein the plurality of two-dimensional images, which are associated with a plurality of parts of the three-dimensional image, are displayed on a screen in a manner that allows for comparison with the three-dimensional image. In accordance with the given selection instructions, select the portion of interest from the plurality of parts, Select the two-dimensional image of interest that corresponds to the portion of interest from the plurality of two-dimensional images, The aforementioned two-dimensional image of interest is displayed on the screen in a manner that makes it distinguishable from the remaining two-dimensional images among the plurality of two-dimensional images. Image processing device.

12. The aforementioned processor, Multiple position-identifying images, which can identify multiple imaging locations where images were taken to obtain the multiple two-dimensional images, are displayed on the screen in a manner that allows them to be compared with the three-dimensional image. In accordance with the selection instructions, select the location-specific image of interest from the plurality of location-specific images. The two-dimensional image obtained by performing the imaging from the imaging position identified from the aforementioned image of the area of ​​interest is selected from the plurality of two-dimensional images as the two-dimensional image of interest. The image processing apparatus according to claim 11.

13. Multiple two-dimensional images used to generate a three-dimensional image representing an object in real space, wherein the multiple two-dimensional images, each corresponding to a portion of the three-dimensional image, are displayed on a screen in a manner that allows for comparison with the three-dimensional image. Selecting a two-dimensional image of interest from the plurality of two-dimensional images according to the given selection instructions, and To display on the screen the portion of the aforementioned two-dimensional image of interest that corresponds to the portion of the aforementioned multiple portions in a manner that allows for visual identification. An image processing method comprising:

14. Multiple two-dimensional images used to generate a three-dimensional image representing an object in real space, wherein the multiple two-dimensional images, each corresponding to a portion of the three-dimensional image, are displayed on a screen in a manner that allows for comparison with the three-dimensional image. Selecting a two-dimensional image of interest from the plurality of two-dimensional images according to the given selection instructions, and To display on the screen the portion of the aforementioned two-dimensional image of interest that corresponds to the portion of the aforementioned multiple portions in a manner that allows for visual identification. A program that causes a computer to perform a process that includes [a specific action].

Images