Image processing device, image processing method, and program
By generating three-dimensional images associated with multiple two-dimensional images and visually distinguishing the focus part, the problem of comparing and distinguishing complexity between two-dimensional images and three-dimensional images in the prior art is solved, and the efficiency and accuracy of object inspection are improved. sex.
Patent Information
- Application Number
- CN202280093919.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-29
- Filing Date
- 2022-11-09
- Publication Date
- 2024-11-05
AI Technical Summary
The prior art is difficult to visually grasp the correspondence between two-dimensional images and objects effectively, especially in the comparison and distinction between multiple two-dimensional images and three-dimensional images, which affects the inspection and analysis efficiency of objects. .
Through the image processing device and method, a three-dimensional image displaying the object in real space is generated and a corresponding relationship is established with a plurality of two-dimensional images. The focus image is selected from the plurality of two-dimensional images according to the selection instructions, and visually distinguish the focus part from the focus The remaining part realizes the comparison and distinction between multiple two-dimensional images and three-dimensional images.
It improves visual understanding of the correspondence between two-dimensional images and the object area and the ability to compare multiple two-dimensional images and three-dimensional images, enhances the recognition of the focus parts, and simplifies the inspection and analysis process of object objects.
Smart Images

Figure CN118901086A_ABST
Abstract
Description
Technical Field
[0001] The technology of the present invention relates to an image processing device, an image processing method, and a program. Background Art
[0002] In Japanese Patent Gazette No. 2020-005186, an image display system composed of a computer system is disclosed. The computer system inputs an image group including multiple images with different shooting dates, times, locations and directions, displays an overview of the image group on an overview screen, and displays the first image selected from the image group on a separate screen according to the user's operation. In addition, the computer system determines the image adjacent to the first image based on the judgment of the spatial positional relationship between the first image and the group of candidate images spatially surrounding the first image and the judgment of the repetitive state related to the shooting range. On the separate screen, according to the user's operation, the image adjacent to the first image is selected as the second image, and the second image is displayed as the new first image.
[0003] In Japanese Patent Publication No. 2007-093661, a navigation device is disclosed, which is mounted on a car and simultaneously displays a first map and a second map that is represented differently from the first map. The navigation device includes a display device, a map display mechanism, a current position calculation mechanism, a current position display mechanism, and a position designation receiving mechanism. The map display mechanism displays the first map and the second map in different display areas of the display device. The current position calculation mechanism calculates the current position. The current position display mechanism displays a current position mark indicating the current position calculated by the current position calculation mechanism on at least one of the first map and the second map displayed by the map display mechanism. The position designation receiving mechanism receives a designation of a position on the display area where the first map is displayed from a user. The map display mechanism displays the second map in a manner that enables identification of a position on the second map that represents the same place as the place on the first map corresponding to the position designated by the position designation mechanism.
[0004] Japanese Patent Publication No. 2010-200024 discloses a stereoscopic image display device. The stereoscopic image display device includes a display mechanism, an instruction input mechanism, an alignment mechanism, and a display control mechanism. The display mechanism displays a list of thumbnail images generated based on the camera images before stereoscopically displaying the camera images shot at multiple viewpoints. The instruction input mechanism inputs a selection instruction for selecting the thumbnail images in the list. When the alignment mechanism inputs a selection instruction to stereoscopically display the camera images, the camera images of multiple viewpoints corresponding to the selected thumbnail images are aligned with each other within a detection area of a specific object in the camera images. The display control mechanism adds detection area information indicating the detection area of the specific object to the thumbnail image. Summary of the invention
[0005] As an example, an embodiment of the technology according to the present invention provides an image processing device, an image processing method, and a program capable of visually grasping the correspondence between each two-dimensional image and a region in an object corresponding to each two-dimensional image.
[0006] Means for solving technical problems
[0007] The first method involved in the technology of the present invention is an image processing device, which includes a processor, wherein the processor performs the following processing: multiple two-dimensional images and three-dimensional images that are used to generate a three-dimensional image showing an object in real space and have corresponding associations with multiple parts in the three-dimensional image are displayed on the screen in a state that can be compared; a focus two-dimensional image is selected from the multiple two-dimensional images according to a given selection instruction; and a focus part corresponding to the focus two-dimensional image among the multiple parts is displayed on the screen in a state that can be visually determined.
[0008] The second mode involved in the technology of the present invention is an image processing device, wherein, in the image processing device involved in the first mode, the state in which multiple two-dimensional images can be compared with a three-dimensional image refers to a state in which a first area including multiple two-dimensional images and a second area including a three-dimensional image are arranged side by side.
[0009] A third aspect of the technology of the present invention is an image processing device, wherein, in the image processing device of the first aspect or the second aspect, a state in which a focus portion can be visually determined includes a state in which the focus portion can be distinguished from the remaining portions of the plurality of portions.
[0010] The fourth mode involved in the technology of the present invention is an image processing device, wherein, in the image processing device involved in any one of the first to third modes, the state of being able to visually determine the focus portion includes a state in which the focus two-dimensional image can be distinguished from the remaining two-dimensional images among multiple two-dimensional images.
[0011] The fifth mode involved in the technology of the present invention is an image processing device, wherein, in the image processing device involved in any one of the first mode to the fourth mode, the processor performs the following processing: a plurality of position-specific images that can determine a plurality of shooting positions at which shooting was performed in order to obtain a plurality of two-dimensional images are displayed on the screen in a state that can be compared with the three-dimensional image; according to a selection instruction, a shooting position corresponding to a focus position-specific image selected from the plurality of position-specific images is selected from the plurality of shooting positions as a focus shooting position; and a two-dimensional image obtained by shooting from the focus shooting position is selected from the plurality of two-dimensional images as the focus two-dimensional image.
[0012] A sixth aspect of the technology of the present invention is an image processing device, wherein, in the image processing device of the fifth aspect, a state in which a plurality of position-specific images can be compared with a three-dimensional image includes a state in which the plurality of position-specific images and the three-dimensional image are opposed to each other.
[0013] The seventh mode involved in the technology of the present invention is an image processing device, wherein, in the image processing device involved in the fifth mode or the sixth mode, the state in which multiple two-dimensional images can be compared with a three-dimensional image refers to a state in which a third area including multiple two-dimensional images and a fourth area including an image displaying a manner in which multiple position-specific images and a three-dimensional image are opposite to each other are arranged side by side.
[0014] The eighth mode involved in the technology of the present invention is an image processing device, wherein, in the image processing device involved in any one of the fifth to seventh modes, the state in which the focus portion can be visually determined includes a state in which a focus position specific image can be distinguished from the remaining position specific images among multiple position specific images.
[0015] The 9th mode involved in the technology of the present invention is an image processing device, wherein, in the image processing device involved in any one of the 5th to 8th modes, the image processing device has: a first action mode, in which a plurality of two-dimensional images and a three-dimensional image are displayed on the screen in a state in which they can be compared; and a second action mode, in which a plurality of position-specific images are displayed on the screen in a state in which they can be compared with the three-dimensional image, and the processor sets any one of the first action mode and the second action mode according to a given setting instruction.
[0016] A tenth aspect of the technology of the present invention is an image processing device according to any one of the fifth to ninth aspects, wherein a three-dimensional image is displayed on a screen at a viewpoint corresponding to a two-dimensional image of interest.
[0017] The 11th method involved in the technology of the present invention is an image processing device, which includes a processor, wherein the processor performs the following processing: displaying multiple two-dimensional images and three-dimensional images that are used to generate a three-dimensional image showing an object in real space and have corresponding associations with multiple parts in the three-dimensional image on the screen in a state that can be compared; selecting a focus part from multiple parts according to a given selection instruction, selecting a focus two-dimensional image corresponding to the focus part from the multiple two-dimensional images; and displaying the focus two-dimensional image on the screen in a state that can be distinguished from the remaining two-dimensional images in the multiple two-dimensional images.
[0018] The 12th method involved in the technology of the present invention is an image processing device, wherein, in the image processing device involved in the 11th method, the processor performs the following processing: a plurality of position-specific images that can determine a plurality of shooting positions at which shooting was performed in order to obtain a plurality of two-dimensional images are displayed on the screen in a state that can be compared with the three-dimensional image; a focus position-specific image is selected from the plurality of position-specific images according to a selection instruction; and a two-dimensional image obtained by shooting from the shooting position determined according to the focus position-specific image is selected from the plurality of two-dimensional images as the focus two-dimensional image.
[0019] The 13th method involved in the technology of the present invention is an image processing method, which includes the following steps: displaying multiple two-dimensional images and three-dimensional images that are used to generate a three-dimensional image showing an object in real space and have corresponding associations with multiple parts in the three-dimensional image on a screen in a state that can be compared; selecting a focus two-dimensional image from the multiple two-dimensional images according to a given selection instruction; and displaying the focus part corresponding to the focus two-dimensional image among the multiple parts on the screen in a state that can be visually determined.
[0020] The 14th method involved in the technology of the present invention is a program, which is used to cause a computer to execute processing including the following steps: displaying multiple two-dimensional images and three-dimensional images that are used to generate a three-dimensional image showing an object in real space and have corresponding associations with multiple parts in the three-dimensional image on the screen in a state that can be compared; selecting a focus two-dimensional image from the multiple two-dimensional images according to a given selection instruction; and displaying the focus part corresponding to the focus two-dimensional image among the multiple parts on the screen in a state that can be visually determined. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is a perspective view showing an example of the inspection system according to the first embodiment.
[0022] Figure 2 This is a block diagram showing an example of the inspection supporting device according to the first embodiment.
[0023] Figure 3 This is a block diagram showing an example of the imaging device according to the first embodiment.
[0024] Figure 4 This is a block diagram showing an example of a functional configuration for realizing the inspection support information generation process according to the first embodiment.
[0025] Figure 5 This is a block diagram showing an example of data transmitted from the imaging device according to the first embodiment to the inspection supporting device.
[0026] Figure 6This is a block diagram showing an example of the operations of the acquisition unit and the three-dimensional image generation unit according to the first embodiment.
[0027] Figure 7 This is a block diagram showing an example of the operations of the three-dimensional image generating unit and the auxiliary examination information generating unit according to the first embodiment.
[0028] Figure 8 This is a block diagram showing an example of a functional configuration for realizing the inspection support processing according to the first embodiment.
[0029] Fig. 9 This is a block diagram showing an example of the operations of the operation mode setting unit, the first mode processing unit, the second mode processing unit, and the third mode processing unit according to the first embodiment.
[0030] Fig.10 This is a block diagram showing an example of the operation of the first display control unit according to the first embodiment.
[0031] Fig.11 This is a block diagram showing an example of the operation of the first image selection unit according to the first embodiment.
[0032] Fig.12 This is a block diagram showing an example of the operation of the first pixel extraction unit and the first image generation unit according to the first embodiment.
[0033] Fig.13 This is a block diagram showing an example of the operation of the first image generating unit and the first display control unit according to the first embodiment.
[0034] Fig.14 This is a block diagram showing an example of the operation of the second display control unit according to the first embodiment.
[0035] Fig.15 This is a block diagram showing an example of the operation of the second image selection unit according to the first embodiment.
[0036] Fig.16 This is a block diagram showing an example of the operation of the second pixel extraction unit and the second image generation unit according to the first embodiment.
[0037] Fig.17 This is a block diagram showing an example of the operation of the second image generating unit and the second display control unit according to the first embodiment.
[0038] Fig.18 This is a block diagram showing an example of the operation of the third display control unit according to the first embodiment.
[0039] Fig.19This is a block diagram showing an example of the operation of the third image selection unit according to the first embodiment.
[0040] Fig. 20 This is a block diagram showing an example of the operation of the third image generating unit according to the first embodiment.
[0041] Fig.21 This is a block diagram showing an example of the operation of the third image generating unit and the third display control unit according to the first embodiment.
[0042] Fig. 22 This is a flowchart showing an example of the flow of the auxiliary examination information generation process according to the first embodiment.
[0043] Fig.23 This is a flowchart showing an example of the flow of the mode setting process in the inspection support process according to the first embodiment.
[0044] Fig.24 This is a flowchart showing an example of the flow of the first mode process in the inspection support process according to the first embodiment.
[0045] Fig.25 This is a flowchart showing an example of the flow of the second mode process in the inspection support process according to the first embodiment.
[0046] Fig.26 This is a flowchart showing an example of the flow of the third mode process in the inspection support process according to the first embodiment.
[0047] Fig. 27 This is a block diagram showing an example of the operation of the fourth display control unit according to the second embodiment.
[0048] Fig.28 This is a block diagram showing an example of the operation of the fourth image selection unit according to the second embodiment.
[0049] Fig.29 This is a block diagram showing an example of the operation of the fourth pixel extraction unit and the fourth image generation unit according to the second embodiment.
[0050] Fig.30 This is a block diagram showing an example of the operation of the fourth image generating unit and the fourth display control unit according to the second embodiment.
[0051] Fig.31 This is a flowchart showing an example of the inspection support process according to the second embodiment. DETAILED DESCRIPTION
[0052] Hereinafter, an example of an embodiment of an image processing device, an image processing method, and a program according to the technology of the present invention will be described with reference to the drawings.
[0053] First, the terms used in the following description are explained.
[0054] CPU is the abbreviation of "Central Processing Unit". GPU is the abbreviation of "Graphics Processing Unit". HDD is the abbreviation of "Hard Disk Drive". SSD is the abbreviation of "Solid State Drive". RAM is the abbreviation of "Random Access Memory". SRAM is the abbreviation of "Static Random Access Memory". DRAM is the abbreviation of "Dynamic Random Access Memory". EL is the abbreviation of "Electro Luminescence". RAM is the abbreviation of "Random Access Memory". CMOS is the abbreviation of "Complementary Metal Oxide Semiconductor". GNSS is the abbreviation of "Global Navigation Satellite System". GPS is the abbreviation of "Global Positioning System". SfM is the abbreviation of "Structure from Motion". MVS is the abbreviation of "Multi-View Stereo". TPU is the abbreviation of "Tensor processing unit". USB is the abbreviation of "Universal Serial Bus". ASIC is the abbreviation of "Application Specific Integrated Circuit". FPGA is the abbreviation of "Field-Programmable Gate Array". PLD is the abbreviation of "Programmable Logic Device". SoC is the abbreviation of "System-on-a-chip". IC is the abbreviation of "Integrated Circuit".
[0055] [First embodiment]
[0056] First, a first embodiment of the present invention will be described.
[0057] As an example, Figure 1 As shown, the inspection system S includes an inspection support device 10 and an imaging device 100. The inspection system S is a system for inspecting an object 4 in a real space. The object 4 is an example of the "object" of the technology of the present invention.
[0058] As an example, the object 4 is a bridge pier made of reinforced concrete. Here, as an example of the object 4, a bridge pier is cited, but the object 4 may also be a road facility other than the bridge pier. As road facilities, for example, road surfaces, tunnels, guardrails, traffic lights and / or windbreak fences may be cited. The object 4 may be a social infrastructure other than road facilities (for example, airport facilities, port facilities, water storage facilities, gas facilities, medical facilities, firefighting facilities and / or educational facilities, etc.), or private property. Furthermore, the object 4 may also be land (for example, state-owned land and / or private land, etc.). The bridge pier exemplified as the object 4 may also be a bridge pier other than that made of reinforced concrete. In the first embodiment, the inspection, for example, refers to the inspection of the state of the object 4. For example, the inspection system S is used to inspect whether the object 4 is damaged and / or the degree of damage.
[0059] The inspection assisting device 10 is an example of an "image processing device" involved in the technology of the present invention. The inspection assisting device 10 is, for example, a desktop personal computer. Here, a desktop personal computer is illustrated as the inspection assisting device 10, but this is only an example, and it may also be a notebook personal computer. Furthermore, it is not limited to a personal computer, and it may also be a server. In addition, the server may be a large computer used together with the inspection assisting device 10 through local deployment (On-Premises), or it may be an external server implemented through cloud computing. Furthermore, the server may also be an external server implemented through network computing such as fog computing, edge computing, or grid computing. The inspection assisting device 10 is connected to the camera device 100 and is capable of communication. The inspection assisting device 10 is used by the inspector 6. The inspection assisting device 10 may be used at a site where the object 4 is set, or it may be used at a location different from the site where the object 4 is set.
[0060] The camera device 100 is, for example, a lens-interchangeable digital camera. Here, a lens-interchangeable digital camera is exemplified as the camera device 100, but this is only an example, and it may also be a digital camera built into various electronic devices such as a smart device or a wearable terminal. In addition, the camera device 100 may be a glasses-type eyeglass terminal, or a head-mounted display terminal worn on the head. The camera device 100 is used by the photographer 8.
[0061] As an example, Figure 2 As shown, the inspection support device 10 includes a computer 12 , a receiving device 14 , a display 16 , and a communication device 18 .
[0062] The computer 12 is an example of a "computer" involved in the technology of the present invention. The computer 12 includes a processor 20, a storage device 22, and a RAM 24. The processor 20 is an example of a "processor" involved in the technology of the present invention. The processor 20, the storage device 22, the RAM 24, the receiving device 14, the display 16, and the communication device 18 are connected to a bus 26.
[0063] The processor 20 includes, for example, a CPU, and controls the entire inspection auxiliary device 10. Here, an example in which the processor 20 includes a CPU is given, but this is only an example. For example, the processor 20 may include a CPU and a GPU. In this case, for example, the GPU operates under the control of the CPU and is responsible for performing image processing.
[0064] The storage device 22 is a nonvolatile storage device that stores various programs and various parameters. As the storage device 22, for example, a HDD and an SSD can be cited. In addition, the HDD and the SSD are only examples, and a flash memory, a magnetoresistive memory, and / or a ferroelectric memory may be used instead of or in addition to the HDD and / or the SSD.
[0065] The RAM 24 is a memory for temporarily storing information, and is used as a working memory by the processor 20. Examples of the RAM 24 include DRAM and / or SRAM.
[0066] The receiving device 14 has a keyboard, a mouse, a touch panel, etc. (all omitted from the figure), and receives various instructions from the inspector 6. The display 16 has a screen 16A. The screen 16A is an example of a "screen" involved in the technology of the present invention. Under the control of the processor 20, the display 16 displays various information (for example, images and characters, etc.) on the screen 16A. As the display 16, for example, an EL display (for example, an organic EL display or an inorganic EL display) can be cited. In addition, it is not limited to an EL display, and other types of displays such as a liquid crystal display can also be used.
[0067] The communication device 18 is connected to the imaging device 100 and is capable of communication. Here, the communication device 18 is connected to the imaging device 100 by a prescribed wireless communication standard and is capable of wireless communication. Examples of the prescribed wireless communication standard include Wi-Fi (registered trademark) and Bluetooth (registered trademark). The communication device 18 is responsible for sending and receiving information with the inspection auxiliary device 10. For example, the communication device 18 sends information in response to a request from the processor 20 to the imaging device 100. In addition, the communication device 18 receives information sent from the imaging device 100 and outputs the received information to the processor 20 via the bus 26. In addition, the communication device 18 may also be connected to the imaging device 100 by wire and be capable of communication.
[0068] As an example, Figure 3 As shown, the imaging device 100 includes a computer 102 , an image sensor 104 , a positioning unit 106 , an acceleration sensor 108 , an angular velocity sensor 110 , and a communication device 112 .
[0069] The computer 102 includes a processor 114, a storage device 116, and a RAM 118. The processor 114, the storage device 116, the RAM 118, the image sensor 104, the positioning unit 106, the acceleration sensor 108, the angular velocity sensor 110, and the communication device 112 are connected to a bus 120. The processor 114, the storage device 116, and the RAM 118 are implemented by, for example, the same hardware as the processor 20, the storage device 22, and the RAM 24 included in the above-mentioned inspection supporting device 10.
[0070] The image sensor 104 is, for example, a CMOS image sensor. In addition, here, a CMOS image sensor is exemplified as the image sensor 104, but the technology of the present invention is not limited thereto, and other image sensors may also be used. The image sensor 104 captures a subject (for example, the object 4) and outputs image data obtained by the capture.
[0071] The positioning unit 106 is a device that detects the position of the camera device 100. The position of the camera device 100 is detected using, for example, GNSS (for example, GPS). The positioning unit 106 has a GNSS receiver (not shown). The GNSS receiver receives, for example, radio waves transmitted from a plurality of satellites. The positioning unit 106 detects the position of the camera device 100 based on the radio waves received by the GNSS receiver, and outputs positioning data corresponding to the detected position (for example, data indicating latitude, longitude, and altitude).
[0072] The acceleration sensor 108 detects acceleration in the direction of each axis of the pitch axis, the yaw axis, and the roll axis of the camera device 100. The acceleration sensor 108 outputs acceleration data corresponding to the acceleration in the direction of each axis of the camera device 100. The angular velocity sensor 110 detects angular velocity of each axis of the pitch axis, the yaw axis, and the roll axis of the camera device 100. The angular velocity sensor 110 outputs angular velocity data corresponding to the angular velocity around each axis of the camera device 100.
[0073] The processor 114 acquires the position of the camera device 100 based on the positioning data and / or the acceleration data, and generates position data indicating the acquired position. Furthermore, the processor 114 acquires the posture of the camera device 100 (i.e., the amount of change of the posture defined in the relative coordinate system relative to the reference posture) based on the angular velocity data, and generates posture data indicating the acquired posture. Hereinafter, the position of the camera device 100 is referred to as the "shooting position", and the posture of the camera device 100 is referred to as the "shooting posture".
[0074] In addition, when the processor 114 acquires the shooting position based only on the positioning data, the acceleration sensor 108 can be omitted. On the other hand, when the processor 114 acquires the shooting position based only on the acceleration data, the positioning unit 106 can be omitted. When the processor 114 acquires the shooting position based on the positioning data, the shooting position in the absolute coordinate system can be derived based on the positioning data. On the other hand, when the processor 114 acquires the shooting position based on the acceleration data, the amount of change of the shooting position specified in the relative coordinate system relative to the reference position is derived based on the acceleration data.
[0075] The communication device 112 is connected to the inspection support device 10 so as to be communicable. The communication device 112 is realized by, for example, the same hardware as the communication device 18 included in the inspection support device 10 described above.
[0076] The imaging device 100 transmits image data, position data, and posture data to the inspection support device 10. The image data is data representing the two-dimensional image 50 obtained by imaging the object 4 by the imaging device 100. The position data is data representing the imaging position when the imaging device 100 performs imaging, and is associated with the image data. Similarly, the posture data is data representing the imaging posture when the imaging device 100 performs imaging, and is associated with the image data. That is, the position data and the posture data are additional data added to the image data.
[0077] For example, when a plurality of two-dimensional images 50 obtained by photographing the object 4 from a plurality of photographing positions by the imaging device 100 are displayed only on the screen 16A of the display 16 provided in the inspection auxiliary device 10, it takes time to grasp the correspondence between each two-dimensional image 50 and the area corresponding to each two-dimensional image 50 in the object 4 based on the plurality of two-dimensional images 50 displayed on the screen 16A. Then, the more the number of frames of the plurality of two-dimensional images 50 increases, the more complicated the work of grasping the correspondence becomes. In view of this situation, in the first embodiment, the inspection auxiliary device 10 performs inspection auxiliary information generation processing and inspection auxiliary processing. The inspection auxiliary information generation processing and inspection auxiliary processing performed by the inspection auxiliary device 10 are described in detail below.
[0078] As an example, Figure 4 As shown, an inspection auxiliary information generation program 30 is stored in the storage device 22 of the inspection auxiliary apparatus 10. The processor 20 of the inspection auxiliary apparatus 10 reads the inspection auxiliary information generation program 30 from the storage device 22, and executes the read inspection auxiliary information generation program 30 on the RAM 24. The processor 20 performs inspection auxiliary information generation processing for generating inspection auxiliary information 56 according to the inspection auxiliary information generation program 30 executed on the RAM 24.
[0079] The auxiliary examination information generation process is realized by the processor 20 operating as the acquisition unit 32 , the three-dimensional image generation unit 34 , and the auxiliary examination information generation unit 36 according to the auxiliary examination information generation program 30 .
[0080] As an example, Figure 5 As shown, a plurality of points P1 located in the circumferential direction of the object 4 represent the shooting positions by the camera 100. The photographer 8 moves around the object 4 and shoots the object 4 from a plurality of shooting positions in the circumferential direction of the object 4 by means of the camera 100. As an example, the photographer 8 shoots a different area in the object 4 from each shooting position by means of the camera 100. By shooting a different area in the object 4 from each shooting position by means of the camera 100, the entire object 4 including the plurality of areas can be shot.
[0081] The shooting position (i.e., point P1) corresponding to each two-dimensional image 50 obtained by shooting with the camera 100 is equivalent to the starting point of the line of sight L focused on the object 4, and the shooting posture corresponding to each two-dimensional image 50 is equivalent to the direction of the line of sight L focused on the object 4. The point P2 where the object 4 intersects with the line of sight L is equivalent to the viewpoint when the object 4 is observed from the line of sight L. By shooting the object 4 from each shooting position with the camera 100, the two-dimensional image 50 corresponding to each viewpoint can be obtained. Each two-dimensional image 50 is an image corresponding to each area in the object 4.
[0082] In addition, here, an example is given in which the photographer 8 photographs the object 4 from various photographing positions using the camera device 100 while moving around the object 4. However, when the camera device 100 is mounted on a moving body and the moving body moves around the object 4, the object 4 may be photographed from various photographing positions using the camera device 100. Furthermore, the moving body may be, for example, a drone, a pod, a trolley, an aerial work vehicle, an unmanned vehicle, or other vehicles.
[0083] The imaging device 100 associates image data representing the two-dimensional image 50 obtained by photographing from each photographing position, position data representing the photographing position when photographing, and posture data representing the photographing posture when photographing. Then, the imaging device 100 sends each image data and the position data and posture data associated with each image data to the inspection assisting device 10.
[0084] As an example, Figure 6 As shown, the acquisition unit 32 acquires the two-dimensional image 50 based on each image data received by the inspection auxiliary device 10. In addition, the acquisition unit 32 acquires the shooting position corresponding to each two-dimensional image 50 based on each position data received by the inspection auxiliary device 10. In addition, the acquisition unit 32 acquires the shooting posture corresponding to each two-dimensional image 50 based on each posture data received by the inspection auxiliary device 10.
[0085] The three-dimensional image generation unit 34 generates a three-dimensional image 52 of the display object 4 based on the multiple two-dimensional images 50 acquired by the acquisition unit 32. As image processing techniques for generating the three-dimensional image 52 based on the multiple two-dimensional images 50, SfM, MVS, epipolar geometry, and stereo matching processing can be cited. The positions of the multiple pixels included in the three-dimensional image 52 are determined by the multiple three-dimensional coordinates obtained from the multiple two-dimensional images 50. The three-dimensional image 52 is a three-dimensional model defined by the multiple three-dimensional coordinates.
[0086] As an example, Figure 7 As shown, the three-dimensional image 52 generated by the three-dimensional image generating unit 34 has a plurality of parts 54 corresponding to each two-dimensional image 50. Each part 54 is formed by a set of pixels corresponding to each two-dimensional image 50, that is, a pixel group. The inspection auxiliary information generating unit 36 generates the inspection auxiliary information 56, which is information that establishes a corresponding association between each two-dimensional image 50 acquired by the acquiring unit 32, the shooting position corresponding to each two-dimensional image 50, the shooting posture corresponding to each two-dimensional image 50, and the part 54 corresponding to each two-dimensional image 50. The inspection auxiliary information 56 is stored in the storage device 22.
[0087] As an example, Figure 8 As shown, the storage device 22 of the inspection support device 10 stores an inspection support program 40. The inspection support program 40 is an example of a "program" involved in the technology of the present invention. The processor 20 reads the inspection support program 40 from the storage device 22 and executes the read inspection support program 40 on the RAM 24. The processor 20 performs the inspection support program 40 executed on the RAM 24 to assist the inspector 6 (refer to Figure 1 ) to conduct inspection auxiliary processing.
[0088] The inspection support processing is realized by the processor 20 operating as the operation mode setting unit 42 , the first mode processing unit 44 , the second mode processing unit 46 , and the third mode processing unit 48 according to the inspection support program 40 .
[0089] The inspection support device 10 has a first mode, a second mode, and a third mode as operation modes. The operation mode setting unit 42 performs the following mode setting processing to selectively set the first mode, the second mode, and the third mode as the operation mode of the inspection support device 10 .
[0090] In the mode setting process, when the operation mode of the inspection assisting device 10 is set to the first mode by the operation mode setting unit 42, the processor 20 operates as the first mode processing unit 44. The first mode processing unit 44 performs the first mode process. The first mode process is implemented by the first mode processing unit 44 operating as the first display control unit 44A, the first image selection unit 44B, the first pixel extraction unit 44C, and the first image generation unit 44D.
[0091] In the mode setting process, when the operation mode of the inspection assisting device 10 is set to the second mode by the operation mode setting unit 42, the processor 20 operates as the second mode processing unit 46. The second mode processing unit 46 performs the second mode process. The second mode process is realized by the second mode processing unit 46 operating as the second display control unit 46A, the second image selection unit 46B, the second pixel extraction unit 46C, and the second image generation unit 46D.
[0092] In the mode setting process, when the operation mode of the inspection assisting device 10 is set to the third mode by the operation mode setting unit 42, the processor 20 operates as the third mode processing unit 48. The third mode processing unit 48 performs the third mode process. The third mode process is realized by the third mode processing unit 48 operating as the third display control unit 48A, the third image selection unit 48B, and the third image generation unit 48C.
[0093] As an example, Fig. 9 As shown, the operation mode setting unit 42 sets the first mode as the operation mode of the inspection support device 10 by default. When the operation mode of the inspection support device 10 is set to the first mode by the operation mode setting unit 42, the first display control unit 44A displays the first image 61 on the screen 16A. The details of the first image 61 will be described later, but the first image 61 includes a second mode setting button 72 and a third mode setting button 73 as soft keys.
[0094] When the receiving device 14 receives an instruction to press the second mode setting button 72, i.e., a setting instruction, while the first image 61 is displayed on the screen 16A, the receiving device 14 outputs a second mode setting instruction signal to the processor 20. Similarly, when the receiving device 14 receives an instruction to press the third mode setting button 73, i.e., a setting instruction, while the first image 61 is displayed on the screen 16A, the receiving device 14 outputs a third mode setting instruction signal to the processor 20.
[0095] When the operation mode of the inspection assisting device 10 is set to the first mode, the operation mode setting unit 42 determines whether the second mode setting instruction signal or the third mode setting instruction signal is input to the processor 20. When the second mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the second mode as the operation mode of the inspection assisting device 10. On the other hand, when the third mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the third mode as the operation mode of the inspection assisting device 10.
[0096] When the operation mode setting unit 42 sets the operation mode of the inspection support device 10 to the second mode, the second display control unit 46A displays the second image 62 on the screen 16A. The details of the second image 62 will be described later, but the second image 62 includes a first mode setting button 71 and a third mode setting button 73 as soft keys.
[0097] When the receiving device 14 receives an instruction to press the first mode setting button 71, i.e., a setting instruction, while the second image 62 is displayed on the screen 16A, the receiving device 14 outputs a first mode setting instruction signal to the processor 20. Similarly, when the receiving device 14 receives an instruction to press the third mode setting button 73, i.e., a setting instruction, while the second image 62 is displayed on the screen 16A, the receiving device 14 outputs a third mode setting instruction signal to the processor 20.
[0098] When the operation mode of the inspection assisting device 10 is set to the second mode, the operation mode setting unit 42 determines whether the first mode setting instruction signal or the third mode setting instruction signal is input to the processor 20. When the first mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the first mode as the operation mode of the inspection assisting device 10. On the other hand, when the third mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the third mode as the operation mode of the inspection assisting device 10.
[0099] When the operation mode setting unit 42 sets the operation mode of the inspection support device 10 to the third mode, the third display control unit 48A displays the third image 63 on the screen 16A. The third image 63 will be described in detail later, but the third image 63 includes the first mode setting button 71 and the second mode setting button 72.
[0100] When the receiving device 14 receives an instruction to press the first mode setting button 71, i.e., a setting instruction, while the third image 63 is displayed on the screen 16A, the receiving device 14 outputs a first mode setting instruction signal to the processor 20. Similarly, when the receiving device 14 receives an instruction to press the third mode setting button 73, i.e., a setting instruction, while the third image 63 is displayed on the screen 16A, the receiving device 14 outputs a third mode setting instruction signal to the processor 20.
[0101] When the operation mode of the inspection assisting device 10 is set to the third mode, the operation mode setting unit 42 determines whether the first mode setting instruction signal or the second mode setting instruction signal is input to the processor 20. When the first mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the first mode as the operation mode of the inspection assisting device 10. On the other hand, when the second mode setting instruction signal is input to the processor 20, the operation mode setting unit 42 sets the second mode as the operation mode of the inspection assisting device 10.
[0102] Hereinafter, when there is no need to distinguish between the first mode setting instruction signal, the second mode setting instruction signal and the third mode setting instruction signal, the first mode setting instruction signal, the second mode setting instruction signal and the third mode setting instruction signal are referred to as “mode setting instruction signals”.
[0103] The second mode among the plurality of operation modes of the inspection support device 10 is an example of the “first operation mode” of the present invention. The third mode among the plurality of operation modes of the inspection support device 10 is an example of the “second operation mode” of the present invention.
[0104] As an example, in Fig.10 , a state where the first image 61 is displayed on the screen 16A is shown. The first image 61 includes a first image area 81 and a second image area 82. As an example, the first image area 81 and the second image area 82 are displayed on the screen 16A in a state where they are arranged side by side in the left-right direction of the first image 61. The first image area 81 includes a plurality of two-dimensional images 50, and the second image area 82 includes a three-dimensional image 52.
[0105] The first display control unit 44A includes the plurality of two-dimensional images 50 in the first image region 81 based on the plurality of two-dimensional images 50 included in the auxiliary examination information 56. Furthermore, the first display control unit 44A includes the three-dimensional image 52 in the second image region 82 based on the three-dimensional image 52 included in the auxiliary examination information 56.
[0106] The first image region 81 includes a predetermined number of two-dimensional images 50 among the plurality of two-dimensional images 50. The predetermined number is, for example, determined by the inspector 6 to the receiving device 14 (see Fig. 9 ) is set by giving an instruction to designate a predetermined number. And, for example, by giving an instruction to scroll the first image area 81 to the receiving device 14 by the inspector 6, the first image area 81 is scrolled, thereby changing the two-dimensional image 50 included in the first image area 81.
[0107] The three-dimensional image 52 is included in the second image area 82 in a state of being two-dimensionally imaged by rendering. For example, the inspector 6 receives the image from the receiving device 14 (refer to Fig. 9 ) gives an instruction to change the size of the three-dimensional image 52 to change the size of the three-dimensional image 52. Also, for example, the inspector 6 gives an instruction to rotate the three-dimensional image 52 to the receiving device 14, so that the three-dimensional image 52 rotates.
[0108] By displaying the first image area 81 including the plurality of two-dimensional images 50 and the second image area 82 including the three-dimensional image 52 side by side on the screen 16A, the plurality of two-dimensional images 50 and the three-dimensional image 52 can be compared. Fig.10 In the example shown, an example is shown in which the first image area 81 and the second image area 82 are displayed on the screen 16A in a state where they are side by side in the left-right direction of the first image 61, but for example, the first image area 81 and the second image area 82 may be displayed on the screen 16A in a state where they are side by side in the top-bottom direction of the first image 61, and the first image area 81 and the second image area 82 may be displayed on the screen 16A in a state where the first image area 81 is incorporated into a part of the second image area 82.
[0109] The two-dimensional image 50 is an example of a "two-dimensional image" involved in the technology of the present invention. The three-dimensional image 52 is an example of a "three-dimensional image" involved in the technology of the present invention. The first image area 81 is an example of a "first area" involved in the technology of the present invention. The second image area 82 is an example of a "second area" involved in the technology of the present invention.
[0110] As an example, Fig.11As shown, when the receiving device 14 receives an instruction to select any one of the plurality of two-dimensional images 50 included in the first image area 81, that is, a selection instruction, while the first image 61 is displayed on the screen 16A, the receiving device 14 outputs a selection instruction signal indicating the selection instruction to the processor 20. The selection instruction is an example of a “selection instruction” involved in the technology of the present invention.
[0111] When the selection instruction signal is input to the processor 20, the first image selection unit 44B selects a two-dimensional image 50 (hereinafter referred to as "focus two-dimensional image 50A") corresponding to the selection instruction from the plurality of two-dimensional images 50 included in the inspection auxiliary information 56 according to the selection instruction indicated by the selection instruction signal. The focus two-dimensional image 50A is an example of the "focus two-dimensional image" involved in the technology of the present invention.
[0112] As an example, Fig.12 As shown, the first pixel extraction unit 44C acquires the shooting position and shooting posture corresponding to the two-dimensional image 50A of interest from the inspection auxiliary information 56. Furthermore, the first pixel extraction unit 44C derives the viewpoint corresponding to the two-dimensional image 50A of interest based on the acquired shooting position and shooting posture. Then, the first pixel extraction unit 44C extracts pixels for including the three-dimensional image 52 in the second image area 82 at the derived viewpoint from the three-dimensional image 52 included in the inspection auxiliary information 56. Furthermore, when extracting pixels from the three-dimensional image 52, the first pixel extraction unit 44C extracts pixels for including the entire three-dimensional image 52 in a size that can be accommodated in the second image area 82.
[0113] The first image generator 44D includes a predetermined number of two-dimensional images 50 among the plurality of two-dimensional images 50 included in the inspection auxiliary information 56, and generates the first image region 81 in such a manner that the two-dimensional image 50A of interest is surrounded by the frame 90. Furthermore, the first image generator 44D generates the second image region 82 including the three-dimensional image 52 of such a size that the whole is accommodated in the second image region 82 at the viewpoint corresponding to the two-dimensional image 50A of interest, based on the pixels extracted by the first pixel extractor 44C. For example, the three-dimensional image 52 is included in the second image region 82 in such a manner that the viewpoint corresponding to the two-dimensional image 50A of interest is located at the center 82C of the second image region 82.
[0114] As an example, Fig.13 As shown, the first image generating unit 44D generates the first image 61 by synthesizing the generated first image region 81 and second image region 82 .
[0115] The first display control unit 44A outputs the first image data representing the first image 61 generated by the first image generating unit 44D to the display 16. Thus, the first image 61 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of two-dimensional images 50 among the plurality of two-dimensional images 50 are displayed on the screen 16A in a state of being included in the first image area 81, and the focus two-dimensional image 50A is displayed on the screen 16A in a state of being surrounded by the frame 90. By displaying the focus two-dimensional image 50A on the screen 16A in a state of being surrounded by the frame 90, the focus two-dimensional image 50A is in a state of being distinguishable from the remaining two-dimensional images 50 among the plurality of two-dimensional images 50.
[0116] Furthermore, the three-dimensional image 52 is displayed on the screen 16A at a viewpoint corresponding to the two-dimensional image 50A of interest and in a size that is entirely contained in the second image region 82. By displaying the three-dimensional image 52 on the screen 16A at a viewpoint corresponding to the two-dimensional image 50A of interest, a portion 54 (hereinafter referred to as "portion of interest 54A") corresponding to the two-dimensional image 50A of interest in the three-dimensional image 52 becomes a state that can be visually identified. The "portion 54" is an example of a "portion" involved in the technology of the present invention, and the portion of interest 54A in the three-dimensional image 52 is an example of a "portion of interest" involved in the technology of the present invention.
[0117] In addition, Fig.13 In the example shown, the two-dimensional image of interest 50A is displayed on the screen 16A in a state surrounded by a frame 90. However, the two-dimensional image of interest 50A may be displayed on the screen 16A in a state that can be distinguished from the remaining two-dimensional images 50 by other methods. For example, the two-dimensional image of interest 50A may be displayed on the screen 16A by a method of expressing it in a color different from the remaining two-dimensional images 50, by adding a pattern to the two-dimensional image of interest 50A, or by making the brightness of the two-dimensional image of interest 50A higher than that of the remaining two-dimensional images 50. With such an example, the two-dimensional image of interest 50A is also in a state that can be distinguished from the remaining two-dimensional images 50.
[0118] As an example, in Fig.14 , a state where the second image 62 is displayed on the screen 16A is shown. The second image 62 includes the first image area 81 and the third image area 83. As an example, the first image area 81 and the third image area 83 are displayed on the screen 16A in a state where they are arranged side by side in the left-right direction of the second image 62. The first image area 81 and the first image 61 (refer to Fig.10 ) is the same as the first image region 81 of the third image region 83. The third image region 83 includes the three-dimensional image 52. The third image region 83 is an example of the "second region" involved in the technology of the present invention.
[0119] The second display control unit 46A includes the plurality of two-dimensional images 50 in the first image region 81 based on the plurality of two-dimensional images 50 included in the auxiliary examination information 56. Furthermore, the second display control unit 46A includes the three-dimensional image 52 in the third image region 83 based on the three-dimensional image 52 included in the auxiliary examination information 56.
[0120] The three-dimensional image 52 is included in the third image area 83 in a state of being two-dimensionally imaged by rendering. For example, the inspector 6 receives the image from the receiving device 14 (refer to Fig. 9 ) gives an instruction to change the size of the three-dimensional image 52 to change the size of the three-dimensional image 52. Also, for example, the inspector 6 gives an instruction to rotate the three-dimensional image 52 to the receiving device 14, so that the three-dimensional image 52 rotates.
[0121] By displaying the first image area 81 including the plurality of two-dimensional images 50 and the third image area 83 including the three-dimensional image 52 side by side on the screen 16A, the plurality of two-dimensional images 50 and the three-dimensional image 52 can be compared.
[0122] In addition, Fig.14 , an example is shown in which the first image area 81 and the third image area 83 are displayed on the screen 16A in a state where they are side by side in the left-right direction of the second image 62, but for example, the first image area 81 and the third image area 83 may be displayed on the screen 16A in a state where they are side by side in the top-bottom direction of the second image 62, and the first image area 81 and the third image area 83 may be displayed on the screen 16A in a state where the first image area 81 is incorporated into a part of the third image area 83.
[0123] As an example, Fig.15 As shown, when the second image 62 is displayed on the screen 16A, when the receiving device 14 receives an indication of selecting any one of the multiple two-dimensional images 50 included in the first image area 81, that is, a selection indication, the receiving device 14 outputs a selection indication signal indicating the selection indication to the processor 20.
[0124] When the selection instruction signal is input to the processor 20 , the second image selection unit 46B selects the two-dimensional image of interest 50A corresponding to the selection instruction from the plurality of two-dimensional images 50 included in the auxiliary examination information 56 according to the selection instruction indicated by the selection instruction signal.
[0125] As an example, Fig.16 As shown, the second pixel extracting unit 46C extracts the focus portion 54A associated with the focus two-dimensional image 50A from the three-dimensional image 52 included in the inspection support information 56 .
[0126] The second image generator 46D includes a predetermined number of two-dimensional images 50 among the plurality of two-dimensional images 50 included in the inspection auxiliary information 56, and generates a first image region 81 in such a manner that the two-dimensional image of interest 50A is surrounded by a frame 90. Furthermore, the second image generator 46D generates a third image region 83 including the portion of interest 54A in the three-dimensional image 52 based on the portion of interest 54A extracted by the second pixel extractor 46C.
[0127] As an example, Fig.17 As shown, the second image generating unit 46D generates the second image 62 by synthesizing the generated first image region 81 and third image region 83 .
[0128] The second display control unit 46A outputs the second image data representing the second image 62 generated by the second image generating unit 46D to the display 16. As a result, the second image 62 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of the two-dimensional images 50 among the plurality of two-dimensional images 50 are displayed on the screen 16A in a state of being included in the first image region 81, and the focus two-dimensional image 50A is displayed on the screen 16A in a state of being surrounded by the frame 90.
[0129] Furthermore, the focus portion 54A in the three-dimensional image 52 is displayed on the screen 16A in an enlarged state. By displaying the focus portion 54A in the three-dimensional image 52 in an enlarged state on the screen 16A, the focus portion 54A in the three-dimensional image 52 becomes a state that can be visually confirmed. That is, by displaying the focus portion 54A in the three-dimensional image 52 in an enlarged state on the screen 16A, the focus portion 54A becomes a state that can be distinguished from the remaining parts 54 of the plurality of parts 54 forming the three-dimensional image 52. Thus, the focus portion 54A in the three-dimensional image 52 becomes a state that can be visually confirmed.
[0130] Furthermore, by displaying the two-dimensional image of interest 50A on the screen 16A while being surrounded by the frame 90, the two-dimensional image of interest 50A can be distinguished from the remaining two-dimensional images 50 in the plurality of two-dimensional images 50. Thus, the correspondence between the two-dimensional image of interest 50A and the focus portion 54A in the three-dimensional image 52 can be visually confirmed.
[0131] In addition, Fig.17In the example shown, the focus portion 54A in the three-dimensional image 52 is displayed on the screen 16A in an enlarged state. However, the entire three-dimensional image 52 may be displayed on the screen 16A in a size that fits in the third image area 83, and the focus portion 54A in the three-dimensional image 52 may be displayed on the screen 16A in a state that can be distinguished from the remaining portion 54 by other means. For example, the focus portion 54A may be displayed on the screen 16A in a manner that is different from the remaining portion 54, a manner that a pattern is given to the focus portion 54A, a manner that the focus portion 54A is surrounded by a frame, or a manner that the brightness of the pixels that form the outline of the focus portion 54A is higher than the surrounding pixels. Through such an example, the focus portion 54A in the three-dimensional image 52 is also in a state that can be visually determined.
[0132] As an example, in Fig.18 , a state where the third image 63 is displayed on the screen 16A is shown. The third image 63 includes the first image area 81 and the fourth image area 84. As an example, the first image area 81 and the fourth image area 84 are displayed on the screen 16A in a state where they are arranged side by side in the left-right direction of the third image 63. Fig.10 ) is the same as the first image area 81 of the second image area 84. The fourth image area 84 includes the three-dimensional image 52 and a plurality of position-specific images 92. Each position-specific image 92 is an image for determining a plurality of shooting positions at which shooting was performed to obtain a plurality of two-dimensional images 50, and indicates a shooting position corresponding to each two-dimensional image 50.
[0133] The third display control unit 48A includes the plurality of two-dimensional images 50 in the first image region 81 based on the plurality of two-dimensional images 50 included in the auxiliary examination information 56. Furthermore, the third display control unit 48A includes the three-dimensional image 52 in the fourth image region 84 based on the three-dimensional image 52 included in the auxiliary examination information 56.
[0134] The three-dimensional image 52 is included in the fourth image area 84 in a state of being two-dimensionally imaged by rendering. For example, the inspector 6 receives the image from the receiving device 14 (refer to Fig. 9 ) gives an instruction to change the size of the three-dimensional image 52 to change the size of the three-dimensional image 52. Also, for example, the inspector 6 gives an instruction to rotate the three-dimensional image 52 to the receiving device 14, so that the three-dimensional image 52 rotates.
[0135] Furthermore, the third display control unit 48A includes a plurality of position-specific images 92 in the fourth image region 84 according to each imaging position included in the inspection auxiliary information 56. As an example, each position-specific image 92 is displayed in a plate shape. The plurality of position-specific images 92 are included in the fourth image region 84 in a state that can be compared with the three-dimensional image 52. Specifically, the plurality of position-specific images 92 are arranged around the three-dimensional image 52 and included in the fourth image region 84 in a state that they are opposed to the three-dimensional image 52. That is, the fourth image region 84 includes an image showing a style in which the plurality of position-specific images 92 are opposed to the three-dimensional image 52.
[0136] By displaying the first image area 81 and the fourth image area 84 side by side on the screen 16A, the plurality of two-dimensional images 50 and the three-dimensional image 52 can be compared, and the plurality of two-dimensional images 50 and the plurality of position specific images 92 can be compared.
[0137] In addition, Fig.18 , an example is shown in which the first image area 81 and the fourth image area 84 are displayed on the screen 16A in a state where they are side by side in the left-right direction of the third image 63, but for example, the first image area 81 and the fourth image area 84 may be displayed on the screen 16A in a state where they are side by side in the top-bottom direction of the third image 63, and the first image area 81 and the fourth image area 84 may be displayed on the screen 16A in a state where the first image area 81 is incorporated into a part of the fourth image area 84.
[0138] The first image region 81 is an example of the “first region” and “third region” involved in the technology of the present invention. The fourth image region 84 is an example of the “second region” and “fourth region” involved in the technology of the present invention. The position specific image 92 is an example of the “position specific image” involved in the technology of the present invention.
[0139] As an example, Fig.19 As shown in FIG. 1 , when the receiving device 14 receives an instruction to select any one of the plurality of position specific images 92 included in the fourth image area 84, that is, a selection instruction, while the third image 63 is displayed on the screen 16A, the receiving device 14 outputs a selection instruction signal indicating the selection instruction to the processor 20. In addition, hereinafter, the selected position specific image 92 among the plurality of position specific images 92 is referred to as a "position specific image of interest 92A".
[0140] When the selection instruction signal is input to the processor 20, the third image selection unit 48B selects a shooting position corresponding to the focus position specific image 92A from the multiple shooting positions included in the inspection auxiliary information 56 (hereinafter referred to as the "focus shooting position") according to the selection instruction represented by the selection instruction signal. Then, the third image selection unit 48B selects the focus two-dimensional image 50A corresponding to the focus shooting position from the multiple two-dimensional images 50 included in the inspection auxiliary information 56. The focus two-dimensional image 50A is a two-dimensional image 50 obtained by shooting from the focus shooting position. The focus position specific image 92A is an example of the "focus position specific image" involved in the technology of the present invention. The focus shooting position is an example of the "focus shooting position" involved in the technology of the present invention.
[0141] As an example, Fig. 20 As shown, the third image generator 48C generates a fourth image region 84 including the three-dimensional image 52 based on the three-dimensional image 52 included in the inspection auxiliary information 56. Furthermore, the third image generator 48C includes a plurality of position specific images 92 in the fourth image region 84 based on the imaging position and imaging posture included in the inspection auxiliary information 56. The plurality of position specific images 92 are included in the fourth image region 84 in a state of being arranged around the three-dimensional image 52 and facing the three-dimensional image 52. Each position specific image 92 is arranged at a position corresponding to each imaging position in a posture corresponding to each imaging posture.
[0142] Furthermore, the third image generating unit 48C includes the focus position specific image 92A corresponding to the focus shooting position among the plurality of position specific images 92 in the fourth image region 84. The focus position specific image 92A is included in the fourth image region 84 in a state that it can be distinguished from the remaining position specific images 92 among the plurality of position specific images 92. Fig. 20 In the example shown, the focus position specific image 92A is displayed in a color different from the remaining position specific images 92 as an example of a state in which the focus position specific image 92A can be distinguished from the remaining position specific images 92. In addition, the focus position specific image 92A may be surrounded by a frame, or a pattern may be added to the focus position specific image 92A. In addition, the brightness of the pixels forming the outline of the focus position specific image 92A may be higher than that of the surrounding pixels.
[0143] The third image generator 48C generates the first image region 81 including a predetermined number of the two-dimensional images 50 among the plurality of two-dimensional images 50 included in the auxiliary examination information 56 , and such that the two-dimensional image 50A of interest is surrounded by the frame 90 .
[0144] As an example, Fig.21As shown, the third image generating unit 48C generates the third image 63 by synthesizing the generated first image region 81 and fourth image region 84 .
[0145] The third display control unit 48A outputs the third image data representing the third image 63 generated by the third image generating unit 48C to the display 16. As a result, the third image 63 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of the two-dimensional images 50 among the plurality of two-dimensional images 50 are displayed on the screen 16A in a state of being included in the first image area 81, and the focus two-dimensional image 50A is displayed on the screen 16A in a state of being surrounded by the frame 90. In addition, the three-dimensional image 52 is displayed on the screen 16A.
[0146] Furthermore, the plurality of position specific images 92 are arranged around the three-dimensional image 52 and displayed on the screen 16A in a state facing the three-dimensional image 52, and the focus position specific image 92A is displayed on the screen 16A in a state that it can be distinguished from the remaining position specific images 92 in the plurality of position specific images 92. By displaying the focus position specific image 92A on the screen 16A in a state that it can be distinguished from the remaining position specific images 92, the focus portion 54A corresponding to the focus position specific image 92A in the three-dimensional image 52 is in a state that can be visually identified. That is, the shooting position and the shooting posture are determined by the focus position specific image 92A, and the correspondence relationship between the determined shooting position and the shooting posture and the focus portion 54A in the three-dimensional image 52 is in a state that can be visually identified. As a result, the focus portion 54A in the three-dimensional image 52 is in a state that can be visually identified.
[0147] Furthermore, by displaying the two-dimensional image of interest 50A on the screen 16A while being surrounded by the frame 90, the two-dimensional image of interest 50A can be distinguished from the remaining two-dimensional images 50 in the plurality of two-dimensional images 50. Thus, the correspondence between the two-dimensional image of interest 50A and the focus portion 54A in the three-dimensional image 52 can be visually confirmed.
[0148] In addition, Fig.21 In the example shown, the entire three-dimensional image 52 is displayed on the screen 16A in a size that fits in the fourth image area 84. However, the focus portion 54A in the three-dimensional image 52 may be displayed on the screen 16A in an enlarged state. With this example, the focus portion 54A in the three-dimensional image 52 can also be visually identified.
[0149] Next, refer to Figure 22 to Figure 26 The operation of the inspection support device 10 according to the first embodiment will be described.
[0150] First, refer to Fig. 22 , an example of the flow of the inspection support information generation process performed by the processor 20 of the inspection support device 10 is described.
[0151] exist Fig. 22 In the inspection auxiliary information generation process shown in FIG. 1 , first, in step ST10, the acquisition unit 32 (refer to Figure 6 ) acquires the two-dimensional image 50 based on each image data received by the inspection auxiliary device 10. And, the acquisition unit 32 acquires the shooting position corresponding to each two-dimensional image 50 based on each position data received by the inspection auxiliary device 10. In addition, the acquisition unit 32 acquires the shooting posture corresponding to each two-dimensional image 50 based on each posture data received by the inspection auxiliary device 10. After executing the processing of step ST10, the inspection auxiliary information generation processing enters step ST12.
[0152] In step ST12, the three-dimensional image generating unit 34 (refer to Figure 6 ) Based on the plurality of two-dimensional images 50 acquired in step ST10, a three-dimensional image 52 is generated to display the object 4. After the process of step ST12 is executed, the inspection auxiliary information generation process proceeds to step ST14.
[0153] In step ST14, the auxiliary information generating unit 36 (refer to Figure 7 ) generates information that establishes a correspondence between each two-dimensional image 50 acquired in step ST10, the shooting position corresponding to each two-dimensional image 50, the shooting posture corresponding to each two-dimensional image 50, and the portion 54 corresponding to each two-dimensional image 50, that is, inspection auxiliary information 56. After the processing of step ST14 is executed, the inspection auxiliary information generation processing is terminated.
[0154] Next, refer to Figure 23 to Figure 26 , an example of the flow of the inspection support processing performed by the processor 20 of the inspection support device 10 is described. Fig.23 , an example of the process of mode setting processing in inspection auxiliary processing is described.
[0155] exist Fig.23 In the mode setting process shown in FIG. 1 , first, in step ST20, the operation mode setting unit 42 (see FIG. Fig. 9 ) determines whether the mode setting instruction signal is input to the processor 20. In step ST20, when the mode setting instruction signal is input to the processor 20, it is determined that it is approved, and the inspection auxiliary process enters step ST22. In step ST20, when the mode setting instruction signal is not input to the processor 20, it is determined that it is rejected, and the mode setting process enters step ST32.
[0156] In step ST22, the operation mode setting unit 42 determines whether the mode setting instruction signal input to the processor 20 in step ST20 is the first mode setting signal. In step ST22, when the mode setting instruction signal is the first mode setting signal, the determination is approved, and the inspection auxiliary process proceeds to step ST24. In step ST22, when the mode setting instruction signal is not the first mode setting signal, the determination is rejected, and the mode setting process proceeds to step ST26.
[0157] In step ST24, the operation mode setting unit 42 sets the first mode as the operation mode of the inspection support device 10. Thus, the first mode process is executed. After the process of step ST24 is executed, the mode setting process proceeds to step ST32.
[0158] In step ST26, the operation mode setting unit 42 determines whether the mode setting instruction signal input to the processor 20 in step ST20 is the second mode setting signal. In step ST26, when the mode setting instruction signal is the second mode setting signal, the determination is approved, and the inspection auxiliary process proceeds to step ST28. In step ST26, when the mode setting instruction signal is not the second mode setting signal, the determination is rejected, and the mode setting process proceeds to step ST30.
[0159] In step ST28, the operation mode setting unit 42 sets the second mode as the operation mode of the inspection support device 10. Thus, the second mode process is executed. After the process of step ST28 is executed, the mode setting process proceeds to step ST32.
[0160] In step ST30, the operation mode setting unit 42 sets the third mode as the operation mode of the inspection support device 10. Thus, the third mode process is executed. After the process of step ST30 is executed, the mode setting process proceeds to step ST32.
[0161] In step ST32, the processor 20 determines whether the mode setting process end condition (hereinafter referred to as "mode setting process end condition") is satisfied. As an example of the mode setting process end condition, there can be cited a condition in which the end instruction signal from the receiving device 14 is input into the processor 20 by receiving the end instruction from the inspector 6 through the receiving device 14. In step ST32, when the mode setting process end condition is not satisfied, it is determined that it is rejected, and the mode setting process proceeds to step ST20. In step ST32, when the mode setting process end condition is satisfied, it is determined that it is approved, and the inspection auxiliary process including the mode setting process is terminated.
[0162] Next, refer to Fig.24 , an example of the process of the first mode processing in the inspection auxiliary processing is described.
[0163] exist Fig.24 In the first mode process shown in FIG. 1 , first, in step ST40, the first display control unit 44A (see FIG. 1 ) Fig.10 ) The first image 61 is displayed on the screen 16A. After the process of step ST40 is executed, the first mode process proceeds to step ST42.
[0164] In step ST42, the first image selection unit 44B (reference Fig.11 ) determines whether an instruction to select any one of the plurality of two-dimensional images 50, i.e., a selection instruction signal indicating a selection instruction, is input into the processor 20. In step ST42, when the selection instruction signal indicating a selection instruction is input into the processor 20, it is determined that it is approved, and the first mode processing proceeds to step ST44. In step ST42, when the selection instruction signal indicating a selection instruction is not input into the processor 20, it is determined that it is rejected, and the first mode processing proceeds to step ST52.
[0165] In step ST44, the first image selection unit 44B selects the two-dimensional image 50A of interest corresponding to the selection instruction indicated by the selection instruction signal from the plurality of two-dimensional images 50 included in the examination support information 56. After the process of step ST44 is executed, the first mode process proceeds to step ST46.
[0166] In step ST46, the first pixel extraction unit 44C (reference Fig.12 ) The first pixel extraction unit 44C obtains the shooting position and shooting posture corresponding to the two-dimensional image of interest 50A selected in step ST44 from the inspection auxiliary information 56. And, the first pixel extraction unit 44C derives the viewpoint corresponding to the two-dimensional image of interest 50A based on the acquired shooting position and shooting posture. Then, the first pixel extraction unit 44C extracts pixels for including the three-dimensional image 52 in the second image area 82 at the derived viewpoint from the three-dimensional image 52 included in the inspection auxiliary information 56. After executing the process of step ST46, the first mode process enters step ST48.
[0167] In step ST48, the first image generating unit 44D (reference Fig.12 ) includes a predetermined number of two-dimensional images 50 among the plurality of two-dimensional images 50 included in the inspection auxiliary information 56, and generates the first image region 81 in such a manner that the two-dimensional image 50A of interest is surrounded by the frame 90. Furthermore, the first image generating unit 44D generates the second image region 82 including the three-dimensional image 52 having a size that is entirely accommodated in the second image region 82 at the viewpoint corresponding to the two-dimensional image 50A of interest, based on the pixels extracted in step ST46. Then, the first image generating unit 44D (refer to Fig.13) The first image 61 is generated by synthesizing the generated first image region 81 and the second image region 82. After the process of step ST48 is executed, the first mode process proceeds to step ST50.
[0168] In step ST50, the first display control unit 44A (refer to Fig.13 ) outputs the first image data representing the first image 61 generated in step ST48 to the display 16. Thus, the first image 61 is displayed on the screen 16A of the display 16. After the process of step ST50 is executed, the first mode process proceeds to step ST52.
[0169] In step ST52, the processor 20 determines whether the condition for terminating the first mode processing (hereinafter referred to as "the first mode processing termination condition") is satisfied. As an example of the first mode processing termination condition, there can be cited a condition that an end instruction signal from the receiving device 14 is input into the processor 20 by receiving an end instruction from the inspector 6 by the receiving device 14, or a condition that a mode setting instruction signal indicating an instruction to set an operation mode different from the first mode is input into the processor 20. In step ST52, when the first mode processing termination condition is not satisfied, it is determined that it is rejected, and the first mode processing enters step ST42. In step ST52, when the first mode processing termination condition is satisfied, it is determined that it is approved, and the first mode processing is terminated.
[0170] Next, refer to Fig.25 , an example of the process of the second mode processing in the inspection auxiliary processing is described.
[0171] exist Fig.25 In the second mode processing shown in FIG. 1 , first, in step ST60, the second display control unit 46A (see FIG. 1 ) Fig.14 ) The second image 62 is displayed on the screen 16A. After the process of step ST60 is executed, the second mode process proceeds to step ST62.
[0172] In step ST62, the second image selection unit 46B (reference Fig.15 ) determines whether a selection instruction signal indicating the selection of any one of the plurality of two-dimensional images 50 is input to the processor 20. In step ST62, when the selection instruction signal indicating the selection instruction is input to the processor 20, it is determined that it is approved, and the second mode processing proceeds to step ST64. In step ST62, when the selection instruction signal indicating the selection instruction is not input to the processor 20, it is determined that it is rejected, and the second mode processing proceeds to step ST72.
[0173] In step ST64, the second image selection unit 46B selects the two-dimensional image 50A of interest corresponding to the selection instruction indicated by the selection instruction signal from the plurality of two-dimensional images 50 included in the examination support information 56. After executing the process of step ST64, the second mode process proceeds to step ST66.
[0174] In step ST66, the second pixel extraction unit 46C (reference Fig.16 ) extracts the focus portion 54A associated with the focus two-dimensional image 50A from the three-dimensional image 52 included in the inspection auxiliary information 56. After executing the process of step ST66, the second mode process proceeds to step ST68.
[0175] In step ST68, the second image generating unit 46D (reference Fig.16 ) includes a predetermined number of two-dimensional images 50 among the plurality of two-dimensional images 50 included in the inspection auxiliary information 56, and generates a first image region 81 in such a manner that the two-dimensional image 50A of interest is surrounded by a frame 90. Furthermore, the second image generating unit 46D generates a third image region 83 including the portion of interest 54A in the three-dimensional image 52 based on the portion of interest 54A extracted in step ST66. Then, the second image generating unit 46D (refer to Fig.17 ) The second image 62 is generated by synthesizing the generated first image region 81 and the third image region 83. After the process of step ST68 is executed, the second mode process proceeds to step ST70.
[0176] In step ST70, the second display control unit 46A (refer to Fig.17 ) outputs the second image data representing the second image 62 generated in step ST68 to the display 16. Thus, the second image 62 is displayed on the screen 16A of the display 16. After the process of step ST70 is executed, the second mode process proceeds to step ST72.
[0177] In step ST72, the processor 20 determines whether the condition for terminating the second mode processing (hereinafter referred to as "second mode processing termination condition") is satisfied. As an example of the second mode processing termination condition, there can be cited a condition in which an end instruction signal from the receiving device 14 is input into the processor 20 by receiving an end instruction from the inspector 6 by the receiving device 14, or a condition in which a mode setting instruction signal indicating an instruction to set an operation mode different from the second mode is input into the processor 20. In step ST72, when the second mode processing termination condition is not satisfied, it is determined that it is rejected, and the second mode processing enters step ST62. In step ST72, when the second mode processing termination condition is satisfied, it is determined that it is approved, and the second mode processing is terminated.
[0178] Next, refer to Fig.26 , an example of the process of the third mode processing in the inspection auxiliary processing is described.
[0179] exist Fig.26 In the third mode processing shown in FIG. 1 , first, in step ST80, the third display control unit 48A (see FIG. 1 ) Fig.18 ) The third image 63 is displayed on the screen 16A. After the processing of step ST80 is executed, the third mode processing proceeds to step ST82.
[0180] In step ST82, the third image selection unit 48B (reference Fig.19 ) determines whether a selection instruction signal indicating an instruction to select the position specific image 92A of interest among the plurality of position specific images 92, i.e., a selection instruction, is input to the processor 20. In step ST82, when the selection instruction signal indicating the selection instruction is input to the processor 20, it is determined that it is approved, and the third mode processing proceeds to step ST84. In step ST82, when the selection instruction signal indicating the selection instruction is not input to the processor 20, it is determined that it is rejected, and the third mode processing proceeds to step ST90.
[0181] In step ST84, the third image selection unit 48B selects the imaging position of interest corresponding to the imaging position specific image 92A from the plurality of imaging positions included in the inspection auxiliary information 56 according to the selection instruction indicated by the selection instruction signal. Then, the third image selection unit 48B selects the two-dimensional image of interest 50A corresponding to the imaging position of interest from the plurality of two-dimensional images 50 included in the inspection auxiliary information 56. After the processing of step ST84 is executed, the third mode processing proceeds to step ST86.
[0182] In step ST86, the third image generating unit 48C (refer to Fig. 20 ) generates a fourth image region 84 including the three-dimensional image 52 based on the three-dimensional image 52 included in the inspection auxiliary information 56. Furthermore, the third image generating unit 48C includes a plurality of position specific images 92 in the fourth image region 84 based on the shooting positions and shooting postures included in the inspection auxiliary information 56. Furthermore, the third image generating unit 48C includes the focus position specific image 92A corresponding to the focus shooting position selected in step ST84 in the fourth image region 84. Then, the third image generating unit 48C (refer to Fig.21 ) The third image 63 is generated by synthesizing the generated first image region 81 and fourth image region 84. After the process of step ST86 is executed, the third mode process proceeds to step ST88.
[0183] In step ST88, the third display control unit 48A (reference Fig.21) outputs the third image data representing the third image 63 generated in step ST86 to the display 16. Thus, the third image 63 is displayed on the screen 16A of the display 16. After the process of step ST88 is executed, the third mode process proceeds to step ST90.
[0184] In step ST90, the processor 20 determines whether the condition for terminating the third mode processing (hereinafter referred to as "the third mode processing termination condition") is satisfied. As an example of the third mode processing termination condition, there can be cited a condition in which the termination instruction signal from the receiving device 14 is input into the processor 20 by receiving the termination instruction from the inspector 6 by the receiving device 14, or a condition in which a mode setting instruction signal indicating an instruction to set an action mode different from the third mode is input into the processor 20. In step ST90, when the third mode processing termination condition is not satisfied, it is determined that it is rejected, and the third mode processing enters step ST82. In step ST90, when the third mode processing termination condition is satisfied, it is determined that it is approved, and the third mode processing is terminated. In addition, the inspection assisting method described as the role of the above-mentioned inspection assisting device 10 is an example of the "image processing method" involved in the technology of the present invention.
[0185] As described above, in the inspection support device 10 according to the first embodiment, the processor 20 performs the following processing: the plurality of two-dimensional images 50 for generating a three-dimensional image 52 showing the object 4 in the real space and the plurality of two-dimensional images 50 and the three-dimensional image 52 are displayed on the screen 16A in a state where they can be compared (see Fig.10 , Fig.14 and Fig.18 ). Furthermore, the processor 20 selects the focus two-dimensional image 50A (reference Fig.11 , Fig.15 and Fig.19 Then, the processor 20 displays the focus portion 54A corresponding to the focus two-dimensional image 50A among the plurality of portions 54 on the screen 16A in a state that can be visually confirmed (refer to Fig.13 , Fig.17 and Fig.21 ). Therefore, the correspondence between each two-dimensional image 50 and the region corresponding to each two-dimensional image 50 in the object 4 can be visually grasped.
[0186] Furthermore, the state in which the plurality of two-dimensional images 50 and the three-dimensional image 52 can be compared refers to a state in which the first image region 81 including the plurality of two-dimensional images 50 and the image region including the three-dimensional image 52 (i.e., the second image region 82, the third image region 83 or the fourth image region 84) are arranged side by side (refer to Fig.10 , Fig.14 and Fig.18 ). Therefore, the plurality of two-dimensional images 50 and the three-dimensional image 52 can be visually compared.
[0187] Furthermore, the state in which the focus portion 54A can be visually determined includes a state in which the focus portion 54A can be distinguished from the remaining portions 54 of the plurality of portions 54 (refer to Fig.17 and Fig.21 ). Therefore, for example, compared with a case where the focus portion 54A is not distinguished from the remaining portion 54, the visual recognition of the focus portion 54A can be improved.
[0188] Furthermore, the state in which the focus portion 54A can be visually determined includes a state in which the focus two-dimensional image 50A can be distinguished from the remaining two-dimensional images 50 in the plurality of two-dimensional images 50 (refer to Fig.13 , Fig.17 and Fig.21 ). Therefore, for example, compared with a case where the focused two-dimensional image 50A is not distinguished from the remaining two-dimensional images 50, the visual recognition of the focused portion 54A can be improved. In addition, when the plurality of two-dimensional images 50 and the three-dimensional image 52 are displayed on the screen 16A in a state where they can be compared, and the focused portion 54A is displayed in a state where it can be distinguished from the remaining portion 54 (refer to Fig.17 and Fig.21 ), the correspondence between the focus two-dimensional image 50A and the focus portion 54A can be visually determined.
[0189] Furthermore, the processor 20 displays a plurality of position specific images 92 that can identify a plurality of shooting positions at which the images were taken in order to obtain the plurality of two-dimensional images 50 on the screen 16A in a state that can be compared with the three-dimensional image 52 (see FIG. 16A ). Fig.18 ). And, according to the selection instruction, the processor 20 selects, from the plurality of shooting positions, a shooting position corresponding to the focus position specific image 92A selected from the plurality of position specific images 92 as the focus shooting position (reference Fig.19 Then, the processor 20 selects a two-dimensional image 50 obtained by photographing from the photographing position of interest from among the plurality of two-dimensional images 50 as the two-dimensional image of interest 50A (refer to Fig.19 ). Therefore, by selecting the focus position identification image 92A from the plurality of position identification images 92 , it is possible to select the focus two-dimensional image 50A from the plurality of two-dimensional images 50 .
[0190] Furthermore, the state in which the plurality of position specific images 92 can be compared with the three-dimensional image 52 includes a state in which the plurality of position specific images 92 and the three-dimensional image 52 are opposed to each other (see Fig.19). Therefore, based on the state in which the plurality of position specific images 92 and the three-dimensional image 52 are opposed to each other, it is possible to select the focus position specific image 92A corresponding to the focus portion 54A in the three-dimensional image 52 from among the plurality of position specific images 92 .
[0191] Furthermore, the state in which the plurality of two-dimensional images 50 can be compared with the three-dimensional image 52 refers to a state in which the first image area 81 and the fourth image area 84 (i.e., the image area including the image showing the pattern in which the plurality of position-specific images 92 and the three-dimensional image 52 are placed opposite to each other) of the plurality of two-dimensional images 50 are arranged side by side (refer to Fig.18 and Fig.19 ). Therefore, the plurality of two-dimensional images 50, the plurality of portions 54 in the three-dimensional image 52, and the plurality of position-specific images 92 can be visually compared.
[0192] Furthermore, the state in which the focus portion 54A can be visually determined includes a state in which the focus position specific image 92A can be distinguished from the remaining position specific images 92 among the plurality of position specific images 92. Therefore, for example, compared with a case in which the focus position specific image 92A is not distinguished from the remaining position specific images 92, the visual recognition of the focus portion 54A can be improved.
[0193] Furthermore, the inspection auxiliary device 10 (reference Fig. 9 ) has: an action mode for displaying a plurality of two-dimensional images 50 and a three-dimensional image 52 on the screen 16A in a state where they can be compared (for example, a first mode and a second mode); and an action mode for displaying a plurality of position-specific images 92 on the screen 16A in a state where they can be compared with the three-dimensional image 52 (that is, a third mode). Then, the processor 20 sets the action mode according to a given setting instruction. Therefore, for example, the screen 16A can be selectively switched to a state where the plurality of two-dimensional images 50 and the three-dimensional image 52 can be compared and a state where the plurality of position-specific images 92 and the three-dimensional image 52 can be compared according to the focus portion 54A.
[0194] Furthermore, in the first mode, the three-dimensional image 52 is displayed on the screen 16A at the viewpoint corresponding to the focused two-dimensional image 50A (see FIG. Fig.13 ). Therefore, according to the viewpoint corresponding to the two-dimensional image 50A of interest, it is possible to visually determine the portion of interest 54A corresponding to the two-dimensional image 50A of interest among the plurality of portions 54 .
[0195] In addition, the inspection support device 10 according to the first embodiment has the first mode, the second mode, and the third mode, but any one of the first mode, the second mode, and the third mode may be omitted. Furthermore, the inspection support device 10 according to the first embodiment may have only one of the first mode, the second mode, and the third mode.
[0196] [Second embodiment]
[0197] Next, a second embodiment of the present invention will be described.
[0198] In the second embodiment, the structure of the inspection support device 10 is changed as follows compared with the first embodiment.
[0199] As an example, Fig. 27 As shown in FIG. 1 , the processor 20 operates as the fourth display control unit 94A. The fourth display control unit 94A displays the fourth image 64 on the screen 16A. The fourth image 64 includes the first image area 81 and the fifth image area 85. As an example, the first image area 81 and the fifth image area 85 are displayed on the screen 16A in a state where they are arranged side by side in the left-right direction of the fifth image. The first image area 81 and the first image 61 (reference Fig.10 ) is the same as the first image region 81. The fifth image region 85 includes the three-dimensional image 52.
[0200] The fourth display control unit 94A includes the plurality of two-dimensional images 50 in the first image region 81 based on the plurality of two-dimensional images 50 included in the auxiliary examination information 56. Furthermore, the fourth display control unit 94A includes the three-dimensional image 52 in the fifth image region 85 based on the three-dimensional image 52 included in the auxiliary examination information 56.
[0201] The three-dimensional image 52 is included in the fifth image region 85 in a state of being two-dimensionally imaged by rendering. For example, the inspector 6 receives the image from the receiving device 14 (refer to Fig. 9 ) gives an instruction to change the size of the three-dimensional image 52 to change the size of the three-dimensional image 52. Also, for example, the inspector 6 gives an instruction to rotate the three-dimensional image 52 to the receiving device 14, so that the three-dimensional image 52 rotates.
[0202] By displaying the first image region 81 including the plurality of two-dimensional images 50 and the fifth image region 85 including the three-dimensional image 52 side by side on the screen 16A, the plurality of two-dimensional images 50 and the three-dimensional image 52 can be compared.
[0203] In addition, Fig. 27 , an example is shown in which the first image area 81 and the fifth image area 85 are displayed on the screen 16A in a state where they are side by side in the left-right direction of the fourth image 64, but for example, the first image area 81 and the fifth image area 85 may be displayed on the screen 16A in a state where they are side by side in the top-bottom direction of the fourth image 64, and the first image area 81 and the fifth image area 85 may be displayed on the screen 16A in a state where the first image area 81 is incorporated into a part of the fifth image area 85.
[0204] The first image region 81 is an example of the “first region” involved in the technique of the present invention. The fifth image region 85 is an example of the “second region” involved in the technique of the present invention.
[0205] As an example, Fig.28 As shown, when the fourth image 64 is displayed on the screen 16A, when the receiving device 14 receives an instruction to select any part 54 of the three-dimensional image 52 included in the fifth image area 85, that is, a selection instruction, the receiving device 14 outputs a selection instruction signal representing the selection instruction to the processor 20.
[0206] The processor 20 operates as the fourth image selection unit 94B. When the selection instruction signal is input to the processor 20, the fourth image selection unit 94B selects the portion 54 corresponding to the selection instruction (i.e., the focus portion 54A) from the plurality of portions 54 included in the inspection auxiliary information 56 according to the selection instruction indicated by the selection instruction signal. Furthermore, the fourth image selection unit 94B selects the two-dimensional image 50 corresponding to the focus portion 54A (i.e., the focus two-dimensional image 50A) from the plurality of two-dimensional images 50 included in the inspection auxiliary information 56.
[0207] As an example, Fig.29 As shown, the processor 20 operates as a fourth pixel extraction unit 94C and a fourth image generation unit 94D. The fourth pixel extraction unit 94C extracts the focus portion 54A corresponding to the focus two-dimensional image 50A from the three-dimensional image 52 included in the inspection support information 56 .
[0208] The fourth image generator 94D includes a predetermined number of two-dimensional images 50 among the plurality of two-dimensional images 50 included in the inspection auxiliary information 56, and generates the first image region 81 in such a manner that the two-dimensional image 50A of interest is surrounded by the frame 90. Furthermore, the fourth image generator 94D generates a fifth image region 85 including the three-dimensional image 52 based on the three-dimensional image 52 included in the inspection auxiliary information 56. Furthermore, the fourth image generator 94D includes the portion of interest 54A extracted by the fourth pixel extractor 94C in the fifth image region 85 in a state that it can be distinguished from the remaining portion 54. For example, the fourth image generator 94D includes the portion of interest 54A in the fifth image region 85 in such a manner that it is expressed in a color different from that of the remaining portion 54.
[0209] As an example, Fig.30 As shown, the fourth image generating unit 94D generates the fourth image 64 by synthesizing the generated first image region 81 and fifth image region 85 .
[0210] The fourth display control unit 94A outputs the fourth image data representing the fourth image 64 generated by the fourth image generation unit 94D to the display 16. Thus, the fourth image 64 is displayed on the screen 16A of the display 16. Specifically, a predetermined number of the two-dimensional images 50 among the plurality of two-dimensional images 50 are displayed on the screen 16A in a state of being included in the first image area 81, and the two-dimensional image of interest 50A is displayed on the screen 16A in a state of being surrounded by the frame 90. By displaying the two-dimensional image of interest 50A on the screen 16A in a state of being surrounded by the frame 90, the two-dimensional image of interest 50A is in a state of being distinguishable from the remaining two-dimensional images 50 among the plurality of two-dimensional images 50.
[0211] Then, the focus portion 54A included in the three-dimensional image 52 is displayed on the screen 16A in a state distinguishable from the remaining portion 54. By displaying the focus portion 54A on the screen 16A in a state distinguishable from the remaining portion 54, the focus portion 54A in the three-dimensional image 52 can be visually identified.
[0212] In addition, Fig.30 In the example shown, the focus portion 54A is displayed in a color different from that of the remaining portion 54. However, the focus portion 54A may be displayed on the screen 16A in a state that it can be distinguished from the remaining portion 54 by other means. For example, the focus portion 54A may be displayed on the screen 16A in a manner of being surrounded by a frame, a manner of adding a pattern to the focus portion 54A, or a manner of having a higher brightness than the remaining portion 54. In this example, the focus portion 54A is also in a state that it can be distinguished from the remaining portion 54.
[0213] Next, refer to Fig.31 The operation of the inspection support device 10 according to the second embodiment will be described. Fig.31 , an example of the flow of the inspection support process according to the second embodiment is shown.
[0214] exist Fig.31 In the inspection support process shown in FIG. 1 , first, in step ST100, the fourth display control unit 94A (see FIG. 1 ) displays the Fig. 27 ) The fourth image 64 is displayed on the screen 16A. After the process of step ST100 is executed, the inspection auxiliary process proceeds to step ST102.
[0215] In step ST102, the fourth image selection unit 94B (reference Fig.28) determines whether an instruction to select an arbitrary part 54 in the three-dimensional image 52, that is, a selection instruction signal indicating a selection instruction, is input to the processor 20. In step ST102, when the selection instruction signal indicating a selection instruction is input to the processor 20, it is determined that it is approved, and the inspection auxiliary process enters step ST104. In step ST102, when the selection instruction signal indicating a selection instruction is not input to the processor 20, it is determined that it is rejected, and the inspection auxiliary process enters step ST112.
[0216] In step ST104, the fourth image selection unit 94B selects the focus portion 54A corresponding to the selection instruction indicated by the selection instruction signal from the plurality of portions 54 included in the inspection auxiliary information 56. Furthermore, the fourth image selection unit 94B selects the focus two-dimensional image 50A corresponding to the focus portion 54A from the plurality of two-dimensional images 50 included in the inspection auxiliary information 56. After the process of step ST104 is executed, the inspection auxiliary process proceeds to step ST106.
[0217] In step ST106, the fourth pixel extraction unit 94C (reference Fig.29 ) extracts the focus portion 54A corresponding to the focus two-dimensional image 50A from the three-dimensional image 52 included in the inspection auxiliary information 56. After the processing of step ST106 is performed, the inspection auxiliary processing proceeds to step ST108.
[0218] In step ST108, the fourth image generating unit 94D (reference Fig.30 ) includes a predetermined number of two-dimensional images 50 among the plurality of two-dimensional images 50 included in the inspection auxiliary information 56, and generates the first image region 81 in such a manner that the two-dimensional image 50A of interest is surrounded by the frame 90. Furthermore, the fourth image generating unit 94D generates a fifth image region 85 including the three-dimensional image 52 based on the three-dimensional image 52 included in the inspection auxiliary information 56. Furthermore, the fourth image generating unit 94D includes the portion of interest 54A extracted in step ST106 in the fifth image region 85 in a state that it can be distinguished from the remaining portion 54. Then, the fourth image generating unit 94D generates the fourth image 64 by synthesizing the generated first image region 81 and the fifth image region 85. After executing the process of step ST108, the inspection auxiliary process proceeds to step ST110.
[0219] In step ST110, the fourth display control unit 94A (reference Fig.30 ) outputs the fourth image data representing the fourth image 64 generated in step ST108 to the display 16. Thus, the fourth image 64 is displayed on the screen 16A of the display 16. After the process of step ST110 is executed, the inspection auxiliary process proceeds to step ST112.
[0220] In step ST112, the processor 20 determines whether the condition for terminating the inspection auxiliary process (hereinafter referred to as "termination condition") is satisfied. As an example of the termination condition, there can be cited a condition that the termination instruction signal from the receiving device 14 is input to the processor 20 by receiving the termination instruction from the inspector 6 through the receiving device 14. In step ST112, when the termination condition is not satisfied, it is determined that it is rejected, and the inspection auxiliary process proceeds to step ST102. In step ST112, when the termination condition is satisfied, it is determined that it is approved, and the inspection auxiliary process is terminated.
[0221] As described above, in the inspection support device 10 according to the second embodiment, the processor 20 performs the following processing: the plurality of two-dimensional images 50 for generating a three-dimensional image 52 showing the object 4 in the real space and the plurality of two-dimensional images 50 and the three-dimensional image 52, which are associated with the plurality of parts 54 in the three-dimensional image 52, are displayed on the screen 16A in a state where they can be compared (see Fig. 27 ). Furthermore, the processor 20 selects the focus portion 54A from the plurality of portions 54 according to the given selection instruction, and selects the focus two-dimensional image 50A corresponding to the focus portion 54A from the plurality of two-dimensional images 50 (refer to Fig.28 ). Then, the processor 20 displays the two-dimensional image 50A of interest on the screen 16A in a state that it can be distinguished from the remaining two-dimensional images 50 in the plurality of two-dimensional images 50. Therefore, the correspondence between each two-dimensional image 50 and the area corresponding to each two-dimensional image 50 in the object 4 can be visually grasped.
[0222] Then, the processor 20 displays the focus portion 54A on the screen 16A in a state where it can be visually identified (see Fig.30 ). Therefore, the correspondence between the attention two-dimensional image 50A and the attention portion 54A can be visually confirmed.
[0223] Furthermore, the state in which the focus portion 54A can be visually determined includes a state in which the focus portion 54A can be distinguished from the remaining portions 54 of the plurality of portions 54 (refer to Fig.30 ). Therefore, for example, compared with a case where the focus portion 54A is not distinguished from the remaining portion 54, the visual recognition of the focus portion 54A can be improved.
[0224] Furthermore, the operation mode of the inspection supporting device 10 according to the second embodiment may be added as a fourth mode to the operation mode of the inspection supporting device 10 according to the first embodiment.
[0225] Furthermore, in the above embodiment, the processor 20 is illustrated as an example, but at least one other CPU, at least one GPU and / or at least one TPU may be used instead of or together with the processor 20 .
[0226] Furthermore, in the above-mentioned embodiment, an example of a method in which the inspection auxiliary information generation program 30 and the inspection auxiliary program 40 are stored in the storage device 22 is given for explanation, but the technology of the present invention is not limited to this. For example, the inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 can be stored in a portable non-temporary computer-readable storage medium such as an SSD or a USB memory (hereinafter referred to as a "non-temporary storage medium"). The inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 stored in the non-temporary storage medium can also be installed in the computer 12 of the inspection auxiliary device 10.
[0227] In addition, the inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 may be stored in a storage device such as another computer or server device connected to the inspection auxiliary device 10 via a network, and the inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 may be downloaded and installed in the computer 12 in response to a request from the inspection auxiliary device 10.
[0228] Furthermore, it is not necessary to store the entire inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 in a storage device or storage device 22 of another computer or server device connected to the inspection auxiliary device 10, and a portion of the inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 may be stored.
[0229] Furthermore, although the computer 12 is built into the inspection supporting device 10 , the technology of the present invention is not limited thereto, and for example, the computer 12 may be provided outside the inspection supporting device 10 .
[0230] Furthermore, in the above embodiment, the computer 12 including the processor 20, the storage device 22, and the RAM 24 is illustrated, but the technology of the present invention is not limited thereto, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 12. Furthermore, a combination of a hardware structure and a software structure may be used instead of the computer 12.
[0231] Furthermore, as hardware resources for executing various processes described in the above embodiments, various processors shown below can be used. As a processor, for example, a general-purpose processor, i.e., a CPU, which functions as a hardware resource for executing various processes by executing software, i.e., a program, can be cited. Furthermore, as a processor, for example, a dedicated circuit can be cited, which is a processor having a circuit structure specially designed for executing specific processes, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor executes various processes by using the memory.
[0232] The hardware resources for executing various processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resources for executing various processing may be one processor.
[0233] As an example of a system composed of one processor, first, there is a system in which one processor is composed of a combination of one or more CPUs and software, and the processor functions as a hardware resource for executing various processes. Secondly, as represented by SoC, there is a system in which a processor is used to realize the functions of the entire system including multiple hardware resources for executing various processes by one IC chip. In this way, various processes are realized by using one or more of the above-mentioned various processors as hardware resources.
[0234] Moreover, as the hardware structure of these various processors, more specifically, a circuit composed of a combination of circuit elements such as semiconductor elements can be used. In addition, the above-mentioned sight line detection processing is only an example. Therefore, it is of course possible to delete unnecessary steps, add new steps, or switch the processing order within the scope of the present invention.
[0235] The above-described contents and illustrated contents are detailed descriptions of the parts involved in the technology of the present invention, and are only an example of the technology of the present invention. For example, the descriptions related to the above-described structure, function, action, and effect are descriptions related to an example of the structure, function, action, and effect of the parts involved in the technology of the present invention. Therefore, without departing from the scope of the main purpose of the technology of the present invention, it is of course possible to delete unnecessary parts from the above-described contents and illustrated contents, or to add or replace new elements. Furthermore, in order to avoid confusion and facilitate understanding of the parts involved in the technology of the present invention, the descriptions of technical common sense that does not need to be specifically described in the technology that can implement the present invention are omitted in the above-described contents and illustrated contents.
[0236] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. In addition, in this specification, when three or more situations are connected with "and / or", the same consideration method as "A and / or B" is also applicable.
[0237] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each individual document, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.
Claims
1. An image processing device comprising a processor, The processor performs the following processing: Displaying a plurality of two-dimensional images for generating a three-dimensional image representing an object in a real space and correspondingly associated with a plurality of parts in the three-dimensional image and the three-dimensional image on a screen in a state where they can be compared; Selecting a focus two-dimensional image from the plurality of two-dimensional images according to a given selection instruction; and A focus portion corresponding to the focus two-dimensional image among the plurality of portions is displayed on the screen in a state in which it can be visually identified.
2. The image processing device according to claim 1, wherein: The state in which the plurality of two-dimensional images can be compared with the three-dimensional image refers to a state in which the first region including the plurality of two-dimensional images and the second region including the three-dimensional image are arranged side by side.
3. The image processing device according to claim 1 or 2, wherein: The state in which the portion of interest can be visually determined includes a state in which the portion of interest can be distinguished from the rest of the plurality of portions.
4. The image processing device according to any one of claims 1 to 3, wherein: The state in which the focus portion can be visually determined includes a state in which the focus two-dimensional image can be distinguished from the remaining two-dimensional images of the plurality of two-dimensional images.
5. The image processing device according to any one of claims 1 to 4, wherein: The processor performs the following processing: displaying a plurality of position-specific images on the screen in a state that can be compared with the three-dimensional image, the plurality of position-specific images being capable of identifying a plurality of photographing positions at which photographing was performed to obtain the plurality of two-dimensional images; selecting, from the plurality of shooting positions, a shooting position corresponding to the position-specific image of interest selected from the plurality of position-specific images as a shooting position of interest; and A two-dimensional image obtained by performing the photographing from the photographing position of interest is selected from among the plurality of two-dimensional images as the two-dimensional image of interest.
6. The image processing device according to claim 5, wherein: The state in which the plurality of position specifying images can be compared with the three-dimensional image includes a state in which the plurality of position specifying images and the three-dimensional image are opposed to each other.
7. The image processing device according to claim 5 or 6, wherein: The state in which the plurality of two-dimensional images can be compared with the three-dimensional image refers to a state in which a third region including the plurality of two-dimensional images and a fourth region including an image showing a style in which the plurality of position specific images and the three-dimensional image are placed side by side.
8. The image processing device according to any one of claims 5 to 7, wherein: The state in which the attention portion can be visually determined includes a state in which the attention position specific image can be distinguished from the remaining position specific images of the plurality of position specific images.
9. The image processing device according to any one of claims 5 to 8, wherein: The image processing device has: a first operation mode for displaying the plurality of two-dimensional images and the three-dimensional image on the screen in a state where they can be compared; and a second operation mode for displaying the plurality of position-specific images on the screen in a state where they can be compared with the three-dimensional image. The processor sets one of the first operation mode and the second operation mode according to a given setting instruction.
10. The image processing device according to any one of claims 5 to 9, wherein: The three-dimensional image is displayed on the screen at a viewpoint corresponding to the two-dimensional image of interest.
11. An image processing device comprising a processor, The processor performs the following processing: Displaying a plurality of two-dimensional images for generating a three-dimensional image representing an object in a real space and correspondingly associated with a plurality of parts in the three-dimensional image and the three-dimensional image on a screen in a state where they can be compared; According to a given selection indication, selecting a focus portion from the plurality of portions; selecting a two-dimensional image of interest corresponding to the portion of interest from among the plurality of two-dimensional images; and The focus two-dimensional image is displayed on the screen in a state where it can be distinguished from the remaining two-dimensional images among the plurality of two-dimensional images.
12. The image processing device according to claim 11, wherein: The processor performs the following processing: displaying a plurality of position-specific images on the screen in a state that can be compared with the three-dimensional image, the plurality of position-specific images being capable of identifying a plurality of photographing positions at which photographing was performed to obtain the plurality of two-dimensional images; selecting a location-specific image of interest from the plurality of location-specific images according to the selection instruction; and The two-dimensional image obtained by the photographing from the photographing position determined from the attention position specific image is selected from the plurality of two-dimensional images as the attention two-dimensional image.
13. An image processing method comprising the following steps: Displaying a plurality of two-dimensional images for generating a three-dimensional image representing an object in a real space and correspondingly associated with a plurality of parts in the three-dimensional image and the three-dimensional image on a screen in a state where they can be compared; Selecting a focus two-dimensional image from the plurality of two-dimensional images according to a given selection instruction; and A focus portion corresponding to the focus two-dimensional image among the plurality of portions is displayed on the screen in a state in which it can be visually identified.
14. A program for causing a computer to execute a process comprising the following steps: Displaying a plurality of two-dimensional images for generating a three-dimensional image representing an object in a real space and correspondingly associated with a plurality of parts in the three-dimensional image and the three-dimensional image on a screen in a state where they can be compared; Selecting a focus two-dimensional image from the plurality of two-dimensional images according to a given selection instruction; and A focus portion corresponding to the focus two-dimensional image among the plurality of portions is displayed on the screen in a state in which it can be visually identified.
