Image processing device, image processing method, and program

By generating three-dimensional images and using unit length information, the problem of difficulty in conveying the size of the object in the prior art is solved, and fast and accurate measurement and intuitive display of the size are achieved.

CN118901244APending Publication Date: 2024-11-05FUJIFILM CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280093931.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-29
Filing Date
2022-11-09
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

The prior art is difficult to effectively convey the dimensional sense of objects in actual space, and the measurement process is cumbersome and inaccurate enough.

Method used

By acquiring a plurality of two-dimensional images and corresponding shooting positions, a three-dimensional image is generated, and a unit length information generation unit generates a relationship representing the length of an object in the actual space, and outputs a synthetic image for easy comparison.

Benefits of technology

It realizes that users can intuitively perceive the size of objects in real space, and quickly and accurately measure the size of objects through precise unit length information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118901244A_ABST
    Figure CN118901244A_ABST
Patent Text Reader

Abstract

The image processing device includes a processor that performs: a process of acquiring a plurality of three-dimensional coordinates specifying positions of a plurality of pixels included in a three-dimensional image representing an object in an actual space, and a plurality of two-dimensional coordinates specifying positions corresponding to the plurality of pixels in a screen on which the three-dimensional image is rendered; acquiring unit length information indicating a relationship between a first unit length of a three-dimensional coordinate system defining three-dimensional coordinates and a second unit length of an actual space; generating an object capable of specifying a second unit length on the basis of the plurality of three-dimensional coordinates, the plurality of two-dimensional coordinates, and the unit length information; and outputting a first image in which the object and the three-dimensional image are expressed so as to be able to be compared with each other.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The technology of the present invention relates to an image processing device, an image processing method, and a program. Background Art

[0002] In Japanese Patent Publication No. 2020-150405, an image processing device is disclosed, which includes a subject designation unit, a designated position acquisition unit, a change detection unit, an indicator image generation unit, and a display processing unit. The subject designation unit specifies two points of the subject in the camera image. The designated position acquisition unit acquires three-dimensional position information related to the two points. The change detection unit detects changes related to the state of the image processing device. The indicator image generation unit generates an indicator image corresponding to the length and change between the two points based on the three-dimensional position information and the change. The display processing unit superimposes the indicator image on the camera image to obtain a processed image.

[0003] Japanese Patent Application Laid-Open No. 2012-105048 discloses a stereoscopic image display device capable of displaying a stereoscopic image composed of a right-eye image and a left-eye image having parallax with each other, which capture a subject, in a stereoscopic manner. The stereoscopic image display device includes a cursor display mechanism for displaying a scaled stereoscopic cursor at a position on the stereoscopic image determined by an input mechanism capable of inputting a three-dimensional position.

[0004] Japanese Patent Publication No. 2009-015730 discloses an image display system with a stereo measurement display function, which displays a camera parameter specified display image generated by shooting in a specified shooting direction and angle of view on a screen for an all-round image of a shooting location, and displays stereo measurement on the camera specified display image. The image display system with a stereo measurement display function has a three-dimensional space memory and a stereo measurement superimposed image display mechanism. The three-dimensional space memory is composed of a grid that divides the left and right side surfaces and the bottom surface, and stores stereo measurements that define the bottom width, side height, and length of the stereo measurement. The stereo measurement superimposed image display mechanism superimposes the interior of the stereo measurement of the stereo measurement storage mechanism when viewed in any direction and displays it in three dimensions on the camera parameter specified display image.

[0005] Japanese Patent Application Laid-Open No. 10-170227 discloses a display device that has at least one camera mechanism for photographing a subject, and synthesizes a plurality of camera images that overlap each other and are photographed from different viewpoints to display a stereoscopic image. The display device includes a measurement parameter calculation mechanism, a measurement image generation mechanism, and an image synthesis mechanism. The measurement parameter calculation mechanism calculates a measurement magnification and a parallax that serve as a reference for the scale of the subject based on the parallax of the subject between the plurality of camera images. The measurement image generation mechanism generates a measurement image based on the measurement magnification and parallax calculated by the measurement parameter calculation mechanism. The image synthesis mechanism synthesizes the measurement image generated by the measurement image generation mechanism in the stereoscopic image. Summary of the invention

[0006] As an example, an embodiment of the technology according to the present invention provides an image processing device, an image processing method, and a program that enable a user or the like to understand the sense of size of an object in a real space.

[0007] Means for solving technical problems

[0008] The first method involved in the technology of the present invention is an image processing device, which has a processor, wherein the processor performs the following processing: obtaining multiple three-dimensional coordinates that determine the positions of multiple pixels included in a three-dimensional image representing an object in real space and multiple two-dimensional coordinates that determine the positions corresponding to multiple pixels in a screen rendered with the three-dimensional image; obtaining unit length information representing the relationship between a first unit length of a three-dimensional coordinate system that specifies the three-dimensional coordinates and a second unit length in real space; generating an object capable of determining a second unit length based on the multiple three-dimensional coordinates, the multiple two-dimensional coordinates and the unit length information; and outputting a first image that represents the object and the three-dimensional image in a manner that can be compared.

[0009] A second aspect of the technology of the present invention is an image processing device, wherein, in the image processing device of the first aspect, the three-dimensional image is an image generated from a plurality of two-dimensional images obtained by photographing the object from a plurality of photographing positions in a real space.

[0010] A third aspect of the technology of the present invention is an image processing device according to the second aspect, wherein the unit length information is information generated based on a distance between adjacent imaging positions among a plurality of imaging positions.

[0011] A fourth aspect according to the technology of the present invention is an image processing device according to the third aspect, wherein the distance is a distance obtained by a positioning unit.

[0012] A fifth aspect of the technology of the present invention is an image processing device, wherein, in the image processing device according to the second aspect, the second unit length is a length related to a subject image included in at least one of the plurality of two-dimensional images.

[0013] A sixth aspect of the technology of the present invention is an image processing device, wherein, in the image processing device according to any one of the first to fifth aspects, the object is an image generated based on designated two-dimensional coordinates among a plurality of two-dimensional coordinates.

[0014] A seventh aspect of the technology of the present invention is an image processing device, wherein, in the image processing device according to any one of the first to sixth aspects, the object is an image including a graphic and a numerical value representing a length related to the graphic.

[0015] The eighth mode involved in the technology of the present invention is an image processing device involved in any one of the first to seventh modes, wherein the processor performs the following processing: changing the first viewpoint for observing a three-dimensional image through a screen according to a given first instruction; and changing the second viewpoint for observing an object through the screen according to the first viewpoint.

[0016] A ninth aspect of the technology of the present invention is an image processing device according to any one of the first to eighth aspects, wherein the processor changes a third viewpoint at which the object is observed through the screen in accordance with a given second instruction.

[0017] A tenth aspect according to the technology of the present invention is an image processing device according to any one of the first to ninth aspects, wherein the object includes an image representing an object existing in a real space.

[0018] The 11th method involved in the technology of the present invention is an image processing method, which includes the following steps: obtaining multiple three-dimensional coordinates for determining the positions of multiple pixels included in a three-dimensional image representing an object in real space and multiple two-dimensional coordinates for determining the positions corresponding to multiple pixels in a screen rendered with the three-dimensional image; obtaining unit length information representing the relationship between a first unit length of a three-dimensional coordinate system that specifies the three-dimensional coordinates and a second unit length in real space; generating an object capable of determining a second unit length based on the multiple three-dimensional coordinates, the multiple two-dimensional coordinates and the unit length information; and outputting a first image that represents the object and the three-dimensional image in a manner that can be compared.

[0019] The 14th method involved in the technology of the present invention is a program, which is used to cause a computer to execute a processing including the following steps: obtaining multiple three-dimensional coordinates for determining the positions of multiple pixels included in a three-dimensional image representing an object in real space and multiple two-dimensional coordinates for determining the positions corresponding to multiple pixels in a screen rendered with the three-dimensional image; obtaining unit length information representing the relationship between a first unit length of a three-dimensional coordinate system that specifies the three-dimensional coordinates and a second unit length in real space; generating an object capable of determining a second unit length based on the multiple three-dimensional coordinates, the multiple two-dimensional coordinates and the unit length information; and outputting a first image that represents the object and the three-dimensional image in a manner that can be compared. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a perspective view showing an example of the inspection system according to the present embodiment.

[0021] Figure 2 This is a block diagram showing an example of the inspection support device according to the present embodiment.

[0022] Figure 3 This is a block diagram showing an example of the imaging device according to the present embodiment.

[0023] Figure 4 This is a block diagram showing an example of a functional configuration for realizing the inspection support information generation processing involved in this embodiment.

[0024] Figure 5 This is a block diagram showing an example of data transmitted from the imaging device according to the present embodiment to the inspection supporting device.

[0025] Figure 6 This is a block diagram showing an example of the operations of the acquisition unit, the three-dimensional image information generation unit, and the unit length information generation unit according to the present embodiment.

[0026] Figure 7 This is a block diagram showing an example of the operations of the three-dimensional image information generating unit, the unit length information generating unit, and the inspection support information generating unit according to the present embodiment.

[0027] Figure 8 This is a block diagram showing an example of a functional configuration for realizing the inspection support processing involved in this embodiment.

[0028] Fig. 9 This is a block diagram showing an example of the operations of the rendering unit, the instruction determination unit, the instruction acquisition unit, the object generation unit, and the synthesized image output unit according to the present embodiment.

[0029] Fig.10 This is a block diagram showing an example of the operation of the object generation unit that executes the first object generation process according to the present embodiment.

[0030] Fig.11 This is a block diagram showing an example of the operation of the object generation unit that executes the second object generation process according to the present embodiment.

[0031] Fig.12 This is a block diagram showing an example of the operations of the instruction acquisition unit and the synthesized image output unit according to the present embodiment.

[0032] Fig.13 This is a block diagram showing an example of the operations of the instruction acquisition unit, the object generation unit, and the synthesized image output unit according to the present embodiment.

[0033] Fig.14 This is a block diagram showing an example of the operations of the instruction acquisition unit and the synthesized image output unit according to the present embodiment.

[0034] Fig.15 This is a flowchart showing an example of the flow of the inspection support information generation process according to the present embodiment.

[0035] Fig.16 This is a flowchart showing an example of the flow of the inspection support process according to the present embodiment.

[0036] Fig.17 This is a flowchart showing an example of the flow of the first object generation process according to the present embodiment.

[0037] Fig.18 This is a flowchart showing an example of the flow of the second object generation process according to the present embodiment.

[0038] Fig.19 This is a screen diagram showing a first variation of the object involved in this embodiment.

[0039] Fig. 20 It is a screen diagram showing a second variation example of the object involved in this embodiment.

[0040] Fig.21 This is a screen diagram showing a third variation of the object involved in this embodiment.

[0041] Fig. 22 It is a block diagram showing a modified example of the operation of the unit length information generating section according to the present embodiment. DETAILED DESCRIPTION

[0042] Hereinafter, an example of an embodiment of an image processing device, an image processing method, and a program according to the technology of the present invention will be described with reference to the drawings.

[0043] First, the terms used in the following description are explained.

[0044] CPU is the abbreviation of "Central Processing Unit". GPU is the abbreviation of "Graphics Processing Unit". HDD is the abbreviation of "Hard Disk Drive". SSD is the abbreviation of "Solid State Drive". RAM is the abbreviation of "Random Access Memory". SRAM is the abbreviation of "Static Random Access Memory". DRAM is the abbreviation of "Dynamic Random Access Memory". EL is the abbreviation of "Electro Luminescence". RAM is the abbreviation of "Random Access Memory". CMOS is the abbreviation of "Complementary Metal Oxide Semiconductor". GNSS is the abbreviation of "Global Navigation Satellite System". GPS is the abbreviation of "Global Positioning System". SfM is the abbreviation of "Structure from Motion". MVS is the abbreviation of "Multi-View Stereo". TPU is the abbreviation of "Tensor processing unit". USB is the abbreviation of "Universal Serial Bus". ASIC is the abbreviation of "Application Specific Integrated Circuit". FPGA is the abbreviation of "Field-Programmable Gate Array". PLD is the abbreviation of "Programmable Logic Device". SoC is the abbreviation of "System-on-a-chip". IC is the abbreviation of "Integrated Circuit".

[0045] As an example, Figure 1As shown, the inspection system S includes an inspection support device 10 and an imaging device 100. The inspection system S is a system for inspecting an object 4 in a real space. The object 4 is an example of the "object" of the technology of the present invention.

[0046] As an example, the object 4 is a bridge pier made of reinforced concrete. Here, as an example of the object 4, a bridge pier is cited, but the object 4 may also be a road facility other than the bridge pier. As road facilities, for example, road surfaces, tunnels, guardrails, traffic lights and / or windbreak fences may be cited. The object 4 may be a social infrastructure other than road facilities (for example, airport facilities, port facilities, water storage facilities, gas facilities, medical facilities, firefighting facilities and / or educational facilities, etc.), or private property. Furthermore, the object 4 may also be land (for example, state-owned land and / or private land, etc.). The bridge pier exemplified as the object 4 may also be a bridge pier other than that made of reinforced concrete. In the present embodiment, the inspection, for example, refers to the inspection of the state of the object 4. For example, the inspection system S is used to inspect whether the object 4 is damaged and / or the degree of damage.

[0047] The inspection assisting device 10 is an example of an "image processing device" involved in the technology of the present invention. The inspection assisting device 10 is, for example, a desktop personal computer. Here, a desktop personal computer is illustrated as the inspection assisting device 10, but this is only an example, and it may also be a notebook personal computer. Furthermore, it is not limited to a personal computer, and it may also be a server. In addition, the server may be a large computer used together with the inspection assisting device 10 through local deployment (On-Premises), or it may be an external server implemented through cloud computing. Furthermore, the server may also be an external server implemented through network computing such as fog computing, edge computing, or grid computing. The inspection assisting device 10 is connected to the camera device 100 and is capable of communication. The inspection assisting device 10 is used by the inspector 6. The inspection assisting device 10 may be used at a site where the object 4 is set, or it may be used at a location different from the site where the object 4 is set.

[0048] The camera device 100 is, for example, a lens-interchangeable digital camera. Here, a lens-interchangeable digital camera is exemplified as the camera device 100, but this is only an example, and it may also be a digital camera built into various electronic devices such as a smart device or a wearable terminal. In addition, the camera device 100 may be a glasses-type eyeglass terminal, or a head-mounted display terminal worn on the head. The camera device 100 is used by the photographer 8.

[0049] As an example, Figure 2 As shown, the inspection support device 10 includes a computer 12 , a receiving device 14 , a display 16 , and a communication device 18 .

[0050] The computer 12 is an example of a "computer" involved in the technology of the present invention. The computer 12 includes a processor 20, a storage device 22, and a RAM 24. The processor 20 is an example of a "processor" involved in the technology of the present invention. The processor 20, the storage device 22, the RAM 24, the receiving device 14, the display 16, and the communication device 18 are connected to a bus 26.

[0051] The processor 20 includes, for example, a CPU, and controls the entire inspection auxiliary device 10. Here, an example in which the processor 20 includes a CPU is given, but this is only an example. For example, the processor 20 may include a CPU and a GPU. In this case, for example, the GPU operates under the control of the CPU and is responsible for performing image processing.

[0052] The storage device 22 is a nonvolatile storage device that stores various programs and various parameters. As the storage device 22, for example, a HDD and an SSD can be cited. In addition, the HDD and the SSD are only examples, and a flash memory, a magnetoresistive memory, and / or a ferroelectric memory may be used instead of or in addition to the HDD and / or the SSD.

[0053] The RAM 24 is a memory for temporarily storing information, and is used as a working memory by the processor 20. Examples of the RAM 24 include DRAM and / or SRAM.

[0054] The receiving device 14 has a keyboard, a mouse, a touch panel, etc. (all omitted from the figure), and receives various instructions from the inspector 6. The display 16 has a screen 16A. The screen 16A is an example of a "screen" involved in the technology of the present invention. Under the control of the processor 20, the display 16 displays various information (for example, images and characters, etc.) on the screen 16A. As the display 16, for example, an EL display (for example, an organic EL display or an inorganic EL display) can be cited. In addition, it is not limited to an EL display, and other types of displays such as a liquid crystal display can also be used.

[0055] The communication device 18 is connected to the imaging device 100 and is capable of communication. Here, the communication device 18 is connected to the imaging device 100 by a prescribed wireless communication standard and is capable of wireless communication. Examples of the prescribed wireless communication standard include Wi-Fi (registered trademark) and Bluetooth (registered trademark). The communication device 18 is responsible for sending and receiving information with the inspection auxiliary device 10. For example, the communication device 18 sends information in response to a request from the processor 20 to the imaging device 100. In addition, the communication device 18 receives information sent from the imaging device 100 and outputs the received information to the processor 20 via the bus 26. In addition, the communication device 18 may also be connected to the imaging device 100 by wire and be capable of communication.

[0056] As an example, Figure 3 As shown, the imaging device 100 includes a computer 102 , an image sensor 104 , a positioning unit 106 , and a communication device 112 .

[0057] The computer 102 includes a processor 114, a storage device 116, and a RAM 118. The processor 114, the storage device 116, the RAM 118, the image sensor 104, the positioning unit 106, and the communication device 112 are connected to a bus 120. The processor 114, the storage device 116, and the RAM 118 are implemented by, for example, the same hardware as the processor 20, the storage device 22, and the RAM 24 included in the above-mentioned inspection supporting device 10.

[0058] The image sensor 104 is, for example, a CMOS image sensor. In addition, here, a CMOS image sensor is exemplified as the image sensor 104, but the technology of the present invention is not limited thereto, and other image sensors may also be used. The image sensor 104 captures a subject (for example, the object 4) and outputs image data obtained by the capture.

[0059] The positioning unit 106 is a device that detects the position of the camera device 100. The position of the camera device 100 is detected using, for example, GNSS (for example, GPS). The positioning unit 106 has a GNSS receiver (not shown). The GNSS receiver receives, for example, radio waves transmitted from a plurality of satellites. The positioning unit 106 detects the position of the camera device 100 based on the radio waves received by the GNSS receiver, and outputs positioning data corresponding to the detected position (for example, data indicating latitude, longitude, and altitude).

[0060] The processor 114 acquires the position of the camera 100 based on the positioning data and generates position data indicating the acquired position. Hereinafter, the position of the camera 100 is referred to as a “photographing position.” The photographing position acquired based on the positioning data is a photographing position in an absolute coordinate system.

[0061] In addition, an acceleration sensor (not shown) may be used instead of the positioning data to obtain the shooting position based on the acceleration data from the acceleration sensor. The shooting position obtained based on the acceleration data is the shooting position in the relative coordinate system.

[0062] The communication device 112 is connected to the inspection support device 10 so as to be communicable. The communication device 112 is realized by, for example, the same hardware as the communication device 18 included in the inspection support device 10 described above.

[0063] The imaging device 100 transmits image data and position data to the inspection support device 10. The image data is data representing the two-dimensional image 51 obtained by imaging the object 4 by the imaging device 100. The position data is data representing the imaging position when the imaging device 100 performed imaging, and is associated with the image data.

[0064] As an example, Figure 4 As shown, an inspection auxiliary information generation program 30 is stored in the storage device 22 of the inspection auxiliary apparatus 10. The processor 20 of the inspection auxiliary apparatus 10 reads the inspection auxiliary information generation program 30 from the storage device 22, and executes the read inspection auxiliary information generation program 30 on the RAM 24. The processor 20 performs inspection auxiliary information generation processing for generating inspection auxiliary information 74 according to the inspection auxiliary information generation program 30 executed on the RAM 24.

[0065] The inspection support information generation process is realized by the processor 20 operating as the acquisition unit 32 , the three-dimensional image information generation unit 34 , the unit length information generation unit 36 ​​, and the inspection support information generation unit 38 according to the inspection support information generation program 30 .

[0066] As an example, Figure 5 As shown, a plurality of points P1 located in the circumferential direction of the object 4 represent the shooting positions by the camera 100. The photographer 8 moves around the object 4 and shoots the object 4 from a plurality of shooting positions in the circumferential direction of the object 4 by means of the camera 100. As an example, the photographer 8 shoots a different area in the object 4 from each shooting position by means of the camera 100. By shooting a different area in the object 4 from each shooting position by means of the camera 100, the entire object 4 including the plurality of areas can be shot.

[0067] The shooting position (i.e., point P1) corresponding to each two-dimensional image 51 obtained by shooting with the camera 100 is equivalent to the starting point of the line of sight L focused on the object 4, and the shooting posture corresponding to each two-dimensional image 51 is equivalent to the direction of the line of sight L focused on the object 4. The point P2 where the object 4 intersects with the line of sight L is equivalent to the viewpoint when the object 4 is observed from the line of sight L. By shooting the object 4 from each shooting position with the camera 100, the two-dimensional image 51 corresponding to each viewpoint can be obtained. Each two-dimensional image 51 is an image corresponding to each area in the object 4.

[0068] In addition, here, an example is given in which the photographer 8 photographs the object 4 from various photographing positions using the camera device 100 while moving around the object 4. However, when the camera device 100 is mounted on a moving body and the moving body moves around the object 4, the object 4 may be photographed from various photographing positions using the camera device 100. Furthermore, the moving body may be, for example, a drone, a pod, a trolley, an aerial work vehicle, an unmanned vehicle, or other vehicles.

[0069] The imaging device 100 associates image data representing the two-dimensional image 51 obtained by imaging from each imaging position with position data representing the imaging position at the time of imaging. Then, the imaging device 100 transmits each image data and the associated position data to the inspection support device 10 .

[0070] As an example, Figure 6 As shown, the acquisition unit 32 acquires the two-dimensional image 51 based on each image data received from the inspection support device 10 . Furthermore, the acquisition unit 32 acquires the imaging position corresponding to each two-dimensional image 51 based on each position data received from the inspection support device 10 .

[0071] The three-dimensional image information generating unit 34 generates three-dimensional image information 70 based on the plurality of two-dimensional images 51 and the plurality of imaging positions acquired by the acquiring unit 32. The three-dimensional image information 70 is image information representing the three-dimensional image 52 defined by the three-dimensional coordinate system.

[0072] The three-dimensional coordinate system is a relative coordinate system defined by a plurality of shooting positions. That is, the three-dimensional coordinate system is a coordinate system on the three-dimensional virtual space 80 that is set independently of the actual space defined by the absolute coordinate system, that is, the world coordinate system. Axis X1, axis Y1, and axis Z1 represent three coordinate axes in the three-dimensional coordinate system, and axis X2, axis Y2, and axis Z2 represent three coordinate axes in the world coordinate system. The three-dimensional coordinate is an example of the "three-dimensional coordinate" involved in the technology of the present invention.

[0073] The three-dimensional image 52 is a representation of the object 4 (refer to Figure 5 ) and is an image generated from a plurality of two-dimensional images 51. As image processing techniques for generating a three-dimensional image 52 from a plurality of two-dimensional images 51, SfM, MVS, epipolar geometry, and stereo matching processing can be cited. A plurality of three-dimensional coordinates that determine the positions of a plurality of pixels included in the three-dimensional image 52 are coordinates of a three-dimensional coordinate system. A pixel is an example of a "pixel" involved in the technology of the present invention.

[0074] The intervals between the plurality of grid lines 82 set on each coordinate axis of the three-dimensional coordinate system are equivalent to the distance between the centers of the plurality of pixels adjacent to each other in the direction of each coordinate axis. Hereinafter, the length of the interval between the grid lines 82 is referred to as the "first unit length". The first unit length is the unit length of the three-dimensional coordinate system.

[0075] The three-dimensional coordinates of each shooting position (i.e., point P1) are respectively specified by the coordinates of the three-dimensional coordinate system and the coordinates of the world coordinate system. Hereinafter, the distance between adjacent shooting positions specified by the coordinates of the three-dimensional coordinate system is referred to as "relative distance L1", and the distance between adjacent shooting positions specified by the coordinates of the world coordinate system is referred to as "absolute distance L2".

[0076] The relative distance L1 is a distance expressed by a first unit length, and the absolute distance L2 is a distance expressed by a second unit length set in the actual space. The absolute distance L2 is obtained by the positioning unit 106 (reference Figure 3 ) obtained. That is, each shooting position is derived from each position data output from the positioning unit 106, and the distance between adjacent shooting positions, that is, the absolute distance L2, is derived from each shooting position. The absolute distance L2 is an example of the "distance between shooting positions" involved in the technology of the present invention.

[0077] The unit length information generating unit 36 ​​generates unit length information 72 indicating the relationship between the first unit length and the second unit length. Specifically, the unit length information generating unit 36 ​​acquires the relative distance L1 from the three-dimensional coordinate system included in the three-dimensional image information 70 generated by the three-dimensional image information generating unit 34. Furthermore, the unit length information generating unit 36 ​​acquires the absolute distance L2 corresponding to the relative distance L1 from the plurality of imaging positions acquired by the acquiring unit 32. Then, the unit length information generating unit 36 ​​generates the unit length information 72 indicating the relationship between the first unit length and the second unit length based on the relative distance L1 and the absolute distance L2.

[0078] As an example, Figure 7 As shown, the inspection auxiliary information generating unit 38 generates inspection auxiliary information 74 including the three-dimensional image information 70 generated by the three-dimensional image information generating unit 34 and the unit length information 72 generated by the unit length information generating unit 36. The inspection auxiliary information 74 is stored in the storage device 22.

[0079] As an example, Figure 8As shown, the storage device 22 of the inspection support device 10 stores an inspection support program 40. The inspection support program 40 is an example of a "program" involved in the technology of the present invention. The processor 20 reads the inspection support program 40 from the storage device 22 and executes the read inspection support program 40 on the RAM 24. The processor 20 performs the inspection support program 40 executed on the RAM 24 to assist the inspector 6 (refer to Figure 1 ) to conduct inspection auxiliary processing.

[0080] The inspection support processing is realized by the processor 20 operating as a rendering unit 42 , an instruction determination unit 44 , an instruction acquisition unit 46 , an object generation unit 48 , and a synthesized image output unit 50 according to the inspection support program 40 .

[0081] As an example, Fig. 9 As shown, the rendering unit 42 renders the three-dimensional image 52 onto the screen 16A based on a plurality of pixels of the three-dimensional image 52 included in the inspection auxiliary information 74. The three-dimensional image 52 rendered onto the screen 16A includes the object 4 (reference Figure 5 ) corresponding to the object image 53. The positions corresponding to the plurality of pixels in the screen 16A on which the three-dimensional image 52 is rendered are determined by a plurality of two-dimensional coordinates. The plurality of two-dimensional coordinates are coordinates of a two-dimensional coordinate system set on the screen 16A. Axis X3 and axis Y3 represent two coordinate axes in the two-dimensional coordinate system.

[0082] The inspector 6 gives the receiving device 14 an instruction to display the object 54. As a first example of the instruction to display the object 54, there can be given an instruction to designate the start point and the end point of the object 54. Furthermore, as a second example of the instruction to display the object 54, there can be given an instruction to designate the start point of the object 54, the direction from the start point to the end point of the object 54 (i.e., the direction of the object 54), and the length from the start point to the end point of the object 54.

[0083] As the instruction given to the receiving device 14, for example, there can be an instruction by clicking a mouse, an instruction by dragging a mouse, an instruction by dragging and dropping, or an instruction input on a keyboard. When receiving an instruction from the inspector 6 to display the object 54 on the screen 16A, the receiving device 14 outputs instruction data including the instruction of the inspector 6 to the processor 20.

[0084] The instruction determination unit 44 determines whether instruction data is input to the processor 20. When the instruction determination unit 44 determines that instruction data is input to the processor 20, the instruction acquisition unit 46 acquires the instruction data.

[0085] The object generation unit 48 performs an object generation process. The object generation process is a process of generating an object 54 based on the instruction data acquired by the instruction acquisition unit 46. The object 54 is an example of an "object" involved in the technology of the present invention. The object generation unit 48 will be described in detail later.

[0086] The synthesized image output unit 50 generates a synthesized image 56 by synthesizing the three-dimensional image 52 and the object 54 generated by the object generation unit 48. Then, the synthesized image output unit 50 renders the synthesized image 56 on the screen 16A. Thus, the synthesized image 56 showing the object 54 and the three-dimensional image 52 in a manner that allows comparison is displayed on the screen 16A of the display 16. The synthesized image 56 is an example of the "first image" involved in the technology of the present invention.

[0087] Next, the object generation unit 48 and the object 54 will be described in detail.

[0088] As an example, in Fig.10 , a case where instruction data (hereinafter referred to as "first instruction data") including instructions for specifying the start point and the end point of the object 54 is obtained by the instruction acquisition unit 46. Specifically, the instruction for specifying the start point of the object 54 is an instruction for specifying a position corresponding to the start point of the object 54 among positions corresponding to a plurality of pixels within the screen 16A on which the three-dimensional image 52 is rendered. And, specifically, the instruction for specifying the end point of the object 54 is an instruction for specifying a position corresponding to the end point of the object 54 among positions corresponding to a plurality of pixels within the screen 16A on which the three-dimensional image 52 is rendered.

[0089] When the first instruction data is acquired by the instruction acquisition unit 46 , the object generation unit 48 executes a first object generation process among the object generation processes.

[0090] In the first object generation process, the object generation unit 48 obtains first two-dimensional coordinates corresponding to the start point of the object 54 and second two-dimensional coordinates corresponding to the end point of the object 54 based on a plurality of pixels in the screen 16A on which the three-dimensional image 52 is rendered.

[0091] Next, the object generator 48 acquires first three-dimensional coordinates corresponding to the first two-dimensional coordinates and second three-dimensional coordinates corresponding to the second two-dimensional coordinates based on a plurality of pixels of the three-dimensional image 52 included in the inspection support information 74 .

[0092] Next, the object generating unit 48 derives the distance between the first three-dimensional coordinates and the second three-dimensional coordinates (hereinafter referred to as “the distance between three-dimensional coordinates”).

[0093] Next, the object generating unit 48 acquires the relationship between the distance between the three-dimensional coordinates and the first unit length based on the unit length information 72 included in the inspection support information 74 .

[0094] Next, the object generating unit 48 derives the relationship between the distance between the three-dimensional coordinates and the second unit length based on the unit length information 72 included in the inspection support information 74 and the relationship between the distance between the three-dimensional coordinates and the first unit length.

[0095] Next, the object generating unit 48 derives the length of the virtual object 54 when arranged in the real space based on the relationship between the distance between the three-dimensional coordinates and the second unit length.

[0096] Next, the object generating unit 48 generates an object 54 which is an image including a figure 58 extending between the first three-dimensional coordinates and the second three-dimensional coordinates and a numerical value 60 based on the length of the object 54 .

[0097] The graphic 58 can be any graphic. Fig.10 In the example shown, a graphic simulating a "ruler" is included in the object 54 as the graphic 58. Furthermore, the numerical value 60 may be any numerical value as long as it indicates the length related to the graphic 58. Fig.10 In the example shown, the object 54 includes, as the numerical value 60, a first numerical value (e.g., "0") indicating a base point of length, a second numerical value (e.g., "1") indicating a position of a scale added to the graphic 58, and a third numerical value (e.g., "2") indicating the length of the graphic 58. When the object 54 includes the numerical value 60, the second unit length of the object 54 is determined. The numerical value 60 may include, for example, a unit of the length of the object 54 (e.g., meter, etc.).

[0098] In the first object generation process, the object 54 is generated by an instruction to designate the start point and the end point of the object 54 (that is, two points of the object 54 ). However, the object 54 may be generated by an instruction to designate a plurality of points of the object 54 .

[0099] As an example, in Fig.11 , there is shown a case where instruction data (hereinafter referred to as “second instruction data”) including instructions for specifying the starting point of the object 54 , the direction from the starting point to the end point of the object 54 , and the length of the object 54 is acquired by the instruction acquisition unit 46 .

[0100] The instruction to specify the direction from the start point to the end point of the object 54 is, for example, an instruction to specify the direction of the object 54. The instruction to specify the length of the object 54 is, for example, an instruction to specify the length from the start point to the end point of the object 54.

[0101] When the second instruction data is acquired by the instruction acquisition unit 46 , the object generation unit 48 executes the second object generation process among the object generation processes.

[0102] In the second object generation process, the object generation unit 48 acquires the first two-dimensional coordinates corresponding to the start point of the object 54 based on a plurality of pixels in the screen 16A on which the three-dimensional image 52 is rendered.

[0103] Next, the object generator 48 acquires the first three-dimensional coordinates corresponding to the first two-dimensional coordinates based on a plurality of pixels of the three-dimensional image 52 included in the inspection support information 74 .

[0104] Next, the object generation unit 48 derives a three-dimensional virtual space 80 (see FIG. 1 ) corresponding to the direction from the start point to the end point of the object 54 and the length of the object 54 based on the unit length information 72 included in the inspection auxiliary information 74. Figure 6 ) (hereinafter referred to as “virtual space distance”).

[0105] Next, the object generator 48 acquires second three-dimensional coordinates separated by the virtual space distance in the direction from the start point to the end point of the object 54 from the first three-dimensional coordinates based on a plurality of pixels of the three-dimensional image 52 included in the inspection support information 74 .

[0106] Next, the object generation unit 48 generates an object 54, which is an image including a figure 58 extending between the first three-dimensional coordinate and the second three-dimensional coordinate and a numerical value 60 based on the length of the object 54. As an example, Fig.11 The object 54 shown is Fig.10 The objects 54 shown are identical.

[0107] As an example, in Fig.12 , a case where a synthetic image 56 including a three-dimensional image 52 and an object 54 is displayed on the screen 16A, and an instruction (hereinafter referred to as "first instruction") to change the viewpoint (hereinafter referred to as "first viewpoint") for observing the three-dimensional image 52 through the screen 16A is received by the receiving device 14. In this case, instruction data including the first instruction (hereinafter referred to as "third instruction data") is output from the receiving device 14 to the processor 20.

[0108] As the first instruction, for example, there can be mentioned an instruction by clicking the mouse or an instruction by dragging the mouse.

[0109] The synthesized image output unit 50 changes the first viewpoint according to the first instruction represented by the third instruction data. The first viewpoint is an example of the "first viewpoint" involved in the technology of the present invention. In addition, the synthesized image output unit 50 changes the viewpoint (hereinafter referred to as the "second viewpoint") at which the object 54 is observed through the screen 16A according to the first viewpoint. As a result, the direction of the synthesized image 56 including the three-dimensional image 52 and the object 54 is changed.

[0110] Even when the graphic 58 is inverted by changing the first viewpoint, the numerical value 60 can be maintained facing the front of the screen 16A. The second viewpoint is an example of the "second viewpoint" involved in the technique of the present invention.

[0111] As an example, in Fig.13 , a case where a synthetic image 56 including a three-dimensional image 52 and an object 54 is displayed on the screen 16A, and an instruction (hereinafter referred to as "second instruction") to change the viewpoint (hereinafter referred to as "third viewpoint") for observing the object 54 through the screen 16A is received by the receiving device 14. In this case, instruction data including the second instruction (hereinafter referred to as "fourth instruction data") is output from the receiving device 14 to the processor 20.

[0112] As the second instruction, for example, an instruction by clicking the mouse or an instruction by dragging the mouse can be cited. The second instruction is an example of the "second instruction" involved in the technology of the present invention. The third viewpoint is an example of the "third viewpoint" involved in the technology of the present invention.

[0113] When the object generating unit 48 obtains the fourth indication data from the indication acquiring unit 46, the object generating unit 48 obtains the first two-dimensional coordinate corresponding to the starting point of the object 54 and the second two-dimensional coordinate corresponding to the end point of the object 54 when the third viewpoint is assumed to be changed, based on a plurality of pixels within the screen 16A on which the three-dimensional image 52 is rendered.

[0114] Next, the object generating unit 48 generates a new object 54 by performing the same process as the first object generating process based on the first two-dimensional coordinates and the second two-dimensional coordinates.

[0115] The synthesized image output unit 50 generates a synthesized image 56 by synthesizing the three-dimensional image 52 and the object 54 generated by the object generation unit 48. Then, the synthesized image output unit 50 renders the synthesized image 56 on the screen 16A. Thus, the new synthesized image 56 is displayed on the screen 16A of the display 16. In the new synthesized image 56, the direction of the object 54 is changed by changing the third viewpoint.

[0116] In addition, Fig.13In the example shown, the case where the second instruction to change the third viewpoint is received by the receiving device 14 while the synthesized image 56 is displayed on the screen 16A is shown. However, even when the receiving device 14 receives an instruction to move the object 54 or change the length of the object 54, etc. while the synthesized image 56 is displayed on the screen 16A, the object generating unit 48 can obtain the first two-dimensional coordinates corresponding to the start point of the object 54 and the second two-dimensional coordinates corresponding to the end point of the object 54 after the change. Then, the object generating unit 48 can generate a new object 54 based on the obtained first two-dimensional coordinates and second two-dimensional coordinates.

[0117] As an example, in Fig.14 , a case where a synthetic image 56 including a three-dimensional image 52 and an object 54 is displayed on the screen 16A and an instruction (hereinafter referred to as the “third instruction”) to change the size of the three-dimensional image 52 is received by the receiving device 14. In this case, instruction data including the third instruction (hereinafter referred to as the “fifth instruction data”) is output from the receiving device 14 to the processor 20.

[0118] As the third instruction, for example, there can be mentioned an instruction by clicking a mouse, an instruction by scrolling the screen 16A using a scroll wheel provided on the mouse, and the like.

[0119] The synthesized image output unit 50 enlarges or reduces the synthesized image 56 including the three-dimensional image 52 and the object 54 according to the third instruction indicated by the fifth instruction data. Fig.14 , an example in which the synthesized image 56 is enlarged is shown.

[0120] Next, refer to Figures 15 to 18 The operation of the inspection support device 10 will be described.

[0121] First, refer to Fig.15 , an example of the flow of the inspection support information generation process performed by the processor 20 of the inspection support device 10 is described.

[0122] exist Fig.15 In the inspection auxiliary information generation process shown in FIG. 1 , first, in step ST10, the acquisition unit 32 (refer to Figure 6 ) acquires the two-dimensional image 51 based on each image data received by the inspection auxiliary device 10. And, the acquisition unit 32 acquires the shooting position corresponding to each two-dimensional image 51 based on each position data received by the inspection auxiliary device 10. After executing the process of step ST10, the inspection auxiliary information generation process proceeds to step ST12.

[0123] In step ST12, the three-dimensional image information generating unit 34 (refer to Figure 6) Based on the plurality of two-dimensional images 51 and the plurality of shooting positions acquired in step ST10, three-dimensional image information 70 representing the three-dimensional image 52 defined by the three-dimensional coordinate system is generated. After the process of step ST12 is executed, the inspection auxiliary information generation process proceeds to step ST14.

[0124] In step ST14, the unit length information generating unit 36 ​​(refer to Figure 6 ) generates unit length information 72 indicating the relationship between the first unit length and the second unit length. After executing the process of step ST14, the inspection auxiliary information generation process proceeds to step ST16.

[0125] In step ST16, the auxiliary information generating unit 38 (reference Figure 7 ) generates inspection auxiliary information 74 including the three-dimensional image information 70 generated by the three-dimensional image information generating unit 34 and the unit length information 72 generated by the unit length information generating unit 36. After the process of step ST16 is executed, the inspection auxiliary information generating process is terminated.

[0126] Next, refer to Figures 16 to 18 , an example of the flow of the inspection support processing performed by the processor 20 of the inspection support device 10 is described. Fig.16 , an example of the overall process of inspection auxiliary processing is explained.

[0127] exist Fig.16 In the inspection auxiliary process shown in FIG. 1 , first, in step ST20, the rendering unit 42 (refer to FIG. Fig. 9 ) Render the three-dimensional image 52 on the screen 16A according to the plurality of pixels of the three-dimensional image 52 included in the inspection auxiliary information 74. After executing the process of step ST20, the inspection auxiliary process proceeds to step ST22.

[0128] In step ST22, the instruction determination unit 44 (refer to Fig. 9 ) determines whether the instruction data is input to the processor 20. In step ST22, when the instruction data is input to the processor 20, it is determined that it is approved, and the inspection auxiliary process enters step ST24. In step ST22, when the instruction data is not input to the processor 20, it is determined that it is rejected, and the inspection auxiliary process enters step ST30.

[0129] In step ST24, the instruction acquisition unit 46 (refer to Fig. 9 ) obtains the instruction data input into the processor 20. After executing the process of step ST24, the inspection auxiliary process proceeds to step ST26.

[0130] In step ST26, the object generation unit 48 executes object generation processing for generating the object 54 based on the instruction data acquired in step ST24. After the processing in step ST26 is executed, the inspection support processing proceeds to step ST28.

[0131] In step ST28, the synthesized image output unit 50 generates a synthesized image 56 by synthesizing the three-dimensional image 52 and the object 54 generated by the object generation unit 48. Then, the synthesized image output unit 50 renders the synthesized image 56 on the screen 16A. Thus, the synthesized image 56 in which the object 54 and the three-dimensional image 52 are represented in a manner that allows comparison is displayed on the screen 16A of the display 16. After the process of step ST28 is executed, the inspection auxiliary process proceeds to step ST30.

[0132] In step ST30, the processor 20 determines whether the condition for terminating the inspection auxiliary process (hereinafter referred to as "termination condition") is satisfied. As an example of the termination condition, there can be cited a condition that the termination instruction signal from the receiving device 14 is input to the processor 20 by receiving the termination instruction from the inspector 6 through the receiving device 14. In step ST30, when the termination condition is not satisfied, it is determined that it is rejected, and the inspection auxiliary process proceeds to step ST22. In step ST30, when the termination condition is satisfied, it is determined that it is approved, and the inspection auxiliary process is terminated.

[0133] Next, refer to Fig.17 , an example of the flow of the first object generation process in the object generation process executed in the above-mentioned step ST26 is described.

[0134] exist Fig.17 In the first object generation process shown in FIG. 1 , first, in step ST40, the object generation unit 48 (see Fig.10 ) According to a plurality of pixels in the screen 16A on which the three-dimensional image 52 is rendered, a first two-dimensional coordinate corresponding to the start point of the object 54 and a second two-dimensional coordinate corresponding to the end point of the object 54 are acquired. After the process of step ST40 is executed, the first object generation process proceeds to step ST42.

[0135] In step ST42, the object generator 48 obtains a first three-dimensional coordinate corresponding to the first two-dimensional coordinate obtained in step ST40 and a second three-dimensional coordinate corresponding to the second two-dimensional coordinate obtained in step ST40 based on a plurality of pixels of the three-dimensional image 52 included in the inspection auxiliary information 74. After executing the process of step ST42, the first object generation process proceeds to step ST44.

[0136] In step ST44, the object generating unit 48 derives the inter-three-dimensional coordinate distance between the first three-dimensional coordinate and the second three-dimensional coordinate acquired in step ST42. After executing the process of step ST44, the first object generating process proceeds to step ST46.

[0137] In step ST46, the object generator 48 obtains the relationship between the distance between the three-dimensional coordinates derived in step ST44 and the first unit length based on the unit length information 72 included in the inspection support information 74. After executing the process of step ST46, the first object generation process proceeds to step ST48.

[0138] In step ST48, the object generation unit 48 derives the relationship between the distance between the three-dimensional coordinates and the second unit length based on the unit length information 72 included in the inspection auxiliary information 74 and the relationship between the distance between the three-dimensional coordinates and the first unit length derived in step ST46. After the process of step ST48 is executed, the first object generation process proceeds to step ST50.

[0139] In step ST50, the object generator 48 derives the length of the object 54 when it is assumed to be arranged in the real space based on the relationship between the distance between the three-dimensional coordinates derived in step ST48 and the second unit length. After executing the process of step ST50, the first object generation process proceeds to step ST52.

[0140] In step ST52, the object generator 48 generates an object 54 which is an image including a figure 58 extending between the first three-dimensional coordinates and the second three-dimensional coordinates and a numerical value 60 based on the length of the object 54. After the process of step ST52 is executed, the first object generation process ends.

[0141] Next, refer to Fig.18 , an example of the flow of the second object generation process in the object generation process executed in the above-mentioned step ST26 is described.

[0142] exist Fig.18 In the second object generation process shown in FIG. 1 , first, in step ST60, the object generation unit 48 (see Fig.11 ) Acquire the first two-dimensional coordinates corresponding to the starting point of the object 54 based on the plurality of pixels in the screen 16A on which the three-dimensional image 52 is rendered. After executing the process of step ST60, the second object generation process proceeds to step ST62.

[0143] In step ST62, the object generator 48 acquires first three-dimensional coordinates corresponding to the first two-dimensional coordinates acquired in step ST60 based on a plurality of pixels of the three-dimensional image 52 included in the inspection support information 74. After executing the process of step ST62, the second object generation process proceeds to step ST64.

[0144] In step ST64, the object generation unit 48 derives a three-dimensional virtual space 80 (see FIG. 1 ) corresponding to the direction from the start point to the end point of the object 54 and the length of the object 54 based on the unit length information 72 included in the inspection auxiliary information 74. Figure 6 ) on the virtual space distance. After executing the process of step ST64, the second object generation process enters step ST66.

[0145] In step ST66, the object generation unit 48 obtains a second three-dimensional coordinate separated by a virtual space distance in the direction from the start point to the end point of the object 54 from the first three-dimensional coordinate obtained in step ST62 based on a plurality of pixels of the three-dimensional image 52 included in the inspection auxiliary information 74. After executing the process of step ST66, the second object generation process proceeds to step ST68.

[0146] In step ST68, the object generator 48 generates the object 54 which is an image including the figure 58 extending between the first three-dimensional coordinates and the second three-dimensional coordinates and the numerical value 60 based on the length of the object 54. After the process of step ST68 is executed, the second object generation process ends.

[0147] In addition, the inspection support method described as the operation of the above-mentioned inspection support device 10 is an example of the “image processing method” involved in the technology of the present invention.

[0148] As described in detail above, in the inspection support device 10 according to the present embodiment, the processor 20 acquires a plurality of three-dimensional coordinates that determine the positions of a plurality of pixels included in the three-dimensional image 52 representing the object 4 in the real space and a plurality of two-dimensional coordinates that determine the positions corresponding to a plurality of pixels in the screen 16A on which the three-dimensional image 52 is rendered. In addition, the processor 20 acquires the unit length information 72 that indicates the relationship between the first unit length of the three-dimensional coordinate system that specifies the three-dimensional coordinates and the second unit length of the real space. Then, the processor 20 generates the object 54 that can determine the second unit length based on the plurality of three-dimensional coordinates, the plurality of two-dimensional coordinates, and the unit length information 72, and outputs a composite image 56 that represents the generated object 54 and the three-dimensional image 52 in a manner that allows comparison. Therefore, the user or the like can visually compare the object 54 that can determine the unit length of the real space and the three-dimensional image 52 through the composite image 56. Thus, the user (for example, the inspector 6) or the like can grasp the sense of the size of the object 4 in the real space.

[0149] The three-dimensional image 52 is an image generated from a plurality of two-dimensional images 51 obtained by photographing the object 4 from a plurality of photographing positions in the real space. Therefore, the object 4 can be represented by the three-dimensional image 52.

[0150] The unit length information 72 is information generated based on the distance between adjacent imaging positions among the plurality of imaging positions. Therefore, for example, the first unit length can be derived based on the principle of three-dimensional measurement.

[0151] Furthermore, the distance between adjacent imaging positions is the distance obtained by the positioning unit 106. Therefore, for example, the distance between adjacent imaging positions can be measured quickly and accurately compared to the case where the distance between adjacent imaging positions is measured manually.

[0152] Furthermore, the object 54 is an image generated based on the designated two-dimensional coordinates among the plurality of two-dimensional coordinates. Therefore, by designating the two-dimensional coordinates on the screen 16A, the object 54 can be arranged at the designated position on the screen 16A.

[0153] Furthermore, the object 54 is an image including a graph 58 and a numerical value 60 indicating a length related to the graph 58. Therefore, the user or the like can visually compare the object 4, the graph 58, and the numerical value 60 by synthesizing the image 56.

[0154] Furthermore, the processor 20 changes the first viewpoint for observing the three-dimensional image 52 through the screen 16A according to the given first instruction, and changes the second viewpoint for observing the object 54 through the screen 16A according to the first viewpoint. Therefore, even when the direction of the three-dimensional image 52 is changed by changing the first viewpoint, the direction of the object 54 can be changed according to the direction of the three-dimensional image 52.

[0155] Then, the processor 20 changes the third viewpoint from which the object 54 is viewed through the screen 16A according to the given second instruction. Therefore, the direction of the object 54 can be changed independently of the three-dimensional image 52 .

[0156] In addition, in the above embodiment, the object 54 includes a graphic 58 simulating a ruler. However, as an example, Fig.19 and Fig. 20 As shown, the object 54 may include an image 62 representing an object existing in real space instead of the graphic 58 or in addition to the graphic 58 .

[0157] exist Fig.19 In the example shown, the object represented by image 62 is a human being. Fig. 20In the example shown, the object represented by the image 62 is a drum can. In addition, the object represented by the image 62 can be any object such as a doll, a car, a bicycle, a motorcycle, a ladder or an inspection device.

[0158] And, in Fig.19 and Fig. 20 In the example shown, the numerical value 60 is included in the object 54 , but the numerical value 60 may be omitted if the image 62 is an image showing an object whose size can be visually grasped in advance (for example, a human being, etc.).

[0159] And, as an example, Fig.21 As shown, the object 54 only includes a graphic 58 , and other objects 66 including a numerical value 60 and a reference scale 64 may also be displayed in the corner of the screen 16A.

[0160] Furthermore, in the above-mentioned embodiment, the unit length information 72 is generated based on the relationship between the relative distance L1 and the absolute distance L2, and the unit length information 72 indicating the relationship between the first unit length and the second unit length is generated, wherein, regarding the distance between adjacent shooting positions, the relative distance L1 is defined by the coordinates of the three-dimensional coordinate system, and the absolute distance L2 is defined by the coordinates of the world coordinate system (refer to Figure 6 ). However, the unit length information 72 can also be generated by, for example, the following processing.

[0161] exist Fig. 22 In the example shown, a subject 68 is provided next to the object 4 in the real space, and the three-dimensional image 52 includes a subject image 69 that is a photograph of the subject 68. Fig. 22 In the example shown, the subject 68 is a rod-shaped object, but it may be an object in a shape other than a rod. Figure 6 ) in at least one of the two-dimensional images 51.

[0162] The length of the object 68 (equivalent to the length between the first point and the second point of the object image 69) is a known length, which is represented by a second unit length set in the real space. The length of the object 68 is received by the receiving device 14 after being designated by the inspector 6, and is outputted to the processor 20 by the receiving device 14.

[0163] The unit length information generating unit 36 ​​obtains a first two-dimensional coordinate corresponding to a first point of the subject image 69 and a second two-dimensional coordinate corresponding to a second point of the subject image 69 based on a plurality of pixels in the screen 16A on which the three-dimensional image 52 is rendered. The first point and the second point of the subject image 69 are determined based on, for example, an instruction from the inspector 6 received by the receiving device 14.

[0164] Next, the unit length information generating unit 36 ​​obtains first three-dimensional coordinates corresponding to the first two-dimensional coordinates and second three-dimensional coordinates corresponding to the second two-dimensional coordinates based on a plurality of pixels of the three-dimensional image 52 included in the three-dimensional image information 70 .

[0165] Next, the unit length information generating unit 36 ​​derives the inter-three-dimensional coordinate distance between the first three-dimensional coordinate and the second three-dimensional coordinate. The inter-three-dimensional coordinate distance is a distance represented by a first unit length set on the three-dimensional coordinate system.

[0166] Next, the unit length information generating unit 36 ​​generates unit length information 72 indicating the relationship between the first unit length and the second unit length based on the relationship between the length of the object 68 designated by the inspector 6 and the distance between the three-dimensional coordinates.

[0167] exist Fig. 22 In the example shown, the second unit length is equal to the length of the plurality of two-dimensional images 51 (reference Figure 6 ) is related to the length of the subject image 69 included in at least one of the two-dimensional images 51. Therefore, for example, even in the imaging device 100 (refer to Figure 3 ) Even when the positioning unit 106 is not installed, the unit length information 72 can be generated.

[0168] In addition, Fig. 22 In the example shown, the subject 68 is provided next to the object 4 , but the subject 68 may be, for example, a mark drawn on the wall surface of the object 4 (for example, a mark drawn with chalk).

[0169] Furthermore, in the above embodiment, the processor 20 is illustrated as an example, but at least one other CPU, at least one GPU and / or at least one TPU may be used instead of or together with the processor 20 .

[0170] Furthermore, in the above-mentioned embodiment, an example of a method in which the inspection auxiliary information generation program 30 and the inspection auxiliary program 40 are stored in the storage device 22 is given for explanation, but the technology of the present invention is not limited to this. For example, the inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 can be stored in a portable non-temporary computer-readable storage medium such as an SSD or a USB memory (hereinafter referred to as a "non-temporary storage medium"). The inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 stored in the non-temporary storage medium can also be installed in the computer 12 of the inspection auxiliary device 10.

[0171] In addition, the inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 may be stored in a storage device such as another computer or server device connected to the inspection auxiliary device 10 via a network, and the inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 may be downloaded and installed in the computer 12 in response to a request from the inspection auxiliary device 10.

[0172] Furthermore, it is not necessary to store the entire inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 in a storage device or storage device 22 of another computer or server device connected to the inspection auxiliary device 10, and a portion of the inspection auxiliary information generation program 30 and / or the inspection auxiliary program 40 may be stored.

[0173] Furthermore, although the computer 12 is built into the inspection supporting device 10 , the technology of the present invention is not limited thereto, and for example, the computer 12 may be provided outside the inspection supporting device 10 .

[0174] Furthermore, in the above embodiment, the computer 12 including the processor 20, the storage device 22, and the RAM 24 is illustrated, but the technology of the present invention is not limited thereto, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 12. Furthermore, a combination of a hardware structure and a software structure may be used instead of the computer 12.

[0175] Furthermore, as hardware resources for executing various processes described in the above embodiments, various processors shown below can be used. As a processor, for example, a general-purpose processor, i.e., a CPU, which functions as a hardware resource for executing various processes by executing software, i.e., a program, can be cited. Furthermore, as a processor, for example, a dedicated circuit can be cited, which is a processor having a circuit structure specially designed for executing specific processes, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor executes various processes by using the memory.

[0176] The hardware resources for executing various processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resources for executing various processing may be one processor.

[0177] As an example of a system composed of one processor, first, there is a system in which one processor is composed of a combination of one or more CPUs and software, and the processor functions as a hardware resource for executing various processes. Secondly, as represented by SoC, there is a system in which a processor is used to realize the functions of the entire system including multiple hardware resources for executing various processes by one IC chip. In this way, various processes are realized by using one or more of the above-mentioned various processors as hardware resources.

[0178] Moreover, as the hardware structure of these various processors, more specifically, a circuit composed of a combination of circuit elements such as semiconductor elements can be used. In addition, the above-mentioned sight line detection processing is only an example. Therefore, it is of course possible to delete unnecessary steps, add new steps, or switch the processing order within the scope of the present invention.

[0179] The above-described contents and illustrated contents are detailed descriptions of the parts involved in the technology of the present invention, and are only an example of the technology of the present invention. For example, the descriptions related to the above-described structure, function, action, and effect are descriptions related to an example of the structure, function, action, and effect of the parts involved in the technology of the present invention. Therefore, without departing from the scope of the main purpose of the technology of the present invention, it is of course possible to delete unnecessary parts from the above-described contents and illustrated contents, or to add or replace new elements. Furthermore, in order to avoid confusion and facilitate understanding of the parts involved in the technology of the present invention, the descriptions of technical common sense that does not need to be specifically described in the technology that can implement the present invention are omitted in the above-described contents and illustrated contents.

[0180] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. In addition, in this specification, when three or more situations are connected with "and / or", the same consideration method as "A and / or B" is also applicable.

[0181] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each individual document, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An image processing device comprising a processor, The processor performs the following processing: Acquire a plurality of three-dimensional coordinates that determine positions of a plurality of pixels included in a three-dimensional image representing an object in a real space and a plurality of two-dimensional coordinates that determine positions corresponding to the plurality of pixels in a screen on which the three-dimensional image is rendered; Acquire unit length information indicating a relationship between a first unit length of a three-dimensional coordinate system defining the three-dimensional coordinates and a second unit length of the actual space; generating an object capable of determining the second unit length according to the plurality of three-dimensional coordinates, the plurality of two-dimensional coordinates and the unit length information; and A first image is outputted which shows the object and the three-dimensional image in a manner that allows comparison between the two.

2. The image processing device according to claim 1, wherein: The three-dimensional image is an image generated from a plurality of two-dimensional images obtained by photographing the object from a plurality of photographing positions in the real space.

3. The image processing device according to claim 2, wherein: The unit length information is information generated based on the distance between adjacent imaging positions among the plurality of imaging positions.

4. The image processing device according to claim 3, wherein: The distance is a distance obtained by a positioning unit.

5. The image processing device according to claim 2, wherein: The second unit length is a length related to a subject image included in at least one of the plurality of two-dimensional images.

6. The image processing device according to any one of claims 1 to 5, wherein: The object is an image generated according to designated two-dimensional coordinates among the plurality of two-dimensional coordinates.

7. The image processing device according to any one of claims 1 to 6, wherein: The object is an image including a figure and a numerical value indicating a length related to the figure.

8. The image processing device according to any one of claims 1 to 7, wherein: The processor performs the following processing: changing a first viewpoint for observing the three-dimensional image through the screen according to a given first instruction; and A second viewpoint for observing the object through the screen is changed according to the first viewpoint.

9. The image processing device according to any one of claims 1 to 8, wherein: The processor changes a third viewpoint for observing the object through the screen according to a given second instruction.

10. The image processing device according to any one of claims 1 to 9, wherein: The object includes an image representing an object existing in the real space.

11. An image processing method comprising the following steps: Acquire a plurality of three-dimensional coordinates that determine positions of a plurality of pixels included in a three-dimensional image representing an object in a real space and a plurality of two-dimensional coordinates that determine positions corresponding to the plurality of pixels in a screen on which the three-dimensional image is rendered; Acquire unit length information indicating a relationship between a first unit length of a three-dimensional coordinate system defining the three-dimensional coordinates and a second unit length of the actual space; generating an object capable of determining the second unit length according to the plurality of three-dimensional coordinates, the plurality of two-dimensional coordinates and the unit length information; as well as A first image is outputted which shows the object and the three-dimensional image in a manner that allows comparison between the two.

12. A program for causing a computer to execute a process comprising the following steps: Acquire a plurality of three-dimensional coordinates that determine positions of a plurality of pixels included in a three-dimensional image representing an object in a real space and a plurality of two-dimensional coordinates that determine positions corresponding to the plurality of pixels in a screen on which the three-dimensional image is rendered; Acquire unit length information indicating a relationship between a first unit length of a three-dimensional coordinate system defining the three-dimensional coordinates and a second unit length of the actual space; generating an object capable of determining the second unit length according to the plurality of three-dimensional coordinates, the plurality of two-dimensional coordinates and the unit length information; as well as A first image is outputted which shows the object and the three-dimensional image in a manner that allows comparison between the two.