Image processing device, program, and image processing method

Abstract

A shape evaluation device (130) is provided with: an environment map generation unit (144) that generates an environment map image using first shape data indicating the shape of a first object, an initial environment map image, which is an initial image of an environment map image representing a surrounding environment in which an image of the first object is captured, and a trained model that predicts an environment map image such that a rendered image obtained by rendering a virtually captured image of the first object matches a real captured image, which is a captured image of the first object; and a display image generation unit (147) that generates a display image by rendering an image virtually captured of a second object in an environment indicated by the environment map image, using shape data of the second object and the environment map image.

Description

Image processing device, program, and image processing method

[0001] The present disclosure relates to an image processing device, a program, and an image processing method.

[0002] Numerical control (NC) machine tools perform machining based on machining programs created by computer-aided manufacturing (CAM) devices, etc. Workpieces machined by NC machine tools may have machining defects that indicate a finished product that differs from the intention of the machining program designer.

[0003] The movement of an NC tool is controlled by the operating parts of an NC machine tool, but its speed, range of movement, acceleration capability, and deceleration capability are limited. Therefore, the actual movement of the tool may not strictly follow the machining command of the NC machine tool. In such cases, the expected machining quality cannot be obtained due to incomplete cutting or excessive cutting, resulting in machining defects.

[0004] If a machining defect occurs, the machining parameters are reviewed, and the machining is carried out again to search for appropriate past conditions, a cycle of these is carried out.In recent years, advances in computer vision and computer graphics have led to the development of technology that can render certain shape data by simulating how it will actually appear based on the optical environment of the real world.

[0005] For example, Patent Document 1 describes a technique for reducing inaccuracies in the position of a reflection when real space is reflected on the surface of a virtual object.

[0006] Japanese Patent Application Laid-Open No. 2021-157458

[0007] However, because the lighting environment of the simulation differs from the user's lighting environment, it can be difficult to compare the virtual object with the real thing. In particular, in the case of metal, the color of the object changes significantly due to the reflection of the light source on the processed surface, making it difficult to accurately understand the simulation results.

[0008] Furthermore, the input of the lighting environment that simulates the real space can be performed by capturing a panoramic image. However, since it is difficult for all users to prepare photographic equipment that can capture panoramic images, it is not easy to align the environment viewed by the user with the simulation environment.

[0009] Therefore, one or more aspects of the present disclosure aim to easily reproduce a light source environment that matches the user's viewing environment.

[0010] An image processing device according to one aspect of the present disclosure is characterized in that it comprises: an environment map generation unit that generates an environment map image using first shape data indicating the shape of a first object, an initial environment map image that is an initial image of an environment map image that represents the surrounding environment in which the first object is imaged, and a trained model that predicts the environment map image so that a rendered image obtained by rendering an image obtained by virtually imaging the first object matches a real-life image that is an image obtained by imaging the first object; and a display image generation unit that generates a display image by rendering an image obtained by virtually imaging the second object in the environment indicated by the environment map image using second shape data indicating the shape of a second object and the environment map image.

[0011] A program according to one aspect of the present disclosure causes a computer to function as an environment map generation unit that generates an environment map image using first shape data indicating the shape of a first object, an initial environment map image that is an initial image of an environment map image representing the surrounding environment in which the first object is captured, and a trained model that predicts the environment map image so that a rendered image obtained by rendering an image of a virtually captured image of the first object matches a real-life image that is an image of the first object, and a display image generation unit that generates a display image by rendering an image of a virtually captured image of the second object in the environment shown in the environment map image using second shape data indicating the shape of a second object and the environment map image.

[0012] An image processing method according to one aspect of the present disclosure is characterized in that it generates an environment map image using first shape data indicating the shape of a first object, an initial environment map image that is an initial image of an environment map image representing the surrounding environment in which the object is captured, and a trained model that predicts the environment map image so that a rendered image obtained by rendering an image of a virtually captured image of the first object matches a real-life image that is an image of the first object, and generates a display image by using second shape data indicating the shape of a second object and the environment map image to render an image of the second object virtually captured in the environment shown in the environment map image.

[0013] According to one or more aspects of the present disclosure, a light source environment that matches the user's viewing environment can be easily reproduced.

[0014] FIG. 1 is a block diagram schematically showing the configuration of a machining system according to an embodiment. FIG. 2 is a side view schematically showing the configuration of a ball-end milling cutter. FIG. 3 is a perspective view showing a workpiece machined by the ball-end milling cutter. FIG. 4 is a block diagram schematically showing the configuration of a shape evaluation device. FIG. 5 is a schematic diagram for explaining a process for generating an environment map image. FIG. 6 is a schematic diagram showing an example of an environment map image. FIG. 7 is a block diagram schematically showing the configuration of a PC. FIG. 8 is a schematic diagram for explaining a general method for generating an environment map that has conventionally been used as a comparative example. FIG. 9 is a schematic diagram for explaining the operation of learning and generating an environment map image in this embodiment. FIG. 10 is a flowchart showing the operation of the shape evaluation device according to this embodiment.

[0015] Embodiments. The following describes embodiments. The following embodiments are merely examples and can be modified as appropriate.

[0016] 1 is a block diagram showing a schematic configuration of a machining system 100 according to an embodiment. The machining system 100 includes a CAM device 110, an NC machine tool (hereinafter also referred to as an "NC machining machine") 120, and a shape evaluation device 130.

[0017] The CAM device 110 creates a machining program in which operation commands to the NC machine tool 120 are written.

[0018] The NC machine tool 120 machines a workpiece based on a machining program created by the CAM device 110. In the example shown in Fig. 1, the NC machine tool 120 has a ball-end milling cutter 121 as a cutting tool, and a drive unit 122 that drives the ball-end milling cutter 121.

[0019] The ball-end milling cutter 121 is capable of forming a free-form surface on a workpiece. Figure 2 is a side view schematically showing the configuration of the ball-end milling cutter 121. The tip of the ball-end milling cutter 121 has a spherical shape, for example.

[0020] 1 , the drive unit 122 includes, for example, a motor and a transmission mechanism that transmits the driving force of the motor to the ball-end milling cutter 121. The transmission mechanism is, for example, a gear. Note that the NC machine tool 120 is not limited to the ball-end milling cutter 121, and may include other tools such as a drill.

[0021] Fig. 3 is a perspective view showing a workpiece 101 machined by the ball-end milling cutter 121 shown in Fig. 2. The workpiece 101 has a machined surface 102 machined by the ball-end milling cutter 121. In the example shown in Fig. 3, the machined surface 102 has a plurality of cutting surfaces 103A, 103B, and 103C.

[0022] The ball-end milling cutter 121 forms arc-shaped cutting surfaces 103A, 103B, and 103C on the workpiece 101. Triangular apexes 104A and 104B are formed between adjacent cutting surfaces among the multiple cutting surfaces 103A, 103B, and 103C.

[0023] Next, processing defects that occur in a machined workpiece will be described. In a workpiece machined by the ball-end milling cutter 121 shown in Figure 2, processing defects may occur in the machined surface and shape of the workpiece due to defects in the cutting path, etc. If processing defects occur, the product cannot be shipped, so it is desirable to prevent processing defects from occurring in the workpiece.

[0024] Generally, there are two types of processing defects: "scratches" and "uneven processing." A "scratch" is, for example, a groove with a minute depth formed on the processed surface of a workpiece. The depth of the groove is, for example, about 10 μm. In other words, a "scratch" is a processing defect that occurs when a step is formed on a part of the processed surface, which is normally a continuous surface, after processing. The presence or absence of a "scratch" can be recognized by the irregularity of the shape of the processed surface, and can be determined by a person touching or observing the processed surface.

[0025] "Processing irregularities" are, for example, discontinuous irregularities with minute heights formed on the processed surface of a processed product. The height of the irregularities is, for example, about 1 μm. "Processing irregularities" are processing defects in which the texture of the processed surface varies over a wide area. In other words, "processing irregularities" are processing defects in which the shape of the processed product is not homogenized, and the color or pattern of some parts of the processed surface appears different from the color or pattern of other parts. The presence or absence of "processing irregularities" can be determined by the appearance of rainbow-based color or shape irregularities on the processed surface due to light diffraction.

[0026] Generally, the presence or absence of "scratches" and "machining irregularities" is visually evaluated by the user of the NC processing machine. For example, the user performs prototype machining of a machined product and checks for the presence or absence of machining defects by shining indoor lighting or sunlight on the machined surface of the prototype machined product and observing the reflected light.

[0027] When a user determines that a machining defect exists, the machined product cannot be shipped. Therefore, the user works to modify the machining parameters in the machining program. However, the machining preparation process, which includes prototype machining and the modification of the machining parameters, takes a lot of time. Therefore, it is desirable to reduce the machining preparation process and improve the efficiency of the machining work.

[0028] Next, we will explain how humans recognize the texture of an object's surface. In a real environment, the reflected light of light irradiated onto an object changes depending on the material of the object, the shape of the object's surface, the characteristics of the light source located around the object, and the like. For example, humans understand the material of an object from the minute irregularities, gloss, or changes in shading on the object's surface that appear due to changes in reflected light. This allows humans to recognize the surface properties of an object, in other words, the texture of the object.

[0029] Here, the "texture" in this embodiment refers to a material characteristic that humans perceive from the reflected light reflected on the surface of an object (for example, the processed surface of a workpiece). The reflected light has characteristics such as wavelength characteristics (hereinafter also referred to as "color"), directionality, and diffusion.

[0030] Furthermore, when observing the surface of an object, a person actively moves their eyes, head, or body to move the point light or the object so that the point light shines on the observed surface. The gloss or shading of the object's surface changes based on such movements during observation, making it easier for people to understand the material of the object.

[0031] Therefore, if the observer arbitrarily moves the virtual camera position or lighting position during observation, the image of the virtual workpiece (hereinafter also referred to as "object") in the simulation will change, making it possible to display a simulation of the machined surface of the workpiece actually machined by the NC processing machine. The virtual camera position or lighting position may not only be operated by the observer using a mouse or the like during observation, but may also be moved in response to the observer's movements during observation, thereby changing the image of the virtual workpiece.

[0032] Light reflected from the surface of an object includes specularly reflected light components and diffusely reflected light components. Specular reflection refers to reflection in which the angle of incidence and the angle of reflection of light are the same. Diffuse reflection refers to incident light being reflected in various directions from the reflecting surface. Therefore, specularly reflected light components are highly directional, while diffusely reflected light components are reflected in all directions.

[0033] Furthermore, the light reflected from the surface of an object further includes internally reflected light components and inter-reflected reflected light components. Internal reflection occurs when light propagating inside an object is scattered within the object and reflected back toward the outside of the object. Inter-reflection occurs when light reflected from an object hits another object and is further reflected.

[0034] The shape evaluation device 130 is a device that simulates the machining of the workpiece by the NC machine tool 120. The configuration of the shape evaluation device 130 will be described later. If the color of the reflected light added to the image of the virtual workpiece displayed on the display unit (described later) of the shape evaluation device 130 is changed, the realism of the workpiece can be further enhanced. The change in the color of the reflected light can be achieved, for example, by generating a reflection in the image that corresponds to the color of the light illuminating the display unit and a landscape including objects or people present around the display unit.

[0035] 4 is a block diagram showing a schematic configuration of a shape evaluation device 130 as an image processing device. The shape evaluation device 130 includes an input unit 131, a storage unit 132, a display unit 133, and a control unit 140. The shape evaluation device 130 is a device capable of executing an image processing method and a program according to an embodiment.

[0036] The input unit 131 accepts input of various instructions. The storage unit 132 stores programs and data necessary for processing in the shape evaluation device 130. For example, the storage unit 132 stores a trained model, which will be described later.

[0037] The display unit 133 displays various images. For example, the display unit 133 displays a simulation image generated by the control unit 140. The simulation image is a display image of a virtual workpiece processed by a processing simulation.

[0038] The control unit 140 controls the processing in the shape evaluation device 130. The control unit 140 includes a shape data setting unit 141, an illumination condition setting unit 142, a material information setting unit 143, an environment map generation unit 144, an environment map setting unit 145, a virtual camera setting unit 146, and a display image generation unit 147.

[0039] The shape data setting unit 141 sets shape data indicating the shape of the workpiece. For example, the shape data setting unit 141 acquires the shape data by receiving shape data input from the user via the input unit 131. Then, the shape data setting unit 141 sets, for example, three-dimensional volume data as the shape data. Here, in this embodiment, the shape data setting unit 141 sets first shape data indicating the shape of a first object, which is an object used to generate an environment map image (described later), and second shape data indicating the shape of a second object, which is the workpiece. The second object is, for example, an object that is drawn using the generated environment map, its appearance is reproduced, and its appearance is displayed to the user.

[0040] The set shape data is stored in the storage unit 132 of the shape evaluation device 130. The shape data may be, for example, data obtained by measuring the shape of an actually machined workpiece using a three-dimensional shape measuring device (not shown). The first shape data FFD is provided to the environment map generation unit 144, and the second shape data SFD is provided to the display image generation unit 147.

[0041] The illumination condition setting unit 142 sets illumination conditions, which are conditions for illuminating the workpiece indicated by the second shape data set by the shape data setting unit 141. In other words, the illumination condition setting unit 142 sets illumination conditions when virtually capturing an image of the workpiece. For example, the illumination condition setting unit 142 acquires the illumination conditions by receiving input of the illumination conditions from a user via the input unit 131. The illumination conditions include, for example, at least one of the position coordinates from which a virtual light source is emitted, the irradiation position, irradiation angle, light intensity, light distribution, color and wavelength characteristics of the light when the light illuminates the workpiece surface, and the number of virtual light sources.

[0042] The illumination condition setting unit 142 provides the set illumination conditions LC to the display image generation unit 147. In the present embodiment, the illumination condition setting unit 142 acquires the illumination conditions via the input unit 131, but the present embodiment is not limited to this example. For example, predetermined illumination conditions may be stored in the storage unit 132. The storage unit 132 may store a plurality of predetermined illumination conditions. For example, when the material of the workpiece is metal, the reflection characteristics of the light irradiated onto the workpiece surface differ depending on the illumination conditions. The illumination condition setting unit 142 can acquire illumination conditions corresponding to the material or shape of the workpiece by accepting a selection from the plurality of stored illumination conditions via the input unit 131.

[0043] The material information setting unit 143 sets material information indicating the material of the workpiece indicated by the second shape data. The set material information MI is provided to the display image generating unit 147. In the present embodiment, the material information setting unit 143 sets material information indicating the material of the workpiece as the material information. The material of the workpiece includes, for example, any of metal (e.g., aluminum, iron, titanium, etc.), resin (e.g., plastic), wood, and rubber.

[0044] A plurality of predetermined pieces of material information are stored in the storage unit 132, and the material information setting unit 143 acquires the selected material information as material information by accepting a selection from the plurality of stored pieces of material information via the input unit 131. Note that if material information for the workpiece is not stored in the storage unit 132, the material information setting unit 143 may accept an input of new material information as material information via the input unit 131.

[0045] Furthermore, the material information setting unit 143 is not limited to acquiring material information, and may acquire a texture image including information about the object color of the workpiece or information about the specular reflection intensity as the material information, or may acquire BRDF (Bidirectional Reflectance Distribution Function) data as the material information.

[0046] The environment map generation unit 144 generates an environment map image that shows a virtual space in which an image of a workpiece is captured. Here, the environment map generation unit 144 generates the environment map image using first shape data that shows the shape of a first object, an initial environment map image that is an initial image of the environment map image that shows the surrounding environment in which the first object is captured, and a trained model that predicts the environment map image so that a rendered image obtained by rendering a virtually captured image of the first object matches a real image that is an captured image of the first object. Specific processing will be described later.

[0047] FIG. 5 is a schematic diagram illustrating the process of generating an environment map image. For example, the environment map generation unit 144 acquires a live-action image 105, which is an image of an actual first object having the shape indicated by the first shape data, and an initial environment map image 106, which is an initial image of the environment map image. Specifically, the environment map generation unit 144 may acquire the live-action image 105 and the initial environment map image 106 via a communication unit (not shown). If these images are pre-stored in the storage unit 132, the environment map generation unit 144 may acquire these images from the storage unit 132. The initial environment map image 106 may be any image as long as its resolution matches that of the environment map image. As will be described later, a panoramic image is generally used as the environment map image. However, in this example, a normal RGB (Red, Green, Blue) image may also be used. Here, a blank image is used as the initial environment map image 106.

[0048] The environment map generation unit 144 then inputs the rendering image RI, which is rendered using the first shape data and the initial environment map image 106, into the learned model stored in the storage unit 132, and obtains the environment map image 107 as its output. The detailed process of generating the environment map image will be described later. Since the environment map generation unit 144 renders the rendering image using the first shape data and the initial environment map image 106, it is desirable that the shape and light reflection characteristics of the first object are known. The shape of the first object may be any shape, such as a sphere or a rectangle.

[0049] 6 is a schematic diagram showing an example of an environment map image 107. The environment map image 107 is, for example, a panoramic image captured by a spherical camera (not shown). The space shown in the environment map image 107 is reflected in a virtually processed workpiece by processing simulation. In other words, the environment map image 107 is a panoramic landscape image used to reproduce the reflection of a pseudo-surrounding environment on the surface of the three-dimensional shape of the workpiece.

[0050] Returning to FIG. 4, the environment map setting unit 145 sets the environment map image generated by the environment map generating unit 144 and provides the environment map image SMI to the display image generating unit 147 .

[0051] The virtual camera setting unit 146 sets camera position information indicating the position of a virtual camera that virtually replaces the eye of the user viewing the image via the display unit 133. In other words, the virtual camera setting unit 146 sets a position for virtually capturing an image of the workpiece represented by the second shape data. For example, in this embodiment, the virtual camera setting unit 146 acquires the camera position information via the input unit 131. The set camera position information VCP is provided to the display image generation unit 147.

[0052] In addition, if the display unit 133 is a display that can be touched (hereinafter also referred to as a "touch panel display"), the virtual camera setting unit 146 may acquire camera position information via a user's touch operation on the touch panel display.

[0053] The display image generation unit 147 generates a simulation image, which is an image of a virtual workpiece, to be provided to the display unit 133. For example, the display image generation unit 147 uses second shape data indicating the shape of the workpiece and the environment map image to render an image of the workpiece virtually captured in the environment indicated by the environment map image, thereby generating the simulation image as a display image.

[0054] Here, the display image generation unit 147 generates a simulation image based on the shape data, lighting conditions, material information, camera position information, and environment map image. Specifically, the display image generation unit 147 generates an image by rendering the shape data of the workpiece based on the lighting conditions, material information, camera position information, and observation state information. The image of the virtual workpiece is generated in a virtual three-dimensional space.

[0055] Specifically, the display image generation unit 147 generates a display image by rendering an image of a workpiece formed from the material set by the material information setting unit 143, virtually captured from a position set by the virtual camera setting unit 146 under the lighting conditions set by the lighting condition setting unit 142.

[0056] The shape evaluation device 130 described above can be realized by, for example, a computer such as the PC 10 shown in Fig. 7. The PC 10 includes a storage 11 such as a hard disk drive (HDD) and a solid state drive (SSD), a memory 12, a processor 13 such as a central processing unit (CPU), a communication interface (I / F) 14 such as a network interface card (NIC), an input interface 15 such as a keyboard and a mouse, and a display 16.

[0057] For example, the input unit 131 can be realized by the input I / F 15. The storage unit 132 can be realized by the storage 11 or the memory 12.

[0058] The display unit 133 can be realized by the display 16. The display 16 is, for example, a liquid crystal display, an organic electroluminescence (EL) display, a micro-LED (light emitting diode) display, etc. The display 16 may also be an aerial display, a head mounted display (HMD), a virtual reality (VR) device, an augmented reality (AR) device, etc.

[0059] The control unit 140 can be realized by loading a program stored in the storage 11 into the memory 12 and having the processor 13 execute the program.

[0060] The program may be downloaded to the storage 11 from a recording medium (not shown) via a reader / writer (not shown) or from a network via the communication I / F 14, and then loaded onto the memory 12 and executed by the processor 13. Alternatively, the program may be directly loaded onto the memory 12 from a recording medium via the reader / writer or from a network via the communication I / F 14, and then executed by the processor 13. In other words, the program may be provided by a program product such as a recording medium.

[0061] In the embodiment described above, the shape evaluation device 130 includes the input unit 131, the storage unit 132, the display unit 133, and the control unit 140, but the embodiment is not limited to this example. For example, the control unit 140 may be configured to operate on a cloud server, and the display unit 133 may be configured to display an image sent from the cloud server. Furthermore, the input unit 131 may be configured to input instructions to the cloud server. Furthermore, the storage unit 132 may be provided in the cloud server. In other words, the processing in the shape evaluation device 130 may be distributed and executed by multiple computers connected to a network. In such a case, a shape evaluation system is configured by multiple computers instead of the shape evaluation device 130.

[0062] 8 is a schematic diagram illustrating a conventional method for generating a general environment map as a comparative example. According to the conventional method, an environment map image is generated by capturing images using an omnidirectional camera, synthesizing a plurality of omnidirectional images 108#1, 108#2, and 108#3 whose exposure times during capture are varied at regular intervals, and generating an omnidirectional image 108 as an HDR (High Dynamic Range Imaging) image. The HDR image allows for the handling of luminance images with a wide dynamic range, improving the reproducibility of texture.

[0063] 9 is a schematic diagram for explaining the operation of learning and generating an environment map in this embodiment. To start learning, shape data 151, a real-life image 105 which is a photograph of an actual object having the same shape as the shape data 151, and an initial environment map image 106 are prepared.

[0064] The object captured in the real image 105 may have any shape, for example, a cubic block or a spherical object. In this case, it is desirable that the material and color or reflection characteristics of the object are known.

[0065] First, the environment map generating unit 144 uses the initial environment map image 106 to generate a rendering image 109#0 from the shape data 151 based on a rendering function so that the angle is the same as that of the actual image 105.

[0066] Next, the environment map generation unit 144 calculates the squared error between the generated rendering image 109#0 and the real-life image 105. Then, the environment map generation unit 144 propagates the error to the rendering function and optimizes the initial environment map image 106#0 to generate an optimized environment map image 106#1.

[0067] Similarly, the environment map generation unit 144 uses the optimized environment map image 106#1, which is an optimized environment map image, to generate a rendering image 109#1 based on a rendering function from the shape data 151 so that the angle is the same as that of the actual image 105.

[0068] Next, the environment map generation unit 144 calculates the squared error between the generated rendering image 109#1 and the real image 105. Then, the environment map generation unit 144 propagates the error to the rendering function and optimizes the optimized environment map image 106#1 to generate an optimized environment map image 106#2.

[0069] The environment map generation unit 144 repeats the above process to generate the final environment map image 107. In other words, the environment map generation unit 144 performs a process of acquiring an optimized environment map image 106#1, which is an image predicted by the trained model, so that the rendering image 109#0 generated by performing rendering using the initial environment map image 106 and the first shape data approaches the actual image 105, and then repeats a process of updating the optimized environment map images 106#1, 106#2, ... using the images predicted by the trained model so that the rendering images 109#1, ... generated by performing rendering using the optimized environment map images 106#1, 106#2, ... and the first shape data approach the actual image 105, thereby generating the environment map image 107.

[0070] Here, the number of times of learning to optimize the environment map image may be any number of times, and the environment map generation unit 144 may perform learning until the squared error becomes equal to or less than a certain value. Furthermore, the environment map generation unit 144 may generate a rendering image based on not only one angle, but also actual photographs taken from multiple angles.

[0071] 10 is a flowchart showing the operation of the shape evaluation device 130 according to this embodiment. First, the environment map generation unit 144 acquires first shape data, image data of the real image, and image data of the initial environment map image (S10).

[0072] Next, as explained using Figure 9, the environment map generation unit 144 generates an environment map image using the trained model stored in the memory unit 132 so that the rendered image rendered using the first shape data and the initial environment map image matches the actual image (S11).

[0073] Next, the shape data setting unit 141 sets the second shape data to the display image generation unit 147, the lighting condition setting unit 142 sets the lighting conditions to the display image generation unit 147, the material information setting unit 143 sets the material information to the display image generation unit 147, the environment map setting unit 145 sets the glasses map image generated in step S11 to the display image generation unit 147, and the virtual camera setting unit 146 sets the camera position information to the display image generation unit 147 (S12).

[0074] Next, the display image generating unit 147 uses the second shape data and the environment map image to render an image of the workpiece virtually captured in the environment shown in the environment map image, thereby generating a simulation image as a display image (S13). The generated display image is sent to the display unit 133 and displayed.

[0075] As described above, the shape evaluation device 130 according to the embodiment can generate an environment map image that simulates a real space by capturing an image of a known object with a general RGB camera. The shape evaluation device 130 then performs rendering using this environment map image, thereby reproducing the color of light in the actual environment and glare reflections on metal, improving the reproducibility of graphics. Furthermore, by capturing an image of an object whose shape or reflection characteristics are known, an environment map image of the actual environment can be obtained with high accuracy.

[0076] Furthermore, by using the generated environment map image to generate a rendering image, it is possible to generate an image with high reproducibility compared to the real thing. Therefore, even for objects that do not exist in reality, it is possible to simulate in advance how they will appear in the real world in the virtual space.

[0077] In the embodiment described above, the shape evaluation device 130 includes the illumination condition setting unit 142, the material information setting unit 143, and the virtual camera setting unit 146. However, the embodiment is not limited to this example. For example, if the illumination conditions are constant, the illumination condition setting unit 142 may not be provided. Also, if the material of the workpiece is known, the material information setting unit 143 may not be provided. Furthermore, if the position of the virtual camera is fixed, the virtual camera setting unit 146 may not be provided. In other words, none of the settings of the illumination condition setting unit 142, the material information setting unit 143, and the virtual camera setting unit 146 may be used, or only one of them may be used, or any combination thereof may be used.

[0078] 100 Machining system, 110 CAM device, 120 NC machine tool, 130 Shape evaluation device, 131 Input unit, 132 Memory unit, 133 Display unit, 140 Control unit, 141 Shape data setting unit, 142 Lighting condition setting unit, 143 Material information setting unit, 144 Environment map generation unit, 145 Environment map setting unit, 146 Virtual camera setting unit, 147 Display image generation unit.

Claims

1. An image processing device comprising: an environment map generation unit that generates an environment map image using first shape data indicating a shape of a first object, an initial environment map image that is an initial image of an environment map image that represents the surrounding environment in which the first object is captured, and a trained model that predicts the environment map image so that a rendered image obtained by rendering an image obtained by virtually capturing the first object matches a real-life image that is an image obtained by capturing the first object; and a display image generation unit that generates a display image by using second shape data indicating a shape of a second object and the environment map image, and rendering an image obtained by virtually capturing the second object in the environment represented by the environment map image.

2. The image processing device of claim 1, wherein the environment map generation unit generates the environment map image by: performing a process of obtaining an optimized environment map image, which is an image predicted by the trained model, so that a rendering image generated by rendering using the initial environment map image and the first shape data approaches the real-life image; and then repeating a process of updating the optimized environment map image using the image predicted by the trained model, so that a rendering image generated by rendering using the optimized environment map image and the first shape data approaches the real-life image.

3. An image processing device according to claim 1 or 2, characterized in that the initial environment map image is an arbitrary image having the same resolution as the environment map image.

4. An image processing device according to any one of claims 1 to 3, characterized in that the shape and light reflection characteristics of the first object are known.

5. An image processing device as claimed in any one of claims 1 to 4, further comprising a lighting condition setting unit which sets lighting conditions when virtually capturing an image of the second object, and wherein the display image generating unit generates the display image by rendering an image virtually captured under the lighting conditions in an environment in which the second object is indicated in the environment map image.

6. An image processing device as claimed in any one of claims 1 to 4, further comprising a material information setting unit which sets the material of the second object, and wherein the display image generating unit generates the display image by rendering an image of the second object formed from the material virtually captured in an environment indicated by the environment map image.

7. An image processing device as claimed in any one of claims 1 to 4, further comprising a virtual camera setting unit which sets a position for virtually capturing an image of the second object, and wherein the display image generating unit generates the display image by rendering an image of the second object captured virtually from the position.

8. An image processing device as claimed in any one of claims 1 to 4, further comprising: a lighting condition setting unit which sets lighting conditions when virtually capturing an image of the second object; and a material information setting unit which sets a material of the second object, wherein the display image generating unit generates the display image by rendering an image of the second object formed of the material captured virtually under the lighting conditions.

9. An image processing device as described in any one of claims 1 to 4, further comprising: a lighting condition setting unit that sets lighting conditions when virtually capturing an image of the second object; and a virtual camera setting unit that sets a position for virtually capturing an image of the second object, wherein the display image generating unit generates the display image by rendering an image of the second object virtually captured from the position under the lighting conditions.

10. An image processing device as described in any one of claims 1 to 4, further comprising: a material information setting unit that sets the material of the second object; and a virtual camera setting unit that sets a position for virtually capturing an image of the second object, wherein the display image generating unit generates the display image by rendering an image of the second object formed from the material captured virtually from the position.

11. An image processing device as described in any one of claims 1 to 4, further comprising: a lighting condition setting unit which sets lighting conditions when virtually capturing an image of the second object; a material information setting unit which sets the material of the second object; and a virtual camera setting unit which sets a position at which the second object is virtually captured, wherein the display image generating unit generates the display image by rendering an image of the second object formed of the material virtually captured from the position under the lighting conditions.

12. A program causing a computer to function as: an environment map generation unit that generates an environment map image using first shape data indicating a shape of a first object, an initial environment map image that is an initial image of an environment map image representing the surrounding environment in which the first object is captured, and a trained model that predicts the environment map image so that a rendered image obtained by rendering an image obtained by virtually capturing the first object matches a real-life image that is an image obtained by capturing the first object; and a display image generation unit that generates a display image by using second shape data indicating a shape of a second object and the environment map image, and rendering an image obtained by virtually capturing the second object in the environment represented by the environment map image.

13. An image processing method comprising: generating an environment map image using first shape data indicating a shape of a first object, an initial environment map image which is an initial image of an environment map image representing the surrounding environment in which the object is captured, and a trained model which predicts the environment map image so that a rendered image obtained by rendering an image obtained by virtually capturing the first object matches a real-life image which is an image obtained by capturing the first object; and generating a display image by using second shape data indicating a shape of a second object and the environment map image to render an image obtained by virtually capturing the second object in the environment represented by the environment map image.

Images