Image Processing Apparatus, Image Processing Method, and Storage Medium

Abstract

The present invention discloses an image processing apparatus, an image processing method, and a storage medium. The image processing apparatus includes: an acquisition unit configured to acquire target data; a determination unit configured to determine whether the target data has material appearance information of an object; and a display control unit configured to display, based on a determination result of the determination unit, an image corresponding to the target data on a display unit, wherein, when the target data has the material appearance information of the object, the display control unit displays, on the display unit, a moving image or a series of still images corresponding to the target data and including a plurality of images under different observation conditions.

Description

Technical Field

[0001] The present invention relates to an image processing technique for representing the material appearance of an object. Background Art

[0002] With the widespread use of the Internet, there is an increasing need to display not only the color and shape of a product but also the material appearance of the product on a personal computer (PC) screen, a tablet terminal screen, or a smartphone screen. Various three-dimensional computer graphics (3DCG) methods have been used to represent the material appearance, coating, etc. of an object. Japanese Patent Application Laid-Open No. 2005-122719 discusses a technique for generating a two-dimensional image representing the material appearance of an object based on the light source position, the object position, or the viewpoint position.

[0003] However, in the case where it is desired to examine the material appearance of an object under multiple observation conditions, according to the technique of Japanese Patent Application Laid-Open No. 2005-122719, every time the user changes the observation conditions, the user is required to input light source information, viewpoint information, etc. Summary of the Invention

[0004] An object of the present invention is to provide a process for presenting the material appearance of an object under multiple observation conditions without requiring the user to input information to change the observation conditions.

[0005] According to one aspect of the present invention, there is provided an image processing apparatus, an image processing method, and a storage medium. The image processing apparatus includes: an acquisition unit configured to acquire target data; a determination unit configured to determine whether the target data has material appearance information of an object; and a display control unit configured to display an image corresponding to the target data on a display unit based on a determination result of the determination unit, wherein, when the target data has the material appearance information of the object, the display control unit displays a moving image or a series of still images including a plurality of images under different observation conditions corresponding to the target data on the display unit.

[0006] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0007] Figure 1 A block diagram showing the hardware configuration of the image processing apparatus.

[0008] Figure 2 A block diagram showing the functional configuration of the image processing apparatus.

[0009] Figure 3 A flowchart showing the process executed by the image processing apparatus.

[0010] Figure 4A andFigure 4B Figures showing user interface examples.

[0011] Figure 5 A block diagram showing the functional configuration of an image processing apparatus.

[0012] Figure 6 Figures showing user interface examples.

[0013] Figure 7 A flowchart showing the processing performed by the image processing apparatus. Detailed implementation manners

[0014] Embodiments of the present invention will be described below with reference to the accompanying drawings. The following exemplary embodiments do not necessarily limit the present invention. All combinations of the features described in the exemplary embodiments are not all necessary for solving the problems of the present invention.

[0015] <Hardware configuration of the image processing apparatus>

[0016] Figure 1 A block diagram showing the hardware configuration of the image processing apparatus 1 according to the first exemplary embodiment. The image processing apparatus 1 includes a central processing unit (CPU) 101, a read-only memory (ROM) 102, and a random access memory (RAM) 103. The image processing apparatus 1 further includes a video card (VC) 104, a general-purpose interface (I / F) 105, a serial advanced technology attachment (SATA) interface 106, and a network interface card (NIC) 107. The CPU 101 uses the RAM 103 as a working memory to execute an operating system (OS) and various programs stored in the ROM 102, a hard disk drive (HDD) 112, etc. The CPU 101 controls each component element via a system bus 108. The processing in the flowcharts described below is implemented by the CPU 101 loading program codes stored in the ROM 102, the HDD 112, etc. into the RAM 103 and executing the program codes. A display 114 is connected to the VC 104. An input device 110 (such as a mouse and a keyboard) is connected to the general-purpose interface 105 through a serial bus 109. A general-purpose drive 113 that performs reading and writing from / to the HDD 112 or various recording media is connected to the SATA interface 106 through a serial bus 111. The NIC 107 inputs and outputs information to / from an external device. The CPU 101 uses various recording media installed on the HDD 112 or the general-purpose drive 113 as storage locations for various data. The CPU 101 displays a graphical user interface (GUI) provided by a program on the display 114 and receives inputs such as user instructions received via the input device 110.

[0017] <Functional configuration of the image processing apparatus>

[0018] Figure 2 A block diagram showing the functional configuration of the image processing apparatus 1. The CPU 101 reads and executes programs stored in the ROM 102 or the HDD 112 using the RAM 103 as a working memory, and thus functions as Figure 2 the functional configuration shown. Not all of the processes described below must be executed by the CPU 101. The image processing apparatus 1 may have a configuration in which one or more processing circuits other than the CPU 101 execute part or all of the processes.

[0019] The image processing apparatus 1 includes an acquisition unit 201, a determination unit 202, a generation unit 203, and a display control unit 204. The acquisition unit 201 acquires data having material appearance information of an object or two-dimensional still image data as target data. In the present exemplary embodiment, the data including the material appearance information is data in a three-dimensional (3D) model format and is described in the Wavefront object (OBJ) format. When performing rendering using the data in the Wavefront OBJ format, the image processing apparatus 1 refers to a file in the Material Template Library (MTL) format as a 3D texture format. The two-dimensional still image data is image data in the Joint Photographic Experts Group (JPEG) format.

[0020] The determination unit 202 determines whether the target data acquired by the acquisition unit 201 is data having material appearance information of an object. The generation unit 203 generates moving image data in the Moving Picture Experts Group (MPEG) 4 format or the like including a plurality of frames under different viewing conditions for the target data determined by the determination unit 202 to have material appearance information of an object. The display control unit 204 displays (automatically reproduces) the moving image data including a plurality of frames under different viewing conditions, generated by the generation unit 203, for the target data determined by the determination unit 202 to have material appearance information. The display control unit 204 displays the target data determined by the determination unit 202 not to have material appearance information as a two-dimensional still image.

[0021] <Processing flow executed by the image processing apparatus>

[0022] The processing executed by the image processing apparatus 1 will be described in detail below with reference to Figure 3 The processing executed by the image processing apparatus 1 will be described in detail below with reference to Figure 3The processing shown in the flowchart in starts with the user inputting an instruction via the input device 110 and the CPU 101 receiving the input instruction. In the following description, each step (process) is represented by appending S before the corresponding number. In step S301, the acquisition unit 201 acquires target data. The acquisition unit 201 in this exemplary embodiment acquires four types of target data from a storage device such as the HDD 112 according to the input instruction from the user.

[0023] In step S302, the acquisition unit 201 sets the material appearance flag (flg) of a one-bit signal to OFF (flg = 0) for the target data. The material appearance flag is given to each target data acquired by the acquisition unit 201. In step S303, the determination unit 202 determines whether the target data acquired in step S301 is data with material appearance information. When the determination unit 202 determines that the target data is data with material appearance information (Yes in step S303), the processing proceeds to step S304. In step S304, the determination unit 202 sets (updates) the material appearance flag corresponding to the determined target data to ON (flg = 1).

[0024] In step S305, the generation unit 203 generates moving image data including a plurality of frames under different observation conditions for the target data whose material appearance flag is set to ON. Known methods such as bump mapping, ray tracing, or photon mapping can be used to generate the moving image data. When the determination unit 202 determines that the target data is not data with material appearance information (No in step S303), the processing skips steps S304 and S305.

[0025] In step S306, the display control unit 204 displays a moving image represented by moving image data or a still image represented by still image data based on the material appearance flag corresponding to the target data. When the material appearance flag of the target data is set to ON (flg = 1), the display control unit 204 displays, on the display 114, a moving image represented by the moving image data generated in step S305. When the material appearance flag of the target data is set to OFF (flg = 0), the display control unit 204 displays, as it is, the two-dimensional still image represented by the still image data acquired in step S301 on the display 114. In step S307, the display control unit 204 determines whether the display regarding all the acquired target data has been completed. If the display regarding all the acquired target data has not been completed (No in step S307), the processes in step S302 and subsequent steps are executed for the next target data. If the display regarding all the acquired target data has been completed (Yes in step S307), the process ends.

[0026] <Displayed by the display control unit>

[0027] Refer to Figure 4A and Figure 4B The processes performed by the display control unit 204 are described in detail. Figure 4A and Figure 4B FIG. is an example of a user interface (UI) displayed on the display 114. The window 401 is a display area for displaying an image. The display images 402 to 405 are images of the display regarding four types of target data acquired in step S301. In Figure 4A and Figure 4B In the example shown, four types of objects with different display colors and material appearances are shown, and the highlight areas on the surface of each object indicate the light reflected on the object surface. Figure 4A The display result at time A is shown. Figure 4B The display result at time B, which is different from time A, is shown. The display images 402 and 405 are display results corresponding to the target data determined to have material appearance information in step S303 and are moving images including multiple frames under different observation conditions. The display images 403 and 404 are display results corresponding to the target data determined not to have material appearance information in step S303 and are two-dimensional still images. By comparing Figure 4A and Figure 4B the display images 402 and 405 between, the positions of the highlight areas on the corresponding object surfaces are changed according to the display time. This is because the results of rendering performed by changing the observation conditions (lighting direction) are displayed according to the display time.

[0028] The display control unit 204 synchronizes the reproduction start frame and the frame rate in the moving image to be displayed, and displays the moving image under the same viewing conditions. This enables comparison of the colors and material appearances between different objects under multiple viewing conditions. The button 406 is used to switch between the moving image and the still image. When the pressing of the button 406 is accepted, the display control unit 204 switches between the reproduction and the stop of the reproduction of the displayed moving image.

[0029] <Effect>

[0030] As described above, the image processing apparatus 1 according to the first exemplary embodiment acquires target data and determines whether the target data has material appearance information of an object. The image processing apparatus 1 displays an image corresponding to the target data based on the determination result. In a case where the target data has material appearance information of an object, the image processing apparatus 1 displays a moving image including a plurality of images under different viewing conditions and corresponding to the target data. This configuration enables presentation of the material appearance of an object under multiple viewing conditions without user input for changing the viewing conditions.

[0031] <Modification example>

[0032] In the present exemplary embodiment, when the illumination direction is changed as a viewing condition, any one of the conditions regarding the distance between the illumination and the object, the direction of the object, the viewing direction, and the viewing distance can be changed as long as the condition is a parameter that can be changed in computer graphics (CG) rendering. In addition, a plurality of conditions can be changed in a combined manner.

[0033] In the first exemplary embodiment, the image processing apparatus 1 generates and automatically reproduces a moving image including a plurality of frames under different viewing conditions for data having material appearance information. In the second exemplary embodiment, the image processing apparatus 1 sets viewing conditions for data having material appearance information and reproduces a moving image based on the set viewing conditions. Since the hardware configuration of the image processing apparatus 1 according to the second exemplary embodiment is similar to that of the first exemplary embodiment, the description thereof is omitted. The parts different between the present exemplary embodiment and the first exemplary embodiment will be mainly described. The constituent elements identical to those of the first exemplary embodiment are denoted by the same reference numerals, and the description thereof is omitted.

[0034] <Functional configuration of the image processing apparatus>

[0035] Figure 5is a block diagram showing the functional configuration of the image processing apparatus 1. The image processing apparatus 1 includes an acquisition unit 201, an observation condition setting unit 501, a determination unit 202, a generation unit 203, and a display control unit 204. The observation condition setting unit 501 sets the manner of changing the observation conditions. The processing of the observation condition setting unit 501 will be described in detail below with reference to Figure 6 The processing of the observation condition setting unit 501 will be described in detail.

[0036] Figure 6 is a diagram showing an example of a UI displayed to enable setting of observation conditions. The window 601 is a setting area for setting observation conditions. The area 602 is for setting the lighting direction, the area 603 is for setting the direction of the object, the area 604 is for setting the observation direction, and the area 605 is for setting the observation distance.

[0037] When setting the manner of changing the lighting direction in the area 602, the user presses the start button in the area 602, performs a drag operation using a mouse, finger, etc. to continuously move the light source 606 in a freely selected direction, and then presses the stop button in the area 602. When setting the manner of changing the direction of the object in the area 603, the user presses the start button in the area 603, performs a drag operation using a mouse, finger, etc. to continuously rotate the coordinate axis 607 of the object in a freely selected direction, and then presses the stop button in the area 603. When setting the manner of changing the observation direction in the area 604, the user presses the start button in the area 604, performs a drag operation using a mouse, finger, etc. to continuously move the camera 608 indicating the observation position in a freely selected direction, and then presses the stop button in the area 604. When setting the manner of changing the observation distance in the area 605, the user presses the start button in the area 605, performs a drag operation using a mouse, finger, etc. to change the position of the slider 609 indicating the observation distance at a freely selected position, and then presses the stop button in the area 605.

[0038] In response to the pressing of the start button and the stop button, the observation condition setting unit 501 stores the respective changes in the observation conditions indicated by the user within a period along the time axis. When the complete setting button on the window 601 is pressed, the observation condition setting unit 501 completes the setting of the observation conditions. One or more observation conditions can be set. The initial values are applied as they are to the unchanged items.

[0039] <Processing flow executed by the image processing apparatus>

[0040] The processing executed by the image processing apparatus 1 will be described in detail below with reference to Figure 7 The processing executed by the image processing apparatus 1 will be described in detail. Figure 7The processing shown in the flowchart in starts with the user inputting an instruction via the input device 110 and the CPU 101 receiving the input instruction. Since the processing in steps S301 to S304, S306, and S307 is similar to the processing in the first exemplary embodiment, its description is omitted.

[0041] In step S701, the observation condition setting unit 501 sets a way to change the observation condition for the target data. The observation condition setting unit 501 in this exemplary embodiment sets the way to change the same observation condition for multiple target data. In step S702, the generation unit 203 generates moving image data including multiple frames under different observation conditions based on the set way to change the observation condition.

[0042] <Effect>

[0043] As described above, the image processing apparatus 1 according to the second exemplary embodiment sets the way to change the observation condition when reproducing a moving image, and generates moving image data based on the set way to change the observation condition. This enables comparing the colors and material appearances between different objects in the order desired by the user under multiple observation conditions.

[0044] <Modification Example>

[0045] In this exemplary embodiment, the way to change the same observation condition is set for multiple target data. However, the way to change the observation condition can be set individually for each target data. Alternatively, the way to change multiple observation conditions can be set for one target data. This enables simultaneously confirming and comparing the colors and material appearances of an object under multiple observation conditions.

[0046] In this exemplary embodiment, when setting the observation condition after acquiring the target data, the processing order is not limited to the above example. For example, the image processing apparatus 1 can first set the observation condition and then acquire the target data. Alternatively, the image processing apparatus 1 can acquire the target data and set the observation condition simultaneously. Still alternatively, the image processing apparatus 1 can set the observation condition after updating the material appearance flag.

[0047] In the above exemplary embodiment, the image processing apparatus 1 generates and displays moving image data regarding target data having material appearance information. However, the image processing apparatus 1 can generate and display continuous still image data, such as a Graphics Interchange Format (GIF) animation, instead of moving image data. In this case, the multiple still images included in the continuous still image data correspond to respective different observation conditions.

[0048] In the above exemplary embodiment, data having material appearance information is described in the Wavefront OBJ format, but it may be described in other formats, such as the Autodesk Filmbox (FBX) format.

[0049] In the above exemplary embodiment, the generation unit 203 generates moving image data based on the determination result of the determination unit 202. However, the image processing apparatus 1 may use consecutive still images obtained by initially performing CG rendering, or moving images or consecutive still images obtained by imaging an object while changing the observation conditions. In this case, in step S301, the acquisition unit 201 acquires together information indicating that the material appearance flag is ON and target data obtained by initially performing CG rendering or imaging. The acquisition unit 201 acquires together information indicating that the material appearance flag is OFF and still images. With this configuration, the image processing apparatus 1 can perform display using the display control unit 204 without performing the processing of the determination unit 202 and the processing of the generation unit 203.

[0050] The above exemplary embodiment may be provided as a program that automatically describes a hypertext markup language (HTML) file. By generating a file in the Extensible 3D (X3D) format for data having material appearance information, the data having material appearance information can be referred to from the HTML file. By generating an HTML file, the data having material appearance information can be publicly disclosed on the Internet and displayed in a browser.

[0051] The present invention can present the material appearance of an object under multiple observation conditions without the user inputting information to change the observation conditions.

[0052] Other embodiments

[0053] Embodiments of the present invention can also be implemented by the following method, that is, by providing software (program) that executes the functions of the above embodiments to a system or apparatus via a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or apparatus reads and executes the program.

[0054] Although the present invention has been described in conjunction with exemplary embodiments, it should be recognized that the present invention is not limited to the disclosed exemplary embodiments. The scope of the claims should be given the broadest interpretation so as to encompass all modifications, equivalent structures, and functions.

[0055] This application claims the benefit of Japanese Patent Application No. 2020-161332, filed on September 25, 2020, the entire content of which is incorporated herein by reference.

Claims

1. An image processing apparatus, comprising: An acquisition unit configured to acquire target data; A determination unit configured to determine whether the target data has material appearance information of an object; wherein the data having material appearance information is data in a three-dimensional model format, A generation unit that, for the target data determined by the determination unit to have material appearance information of an object, generates moving image data including a plurality of images under different observation conditions, and A display control unit configured to display an image corresponding to the target data on a display unit based on the determination result of the determination unit, wherein, when the target data has material appearance information of an object, the display control unit displays, on the display unit, a moving image or a series of still images corresponding to the target data generated by the generation unit and including a plurality of images under different observation conditions, wherein, when the target data does not have material appearance information of an object, the display control unit displays a still image corresponding to the target data on the display unit.

2. The image processing apparatus according to claim 1, wherein the observation conditions include at least one of an illumination direction, a direction of an object, an observation direction, an observation distance, and a distance between the object and the illumination.

3. The image processing apparatus according to claim 1, further comprising a generation unit configured to generate moving image data or still image data based on a determination result of the determination unit, wherein the display control unit displays an image represented by the moving image data or the still image data generated by the generation unit on the display unit.

4. The image processing apparatus according to claim 3, wherein the generation unit uses any one of bump mapping, ray tracing, and photon mapping.

5. The image processing apparatus according to claim 1, further comprising a setting unit configured to set a manner of changing the observation conditions when displaying a moving image corresponding to the target data, wherein the display control unit displays a moving image corresponding to the target data on the display unit based on the set manner of changing the observation conditions.

6. The image processing apparatus according to claim 5, wherein, The acquisition unit acquires a plurality of pieces of target data, and wherein, when the display control unit displays moving images corresponding to respective pieces of target data on the display unit, the setting unit makes the ways of changing the observation conditions different among the plurality of moving images.

7. The image processing apparatus according to claim 5, wherein, The acquisition unit acquires a plurality of pieces of target data, and wherein, when the display control unit displays moving images corresponding to respective pieces of target data on the display unit, the setting unit makes the ways of changing the observation conditions the same among the plurality of moving images.

8. The image processing apparatus according to claim 5, wherein, When the display control unit displays a plurality of moving images corresponding to one piece of target data on the display unit, the setting unit makes the ways of changing the observation conditions different among the plurality of moving images.

9. The image processing apparatus according to claim 5, wherein, The setting unit sets the way of changing the observation conditions along the time axis based on a user instruction.

10. The image processing apparatus according to claim 1, wherein, When a plurality of moving images corresponding to the target data are displayed on the display unit, the display control unit synchronizes the reproduction start frames and frame rates among the plurality of moving images.

11. The image processing apparatus according to claim 1, wherein, The determination unit performs the determination based on binary data indicating whether the target data has the material appearance information of the object.

12. An image processing method, comprising: Acquire target data; Determine whether the target data has material appearance information of an object; wherein the data having material appearance information is data in a three-dimensional model format, For the target data determined to have material appearance information of an object, generate moving image data including a plurality of images under different observation conditions, and Perform display control to display an image corresponding to the target data on a display unit based on the determination result, wherein, when the target data has material appearance information of an object, the display control displays, on the display unit, a moving image or a series of still images corresponding to the target data and including a plurality of images under different observation conditions, wherein, when the target data does not have material appearance information of an object, a still image corresponding to the target data is displayed on the display unit.

13. A non - transitory computer - readable storage medium storing instructions that, when executed by a computer, cause the computer to perform an image - processing method, the image - processing method comprising: Acquire target data; Determine whether the target data has material appearance information of an object; wherein the data having material appearance information is data in a three-dimensional model format, Generate moving image data including a plurality of images under different observation conditions for target data determined to have material appearance information of an object, and Execute display control to display an image corresponding to the target data on a display unit based on the determination result, wherein, when the target data has material appearance information of an object, the display control displays, on the display unit, the generated moving image or consecutive still images corresponding to the target data and including a plurality of images under different observation conditions, wherein, when the target data does not have material appearance information of an object, a still image corresponding to the target data is displayed on the display unit.

Images