Visualization program, visualization device, and visualization method

Abstract

To efficiently present information about the internal structure of materials to the user. [Solution] The visualization program causes a computer to perform the following actions: acquire user actions, generate a VR image that visualizes information about the internal structure of a material in stereoscopic vision within the field of view corresponding to the acquired actions, and display the VR image on a display device.

Description

[Technical Field]

[0001] The present invention relates to a visualization program, a visualization device, and a visualization method. [Background technology]

[0002] With the improvement of computer performance, the results of numerically solving various models and equations are increasingly being used to design new materials. For example, a calculated phase diagram of a material allows users to visually gain insights into that material. Furthermore, in the field of materials science, visualizing simulation results regarding the internal structure of a material allows users to grasp important insights such as trends, outliers, and correlations in the data that cannot be understood from numerical values ​​alone. The same applies to experimental results; visualizing experimental values ​​allows users to intuitively understand the results and gain insights that might be overlooked if only numerical values ​​are used.

[0003] However, displaying simulation or experimental results on flat-panel displays such as liquid crystal displays and visualizing the results only within a flat surface limits the amount of information that can be displayed, potentially causing users to overlook important information contained in the results. While there are technologies for displaying molecular structures in three dimensions (Patent Documents 1 and 2), these technologies are limited to displaying molecular structures and cannot predict the internal structure of various materials such as metals, insulators, and semiconductors. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-054603 [Patent Document 2] Japanese Patent Publication No. 2021-140701 [Overview of the project] [Problems that the invention aims to solve]

[0005] In one aspect, the present invention aims to efficiently present information about the internal structure of a material to the user. [Means for solving the problem]

[0006] According to one aspect of the present invention, a visualization program causes a computer to perform the following actions: acquire user actions; generate a VR image that visualizes information about the internal structure of a material in stereoscopic vision within a field of view corresponding to the acquired actions; and display the VR image on a display device.

[0007] In the visualization program described above, the information may include predictions or experimental results regarding the internal structure of the material.

[0008] In the visualization program described above, the computer is further instructed to run a simulation program that performs the prediction, and the information may be information indicating the results of the simulation program.

[0009] In the visualization program described above, the computer may be made to execute the simulation program in real time.

[0010] In the visualization program described above, the computer may be instructed to temporarily suspend the execution of the simulation program.

[0011] In the visualization program described above, the information may be information generated by a computer other than the computer mentioned above.

[0012] In the visualization program described above, the computer may be further instructed to perform additional calculations not performed in the prediction, and to overlay the results of the additional calculations onto the VR image when generating the VR image.

[0013] In the above visualization program, the prediction may be a prediction of the state diagram of the material, and the additional calculation may be a calculation for specifying a tie line in the state diagram.

[0014] In the above visualization program, the computer may be further caused to receive an input operation of the user, and the additional calculation may be performed when the input operation is received.

[0015] In the above visualization program, the input operation is an operation on an operation input device, and the computer may be caused to perform the additional calculation based on the position of a point indicated by the operation input device in the VR image.

[0016] In the above visualization program, the computer may be further caused to generate a selection screen for prompting the user to select any one of a plurality of simulation programs for performing the prediction or any one of the results of a plurality of the experiments.

[0017] In the above visualization program, the prediction is a prediction regarding a field in the material, and in generating the VR image, a VR image visualizing the field may be generated.

[0018] In the above visualization program, in generating the VR image, a VR image visualizing an isosurface or a cross section of the field may be generated.

[0019] In the above visualization program, the field may be any one of a phase field, a concentration field, a temperature field, a magnetic field, an electric field, a velocity field, a stress field, a strain field, an energy field, a potential field, and a regularity field.

[0020] In the above visualization program, the prediction may be a numerical calculation regarding the internal structure of the material.

[0021] In the visualization program described above, the numerical calculation may be performed using any of the following methods: CALPHAD method, phase-field method, density functional theory, molecular orbital method, GW method, Hartree-Fock method, or molecular dynamics method.

[0022] In the visualization program described above, the display device may be a head-mounted display.

[0023] In the visualization program described above, the acquisition of the aforementioned actions may be performed by acquiring signals indicating the user's position and orientation from the head-mounted display.

[0024] According to another aspect of the present invention, the visualization device includes an acquisition unit that acquires user actions, a VR image generation unit that generates a VR image in which information regarding the internal structure of a material is visualized in stereoscopic view within a field of view corresponding to the acquired actions, and a display control unit that displays the VR image on a display device.

[0025] According to another aspect of the present invention, the visualization method involves a computer performing the following actions: acquiring user movements; generating a VR image in which information regarding the internal structure of a material is visualized in stereoscopic view within a field of view corresponding to the acquired movements; and displaying the VR image on a display device. [Effects of the Invention]

[0026] According to the present invention, information regarding the internal structure of a material can be efficiently presented to the user. [Brief explanation of the drawing]

[0027] [Figure 1] Figure 1 is an overall diagram of the visualization system according to this embodiment. [Figure 2] Figure 2(a) is a front view of the controller, and Figure 2(b) is a side view of the controller. [Figure 3] Figure 3 shows an example in this embodiment where a state diagram is used as the VR image. [Figure 4]Figure 4 is an example of a VR image obtained by performing additional calculations on the state diagram in Figure 3 in this embodiment. [Figure 5] Figure 5 shows an example of a VR image enlarged by pressing button A in this embodiment. [Figure 6] Figure 6 is an example of a VR image obtained in this embodiment when additional calculations are performed on the enlarged state diagram as shown in Figure 5. [Figure 7] Figure 7 shows an example in this embodiment where a density field is used as the VR image. [Figure 8] Figure 8 shows an example of a VR image taken at a later time than that shown in Figure 7. [Figure 9] Figure 9 shows an example of a VR image taken at a later time than that shown in Figure 8. [Figure 10] Figure 10 shows an example in this embodiment where the microstructure and stress field inside steel undergoing martensitic transformation are used as VR images. [Figure 11] Figure 11 shows an example of generating a VR image from 3D atom probe data of a Mg-doped GaN sample. [Figure 12] Figure 12 is an example of a VR image obtained by moving the viewpoint to the inside of the GaN sample from the state shown in Figure 11. [Figure 13] Figure 13 shows an example of generating a VR image from FIB / SEM tomography data of a neodymium magnet. [Figure 14] Figure 14 is an example (part 1) of a VR image obtained by moving the viewpoint inside the sample from the state shown in Figure 13. [Figure 15] Figure 15 is an example (part 2) of a VR image obtained by moving the viewpoint inside the sample from the state shown in Figure 13. [Figure 16] Figure 16 is an example of a functional configuration diagram of the visualization device according to this embodiment. [Figure 17] Figure 17 is a schematic diagram showing an example of the data structure of the display information according to this embodiment. [Figure 18] Figures 18(a) to (d) are schematic diagrams illustrating examples of the association between the first to third coordinates and attribute values ​​in this embodiment. [Figure 19] Figure 19 shows an example of the selection screen according to this embodiment. [Figure 20] Figure 20 shows an example of the selection screen when the pointer is positioned in this embodiment. [Figure 21] Figure 21 is a flowchart (part 1) showing an example of the processing performed by the visualization device according to this embodiment. [Figure 22] Figure 22 is a flowchart (part 2) showing an example of the processing performed by the visualization device according to this embodiment. [Figure 23] Figure 23 is a flowchart (part 3) showing an example of the processing performed by the visualization device according to this embodiment. [Figure 24] Figure 24 is a flowchart (part 4) showing an example of the processing performed by the visualization device according to this embodiment. [Figure 25] Figure 25 is an example of a hardware configuration diagram of the visualization device according to this embodiment. [Modes for carrying out the invention]

[0028] (Embodiment) Embodiments of the present invention will be described below with reference to the drawings. Similar elements are denoted by the same reference numerals, and their descriptions are omitted.

[0029] Figure 1 is an overall configuration diagram of the visualization system according to this embodiment. As shown in Figure 1, the visualization system 1 comprises a visualization device 2, an HMD (head-mounted display) 3, and a controller 4.

[0030] Visualization device 2 is a computer such as a PC (Personal Computer) that generates a VR (Virtual Reality) image 5 that visualizes information about the internal structure of a material and controls its display on the HMD3. The material to be visualized is not particularly limited, but examples include metals, semiconductors, and periodic solids such as magnetic materials. In the following, the virtual three-dimensional space on which the VR image 5 is projected will be called the virtual space, and the real three-dimensional space in which user U exists will be called the real space. Visualization device 2 generates two VR images 5, one for the left eye and one for the right eye, so as to create parallax, but unless otherwise specified, the following VR image 5 may be for either the left or right eye.

[0031] Furthermore, the information displayed in VR image 5 is not particularly limited. For example, VR image 5 may represent isosurfaces or cross-sections of various fields within a material, such as market conditions, concentration fields, temperature fields, magnetic fields, electric fields, velocity fields, stress fields, strain fields, energy fields, potential fields, and order fields. Alternatively, VR image 5 may represent a phase diagram of a material composed of multiple elements, such as a ternary or quaternary system.

[0032] HMD3 is an example of a display device, a wearable device worn on the head of user U. HMD3 has the function of displaying separate VR images 5 for the left and right eyes with parallax. This allows the user to view the VR images 5 in stereoscopic vision within the virtual space. HMDs come in various forms such as goggles, helmets, and glasses, and any of these can be used as HMD3.

[0033] The HMD3 incorporates a gyroscope, an accelerometer, and multiple cameras (not shown) to capture the user's movements. The gyroscope detects the direction of the user's head. The accelerometer detects the user's horizontal acceleration. The multiple cameras capture images of the surroundings of the HMD3.

[0034] The HMD3 incorporates a processor that estimates the position and orientation of the user U's head in real space based on information obtained from the gyroscope, accelerometer, and camera. This method of estimating position and orientation using camera images in conjunction with the HMD3 is called the inside-out method. The processor sends an HMD tracking signal indicating the head position and orientation estimated using the inside-out method to the visualization device 2 via a USB (Universal Serial Bus) cable 7. Alternatively, the visualization device 2 and the HMD3 may be connected using DisplayPort, wireless LAN, or other means instead of USB.

[0035] Controller 4 is an input device that allows user U to manually input commands, and is wirelessly connected to HMD3 via short-range wireless communication such as Bluetooth®. Input commands to Controller 4 are notified to Visualization Device 2 via HMD3.

[0036] Furthermore, the controller 4 has a built-in gyro sensor and an accelerometer (not shown), and an infrared LED is provided on the surface of the controller 4's casing. The gyro sensor is a sensor that detects the direction of the controller 4. The accelerometer is a sensor that detects the acceleration of the controller 4.

[0037] The HMD3's processor detects the position of the controller 4's infrared LED using its built-in camera and estimates the relative position between the HMD3 and the controller 4 based on that position. In addition to this relative position, the HMD3's processor uses information from the gyro sensor and accelerometer acquired from the controller 4 via short-range wireless communication to estimate the position and orientation of the controller 4 in real space, and notifies the visualization device 2 of the estimated position and orientation via a controller tracking signal through the USB cable 7.

[0038] Figure 2(a) is a front view of controller 4, and Figure 2(b) is a side view of controller 4. Controller 4 is available in right-handed and left-handed versions, but since the structure is almost the same for both, Figures 2(a) and (b) show the right-handed controller 4.

[0039] As shown in Figures 2(a) and 2(b), the controller 4 includes an A button 4a, a B button 4b, a joystick 4c, a grip button 4d, a trigger 4e, and a system button 4f.

[0040] The user performs input operations by operating these buttons. For example, pressing button A 4a or button B 4b causes the visualization device 2 to enlarge or reduce the VR image 5, respectively. Pressing the grip button 4d or trigger 4e causes the visualization device 2 to lower or raise the field of view in the VR image 5, respectively. Furthermore, tilting the joystick 4c causes the visualization device 2 to shift the viewpoint in the VR image 5 in the direction in which the joystick 4c is tilted. Finally, pressing the system button 4f terminates the process of displaying the VR image 5.

[0041] Alternatively, instead of using the right-hand controller 4, the left-hand controller 4 may be used to operate the VR image 5 in the same manner as described above. In this case, the user should operate the X and Y buttons provided on the left-hand controller 4, which replace the A button 4a and B button 4b, respectively.

[0042] Next, we will explain an example of VR image 5, referring to Figures 3 to 15.

[0043] Figure 3 shows an example where a phase diagram is used as VR image 5. In this example, the phase diagram of a ternary alloy of Mg (magnesium), Ca (calcium), and Zn (zinc) is displayed as VR image 5. The phase diagram consists of a phase interface 10, labels 11 representing each element, a horizontal axis 12 showing the composition of each element, and a vertical axis 13 showing the temperature of the alloy.

[0044] In this case, the visualization device 2 generates a VR image 5 in which the state diagram is visualized in stereoscopically within the user's field of view. Note that at the display angle shown in Figure 3, the label 11 indicating Zn is hidden and not visible.

[0045] The display information that forms the basis of VR image 5 may be generated by the visualization device 2 by performing a state diagram calculation, or it may be generated by a computer different from the visualization device 2. An example of a calculation code for performing the state diagram calculation is Thermo-Calc Software AB's Termo-Calc. Termo-Calc is a calculation code that performs state diagram calculations using the CALPHAD method.

[0046] Furthermore, the visualization device 2 may perform additional calculations from the state shown in Figure 3, as follows.

[0047] Figure 4 is an example of a VR image 5 obtained when additional calculations are performed on the state diagram in Figure 3. In this example, when user U points to a point in the two-phase coexistence region of the state diagram using controller 4, visualization device 2 displays a pointer 9 at that point. When the user performs an input operation by pressing the trigger 4e on controller 4 in this state, visualization device 2 identifies a tie line 14 passing through that point through additional calculations and displays the tie line 14 superimposed on the state diagram.

[0048] Furthermore, the visualization device 2 displays labels 15 indicating the type of phase at the endpoints of the connecting line 14. In this example, the label 15, "BCC_B2#1, LIQUID," indicates that one endpoint of the connecting line 14 is the compound phase "BCC_B2#1" and the other endpoint is the liquid phase.

[0049] The field of view of each VR image 5 in Figures 3 and 4 can be varied by the visualization device 2 based on the aforementioned HMD tracking signal indicating the user's head position and posture. For example, when user U tilts their head up, down, left, or right, the visualization device 2 generates VR images 5 with the field of view tilted in those directions. Also, when the user walks around in real space, the visualization device 2 generates VR images 5 that have moved by the direction and distance the user walked.

[0050] This allows VR image 5 to be displayed stereoscopically within the virtual space from the user's desired viewpoint or gaze. As a result, users can adjust VR image 5 to their liking while feeling immersed in the virtual space, and gain insights that might be overlooked on a flat display.

[0051] Furthermore, the field of view of VR image 5 can also be changed by the user operating the buttons on controller 4, as mentioned above.

[0052] Figure 5 shows an example of a VR image 5 that has been enlarged by pressing button A 4a.

[0053] By zooming in on VR image 5 in this way, the user can examine the details of the state diagram.

[0054] Figure 6 is an example of VR image 5 when additional calculations are performed on an enlarged phase diagram as shown in Figure 5. When additional calculations are performed on an enlarged phase diagram in this way, the user can clearly see the position of the connecting line 14 within the phase diagram. This allows the user to accurately grasp the position of the phase indicated by the endpoint of the connecting line 14.

[0055] Figure 7 shows an example where a concentration field is used as the VR image 5. In this example, the visualization device 2 generates VR image 5 showing the material concentration at each time point by performing a PF (Phase Field) simulation in real time. The material being calculated is an Fe-Cr binary system.

[0056] This VR image 5 displays isosurfaces 16 representing the concentration of the material. By tracking the time evolution of these isosurfaces 16, the user can understand how the material undergoes spinodal decomposition. Here, we focus on the internal structure of the material in region A.

[0057] Figure 8 shows an example of VR image 5 at a later time than that shown in Figure 7. Figure 9 also shows an example of VR image 5 at a later time than that shown in Figure 8.

[0058] As shown in Figure 8, the internal tissue in region A becomes thinner over time, and eventually the internal tissue is severed as shown in Figure 9.

[0059] As shown in Figures 7 to 9, the visualization device 2 generates VR images 5 at each time point, allowing the user to understand the changes in the internal structure of the material over time. In this case as well, the user U can move their head or walk around to observe the changes in the internal structure from a position that is easy to see. Therefore, for example, the user can get close to region A in the virtual space to observe the internal structure, and understand the detailed progress of spinodal decomposition, which is often overlooked when viewing the whole picture from above.

[0060] Figure 10 shows an example in which the microstructure and stress field inside steel undergoing martensitic transformation are used as VR image 5. In this example, the visualization device 2 calculates the stress field 18 inside the steel and multiple crystals (variants) 17a to 17c with different crystal orientations by performing a PF simulation, and generates VR image 5 showing the calculation results.

[0061] This allows users to understand the relationship between the microstructure and stress field of steel materials undergoing martensitic transformation while experiencing an immersive virtual environment.

[0062] In the examples of Figures 3 to 10 mentioned above, the results of a simulation program that predicts the internal structure of the material were generated as VR image 5. However, the visualization device 2 may also generate the VR image 5 from experimental results regarding the internal structure of the material.

[0063] Figure 11 shows an example of generating VR image 5 from 3D atom probe (3D-AP) data of a Mg-doped GaN (gallium nitride) sample.

[0064] In this VR image 5, the elemental distribution is represented by shades of gray. Furthermore, the example in Figure 11 shows the user's viewpoint outside the GaN sample, illustrating a scenario where the user is viewing the sample from above. By displaying a VR image from this overhead perspective, the user can gain a comprehensive understanding of the elemental distribution of the sample, which unfolds three-dimensionally before their eyes.

[0065] Figure 12 is an example of VR image 5 obtained by moving the viewpoint to the inside of the GaN sample from the state shown in Figure 11.

[0066] For example, a user can obtain a VR image like this 5 by walking in real space and moving their viewpoint inside a GaN sample. This allows the user to check the elemental distribution from inside the GaN sample and grasp detailed information about the inside of the sample that cannot be obtained from a 3D image viewed from the outside.

[0067] Figure 13 shows an example of generating VR image 5 from FIB (Focused Ion Beam) / SEM (Scanning Electron Microscopy) tomography data of a neodymium magnet.

[0068] In this VR image 5, the elemental distribution is represented by shades of gray. Furthermore, the example in Figure 13 shows the user's viewpoint being outside the sample, such as a neodymium magnet, and the user viewing the sample from above. By displaying a VR image from this overhead viewpoint, the user can grasp the three-dimensional atomic distribution unfolding before their eyes.

[0069] Figures 14 and 15 are examples of VR images 5 obtained by shifting the viewpoint to the interior of the sample from the state shown in Figure 13. By shifting the viewpoint to the interior of the sample in this way, the user can check the detailed elemental distribution from inside the sample.

[0070] Furthermore, in all the examples shown in Figures 11 to 15, the data underlying VR image 5 is experimental data such as 3D atom probe data and FIB / SEM tomography data. Therefore, users can understand the actual state of the internal structure of the material, which cannot be obtained through simulation.

[0071] As described above, by generating VR images 5 as shown in Figures 3 to 15, the visualization device 2 can efficiently present the user with the results of simulation programs and experiments related to the internal structure of materials.

[0072] Next, the functional configuration of the visualization device according to this embodiment will be described.

[0073] Figure 16 is an example of a functional configuration diagram of the visualization device 2 according to this embodiment. As shown in Figure 16, the visualization device 16 comprises a storage unit 20, a first communication unit 21, a second communication unit 22, and a control unit 23.

[0074] The memory unit 20 stores display information 31 and multiple simulation programs 32a to 32d.

[0075] Simulation programs 32a to 32d are programs that predict the internal structure of materials using different numerical calculations. Examples of numerical calculations include the CALPHAD method, phase-field method, density functional method, molecular orbital method, GW method, Hartree-Fock method, and molecular dynamics method. Of these, the CALPHAD method is a technique that performs phase diagram calculations as shown in Figures 3 to 6, and is implemented, for example, in Thermo-Calc Software AB's Thermo-Calc. The phase-field method is a technique that calculates the concentration field shown in Figures 7 to 9 and the stress field shown in Figure 10.

[0076] Display information 31 is an example of information regarding the internal structure of a material, and is the basis for the VR images 5 shown in Figures 3 to 15. For example, the execution results or experimental results of each simulation program 32a to 32d become the display information 31.

[0077] While the visualization device 2 may execute the simulation program and generate the display information 31, another computer different from the visualization device 2 may execute the simulation program and generate the execution results as the display information 31. In that case, the control unit 23 can acquire the display information 31 generated by the other computer via the network 30 and store it in the storage unit 20. Alternatively, the control unit 23 may refer to the display information 31 stored in the other computer. This makes it possible to effectively utilize the execution results of the simulation program executed by the other computer. In particular, when the execution results of a simulation program exist but there is no device to visualize them, the user can grasp the detailed internal organization by visualizing the execution results as a VR image 5 using the visualization device 2 in this way.

[0078] Furthermore, when the experimental results are to be used as display information 31, the control unit 23 may acquire the display information 31 stored in another computer via the network 30 and store it in the storage unit 20, or the control unit 23 may refer to the display information 31 stored in another computer.

[0079] Figure 17 is a schematic diagram showing an example of the data structure of the display information 31 according to this embodiment.

[0080] As shown in Figure 17, the displayed information 31 is information that associates the first coordinate, second coordinate, third coordinate, and attribute value. The first, second, and third coordinates are coordinates that specify each point in the three-dimensional virtual space. The attribute value is a value that indicates the attribute of each point in the virtual space. There are no particular limitations on how the first to third coordinates and the attribute value are associated.

[0081] Figures 18(a) to (d) are schematic diagrams illustrating examples of the association between the first to third coordinates and attribute values.

[0082] Figure 18(a) shows an example of displaying a phase diagram of a ternary material as a VR image 5, as shown in Figures 3 to 6. The number of phases is assumed to be n. In this case, the attribute value is a phase identifier (#1, #2, #3, ..., #n) that identifies the type of phase. For each of these phase identifiers, the first to third coordinates are specified in the generated sub-information 31a. The third coordinate indicates temperature, and the first and second coordinates indicate the composition of the ternary material at that temperature. The information collected from this sub-information 31a for all phase identifiers (#1, #2, #3, ..., #n) becomes the display information 31.

[0083] Figure 18(b) shows an example of displaying the material's concentration field as a VR image 5, as shown in Figures 7 to 9. In this case, the X, Y, and Z coordinates of the XYZ Cartesian coordinate system stretched across three-dimensional real space become the first, second, and third coordinates, respectively. The attribute value is the concentration at that location. The information relating these X, Y, Z coordinates and concentrations becomes the display information 31.

[0084] Figure 18(c) shows an example of displaying the stress field in a material undergoing martensitic transformation as VR image 5, as shown in Figure 10. Note that there are three variants. In this case, the attribute values ​​are the first to third variants and the stress value. The first variant indicates whether the first of the three variants exists. The same applies to the second and third variants.

[0085] Sub-information 31b is information that associates the stress value with the first to third variants at a given location. Here, the number of locations is n, and each location is identified by a location identifier (#1, #2, #3, ..., #n). Each location identifier is an identifier associated with the first, second, and third coordinates of the location in three-dimensional real space. The first, second, and third coordinates are the X, Y, and Z coordinates of the XYZ Cartesian coordinate system stretched across three-dimensional real space, respectively. Display information 31 is the collection of such sub-information 31b for all location identifiers (#1, #2, #3, ..., #n).

[0086] Figure 18(d) shows an example of displaying experimental results as VR image 5, as in Figures 11-15. In this case, the X, Y, and Z coordinates of the XYZ Cartesian coordinate system stretched across three-dimensional real space become the first, second, and third coordinates, respectively. The attribute values ​​are experimental values ​​at that location. For example, in the case of 3D atom probe data (Figures 11-12) or FIB / SEM tomography data (Figures 13-15), the presence or absence of elements is the experimental value.

[0087] Refer to Figure 16 again.

[0088] The first communication unit 21 is a communication interface used when the visualization device 2 communicates with the HMD device 3 via USB. As mentioned above, the HMD device 3 is wirelessly connected to the controller 4 via short-range wireless communication such as Bluetooth (registered trademark). The second communication unit 22 is a communication interface for connecting the visualization device 2 to a network 30 such as a LAN (Local Area Network) or the Internet.

[0089] The control unit 23 is a processing unit that controls each part of the visualization device 2. For example, the control unit 23 includes an acquisition unit 24, a VR image generation unit 25, a reception unit 26, an execution unit 27, a calculation unit 28, and a display control unit 29.

[0090] The acquisition unit 24 is a processing unit that acquires the movements of user U from the HMD device 3. For example, the acquisition unit 24 acquires HMD tracking signals from the HMD device 3 that indicate the position and orientation of user U's head.

[0091] Furthermore, the acquisition unit 24 acquires not only the movements of user U, but also the position and orientation of controller 4. For example, the acquisition unit 24 acquires controller tracking signals from the HMD device 3 that indicate the position and orientation of controller 4.

[0092] The VR image generation unit 25 generates a VR image 5 in which display information 31 regarding the internal structure of the material is visualized in stereoscopic vision within the field of view corresponding to the acquired user U's movements. The display information 31 to be visualized is, as mentioned above, the result of a prediction or experiment regarding the internal structure of the material.

[0093] The algorithm for generating the VR image 5 from the display information 31 is not particularly limited, but in this embodiment, the VR image generation unit 25 generates the VR image 5 as follows. First, the first coordinate, second coordinate, and third coordinate in the display information 31 are defined as x, y, and z, respectively, and a four-dimensional vector p containing these is defined by the following equation (1).

[0094]

number

[0095] The coordinate system that defines vector p will also be referred to as the local coordinate system below. The local coordinate system is a three-dimensional coordinate system used to define the position of each point (x, y, z) within the display information 31.

[0096] Also, point p in the two-dimensional space within VR image 5. d We define it by the following equation (2).

[0097]

number

[0098] point p d The coordinate system that defines this will also be called the screen coordinate system below. The screen coordinate system is a two-dimensional coordinate system used to define each point (X, Y) in VR image 5.

[0099] At this time, the VR image generation unit 25 uses the model matrix M, the view matrix V, and the projection matrix P to generate a vector p at position p as shown in equation (3). d By moving to multiple positions p d A VR image 5 composed of the above is generated.

[0100]

number

[0101] Here, the model matrix M is a 4x4 matrix used to transform the local coordinate system into the world coordinate system. The world coordinate system has four dimensions, of which three dimensions define the positional relationships of each point in the virtual space. For example, the model matrix M is a matrix that performs translation, rotation, scaling, etc.

[0102] The view matrix V is a 4x4 matrix used to transform the world coordinate system into the camera coordinate system. The camera coordinate system is a four-dimensional coordinate system with the user U's viewpoint in the virtual space as the origin, and one of the four coordinate axes is defined as the direction of the user U's line of sight.

[0103] The projection matrix P is a 4x4 viewpoint projection matrix used to transform camera coordinates into screen coordinates. Two of the four components obtained by applying the projection matrix P are the point p in equation (2). d This is the result.

[0104] In this embodiment, the VR image generation unit 25 generates two VR images 5, one for the left eye and one for the right eye, by performing the transformation in equation (3) using the OpenGL® library. In this case, each element of the model matrix M and the view matrix V is calculated based on parameters obtained through the OpenGL library, as well as the position of the user U in real space, image scaling by zooming in and out, etc.

[0105] Furthermore, the VR image generation unit 25 slightly changes the viewpoint and line of sight direction of the view matrix V for the left eye and the right eye, thereby causing parallax to occur in the respective VR images 5 for the left and right eyes.

[0106] Furthermore, the VR image generation unit 25 generates a selection screen that prompts the user to select from multiple simulation programs 32a to 32d and multiple experimental results.

[0107] Figure 19 shows an example of the selection screen. As shown in Figure 19, the selection screen 40 is displayed in the virtual space 35 when the VR image 5 is not displayed. The selection screen 40 displays text 40a to 40d that explains the functions of each of the simulation programs 32a to 32d, and text 40e and 40f that explain the contents of the experimental results.

[0108] In texts 40a to 40c, "ternary phase diagram," "quaternary phase diagram," and "martensite" are selected when displaying the results of a completed simulation program, as shown in Figures 3 to 6 and Figure 10.

[0109] Furthermore, the "spinodal decomposition" option in text 40d is selected when running a simulation program in real time and displaying the results during its execution, as shown in Figures 7 to 9.

[0110] Then, the "3D atom probe data" and "tomography data" in texts 40e and 40f are selected when displaying experimental results like those shown in Figures 11 to 15.

[0111] Based on texts 40a to 40f, user U can thus understand the functions and experimental results of each simulation program 32a to 32d.

[0112] When making a selection, user U uses controller 4 to move the pointer to the text corresponding to the desired simulation program 32a-32d or experimental results from among texts 40a-40f.

[0113] Figure 20 shows an example of a selection screen when the pointer is positioned in this manner. When user U points the controller 4 (see Figures 2(a) and (b)) at the selection screen 40, the VR image generation unit 25 displays the pointer 9 at the position indicated by the controller 4 in the virtual space 35. For example, the VR image generation unit 25 identifies the orientation of the controller 4 based on the controller tracking signal acquired by the acquisition unit 24 and displays the pointer 9 in that orientation. Then, when the reception unit 26 receives an input operation to press the trigger 4e (see Figure 2(b)), the VR image generation unit 25 highlights the text at the position of the pointer 9. In Figure 20, the highlighted display is indicated by a frame 41.

[0114] Refer to Figure 16 again.

[0115] The reception unit 26 is a processing unit that receives user input operations from the controller 4.

[0116] The execution unit 27 is a processing unit that executes the simulation programs 32a to 32d. The execution unit 27 also stores the execution results of the simulation programs 32a to 32d as display information 31 in the storage unit 20.

[0117] The calculation unit 28 is a processing unit that performs additional calculations not performed in the predictions made by the simulation programs 32a to 32d. For example, the calculation unit 28 performs additional calculations to draw the connecting lines 14 shown in Figures 4 and 6.

[0118] The display control unit 29 is a processing unit that controls the display of the VR image 5 on the HMD3.

[0119] Of the functions of the control unit 23 described above, the functions of the acquisition unit 24, the VR image generation unit 25, the reception unit 26, and the display control unit 29 can be realized by the visualization program according to this embodiment calling the OpenXR (registered trademark) API (Application Programming Interface). The programming language of the visualization program according to this embodiment is not particularly limited, but in this embodiment the visualization program is written in Python. This makes it easy to link the simulation programs 32a to 32d with a Python interface such as the commercial ThermoCalc program.

[0120] Next, we will describe an example of the processing performed by the visualization device 2.

[0121] Figures 21 to 24 are flowcharts illustrating an example of the processing performed by the visualization device 2. This processing is triggered when the visualization device 2 starts the visualization program.

[0122] First, the VR image generation unit 25 constructs a virtual space 35 (see Figure 19) for projecting the VR image 5 (step S11), and displays a selection screen 40 (see Figure 19) in that virtual space (step S12).

[0123] Next, the reception unit 26 determines whether a simulation program or experimental result has been selected (step S13). For example, as shown in Figure 20, when the input operation of pressing the trigger 4e (see Figure 2(b)) is received with the pointer 9 positioned over any of the texts 40a to 40f, the reception unit 26 determines that it has been selected (YES), and otherwise determines that it has not been selected (NO).

[0124] If it is determined that no selection has been made (NO), the process is repeated in step S13. On the other hand, if it is determined that a selection has been made (YES), the process proceeds to step S14.

[0125] In step S14, the reception unit 26 determines the type of object selected on the selection screen 40. The types are "Calculation Result Display," "Real-time Calculation," and "Experimental Results."

[0126] "Calculation Result Display" is a type that displays the execution results of a simulation program that has already been run. For example, the "Ternative Phase Diagram," "Quaternary Phase Diagram," and "Martensite" types in the selection screen 40 of Figure 19 are all "Calculation Result Display."

[0127] Furthermore, the type of "Spinodal Decomposition" in the selection screen 40 of Figure 19 is "Real-time Calculation".

[0128] Furthermore, in the selection screen 40 of Figure 19, the types of "3D atom probe data" and "tomography data" are "experimental results".

[0129] For example, the user can pre-store type information in the storage unit 20, associating the text 40a to 40f in the selection screen 40 of Figure 19 with the type, and the reception unit 26 can refer to that type information to determine the type.

[0130] If it is determined that "display calculation result" is required, the process proceeds to step S15. In step S15, the VR image generation unit 25 retrieves display information 31 from the storage unit 20. In this example, the user has previously stored display source information in the storage unit 20, associating the texts 40a to 40f on the selection screen 40 in Figure 19 with the display information 31. The VR image generation unit 25 then refers to this display source information to identify the display information 31 associated with the text selected by the user from among the texts 40a to 40f, and retrieves that display information 31 from the storage unit 20.

[0131] Next, the acquisition unit 24 acquires the user U's actions from the HMD device 3 (step S16).

[0132] Next, the calculation unit 28 determines whether there is an instruction for additional calculation (step S17). For example, when the reception unit 26 receives an input operation on the controller 4, the calculation unit 28 determines that there is an instruction for additional calculation. As the input operation, as shown in FIG. 4, the user U points the pointer 9 at a point in the two-phase coexistence region in the state diagram and presses the trigger 4e (see FIG. 2(b)).

[0133] Here, when it is determined that there is an instruction for additional calculation (YES), the process proceeds to step S18.

[0134] In step S18, the calculation unit 28 performs additional calculation. For example, when the VR image 5 is the state diagram, the calculation unit 28 calculates the connecting line 14 (see FIG. 4) passing through the position of the pointer 9 in the VR image 5 by additional calculation. The calculation method is not particularly limited. For example, the calculation unit 28 performs additional calculation as follows. First, the calculation unit 28 refers to a database in which each of the Gibbs energy, temperature, and system composition is associated with each phase. The database can be generated by the calculation unit 28 executing the aforementioned Termo-Calc. Such a database is equivalent to one representing the Gibbs energy G p of the phase p as a function of temperature and composition. Hereinafter, this function is denoted as G p (x p , T). Note that x p is the composition of the phase p and T is the temperature. Using this function G p (x p , T), the calculation unit 28 calculates the total energy G tot of the system based on the following formula (4).

[0135]

Equation

[0136] Note that φ p is the volume fraction of the phase p.

[0137] Next, the calculation unit 28 calculates the total energy G under the given temperature T and average composition. tot Equilibrium calculations are performed to find the composition and abundance fraction of each phase p that minimizes . However, since energy minimization must be performed under conditions where the average composition is fixed, φ p , x p The material must satisfy the conservation conditions of equation (5).

[0138]

number

[0139] x in equation (5) ave This represents the average composition and is the composition at the position indicated by the pointer 9 of controller 4 (see Figures 4 and 6).

[0140] The calculation unit 28 performs equilibrium calculations under the constraints of equation (5), thereby determining the total energy G of equation (4). tot The composition x that minimizes p and the fraction of existence φ p The and are identified for each phase p. Energy minimization required for equilibrium calculations can be performed using methods such as Newton's method. The calculation unit 28 then determines the identified composition x p The line segment connecting these points is identified as connecting line 14, and the position of connecting line 14 in the state diagram is determined.

[0141] Next, the calculation unit 28 stores the calculation results of the additional calculation in the storage unit 20 (step S19). For example, the calculation unit 28 calculates the total energy G tot The composition x that minimizes p The memory unit 20 stores the position of the connecting line 14 in the state diagram and the names of the phases p at both ends of the connecting line 14. The names of the phases p are strings such as "LIQUID" or "BCC_B2#1" which form the basis of the label 15 as shown in Figures 4 and 6.

[0142] Next, the VR image generation unit 25 generates a VR image 5 of the field of view corresponding to the user U's actions, acquired in step S16, based on the display information 31 acquired in step S15 (step S20). The VR image generation unit 25 also refers to the calculation results of the additional calculations stored in the storage unit 20 and overlays those calculation results onto the VR image 5. For example, in the examples of Figures 4 and 6, the VR image generation unit 25 overlays connecting lines 14 and labels 15 onto the VR image 5.

[0143] On the other hand, if it is determined in step S17 that there is no instruction for additional calculation (NO), steps S18 and S19 are skipped and step S20 is performed. In this case, the VR image generation unit 25 generates the VR image 5 based on the display information 31 acquired in step S15, without overlaying the calculation results of the additional calculation onto the VR image 5.

[0144] Next, the display control unit 29 controls the display of the VR image 5 on the HMD device 3 (step S21).

[0145] The VR image generation unit 25 then determines whether there is an operation to terminate the process of displaying the VR image 5 (step S22). The termination operation is, for example, an operation in which the user presses the system button 4f on the controller 4 (see Figures 2(a) and (b)). If the VR image generation unit 25 determines that there is a termination operation (YES), it terminates the process of displaying the VR image 5.

[0146] On the other hand, if there is no termination operation (NO), the process returns to step S16. As a result, the acquisition of user actions (step S16) and the generation of VR image 5 (step S20) are repeated until a termination operation is performed. Consequently, VR image 5 is dynamically generated from a viewpoint corresponding to user U's actions, allowing the user to observe the internal structure of the material predicted by the simulation program while immersing themselves in the virtual space.

[0147] Next, we will explain the process when "real-time calculation" is determined in step S14. In this case, the process moves to step S25, and the execution unit 27 initializes time t to 0. Note that time t is the simulation time used internally for real-time calculation.

[0148] Next, the execution unit 27 executes the simulation program selected by the user on the selection screen 40 (see Figure 19) from among the simulation programs 32a to 32d stored in the storage unit 20, and makes a prediction about the internal structure of the material at time t (step S26). For example, the user may pre-store program information in the storage unit 20 that associates simulation programs 32a to 32d with texts 40a to 40f (see Figure 19), and the execution unit 27 may refer to this program information and execute the simulation program associated with the text selected by the user.

[0149] Next, the execution unit 27 stores the execution result of the simulation program at time t as display information 31 in the storage unit 20 (step S27).

[0150] Next, the acquisition unit 24 acquires the user U's actions from the HMD device 3 (step S28).

[0151] Next, the VR image generation unit 25 generates a VR image 5 of the field of view corresponding to the user U's actions, acquired in step S28, based on the display information 31 stored in step S27 (step S29).

[0152] Next, the display control unit 29 controls the display of the VR image 5 on the HMD device 3 (step S30).

[0153] Next, the VR image generation unit 25 determines whether it is within the pause period (step S31). Pause is performed when an input operation is made to temporarily stop the execution of the simulation program. For example, if the user wants to observe the simulation results at a certain time t in detail, the pause period begins when the user U presses the button assigned to start the pause on the controller 4 and the reception unit 26 receives that input operation. The pause period ends when the user U presses the button assigned to release the pause on the controller 4 and the reception unit 26 receives that input operation. Based on the history of operations on the buttons for starting and ending the pause, the VR image generation unit 25 determines whether the execution of step S31 is included in the pause period. The user can assign any of the buttons 4a to 4f on the controller 4 (see Figures 2(a) and (b)) as the buttons for starting and ending the pause, as long as they do not conflict with other operations using the controller 4.

[0154] If it is determined that a pause period is in progress (YES), the process returns to step S28. This allows the acquisition of actions (step S28) and the generation of VR images (step S29) to be repeated while the progression of time t remains stopped. As a result, the VR image generation unit 25 can generate a field of view corresponding to the user U's actions while fixing the simulation results at time t, allowing the user to observe the simulation results at time t in detail.

[0155] On the other hand, if it is determined in step S31 that the pause period is not in effect (NO), the process proceeds to step S32.

[0156] In step S32, the VR image generation unit 25 determines whether there is a re-execution instruction. A re-execution instruction is an instruction to restart the simulation from the beginning by returning time t to state 0. For example, if the user U wants to observe the changes in the internal structure of a material again from the beginning, the user U will input a re-execution instruction. The input operation for a re-execution instruction is not particularly limited. For example, if the user U presses a button on the controller 4 that is assigned to a re-execution instruction, and the reception unit 26 receives the input operation, the VR image generation unit 25 determines that there is a re-execution instruction. The user can assign any of the buttons 4a to 4f on the controller 4 (see Figures 2(a) and (b)) as a button for a re-execution instruction, as long as it does not conflict with other operations using the controller 4.

[0157] If it is determined that there is a command to rerun (YES), the process returns to step S25. This resets time t to 0, allowing the simulation to be restarted from the beginning.

[0158] On the other hand, if it is determined in step S32 that there is no instruction to re-execute (NO), the process proceeds to step S33. In step S33, the VR image generation unit 25 determines, in the same manner as in step S22 described above, whether there is an operation to terminate the process of displaying the VR image 5. If the VR image generation unit 25 determines that there is a termination operation (YES), it terminates the process of displaying the VR image 5.

[0159] On the other hand, if it is determined that there is no termination operation (NO) in step S33, the process proceeds to step S34. In step S34, the execution unit 27 increments time t by a predetermined increment Δt. As a result, each time step S34 is executed, time t advances by Δt, enabling prediction of the internal structure using a real-time simulation with a time step size of Δt.

[0160] Next, we will explain the process when "Display experimental results" is determined in step S14. In this case, the process moves to step S36, where the VR image generation unit 25 obtains display information 31 from the storage unit 20. For example, the VR image generation unit 25 refers to the aforementioned display source information which associates text 40a to 40f (see Figure 19) with display information 31, identifies the display information 31 associated with the text selected by the user from among the texts 40a to 40f, and obtains that display information 31 from the storage unit 20.

[0161] Next, the acquisition unit 24 acquires the user U's actions from the HMD device 3 (step S37).

[0162] Next, the VR image generation unit 25 generates a VR image 5 of the field of view corresponding to the user U's actions, which was acquired in step S37, based on the display information 31 acquired in step S36 (step S38).

[0163] Next, the display control unit 29 controls the display of the VR image 5 on the HMD device 3 (step S39).

[0164] Then, the VR image generation unit 25 determines, in the same manner as in step S22, whether there is an operation to terminate the process of displaying the VR image 5 (step S40). If the VR image generation unit 25 determines that there is an operation to terminate (YES), it terminates the process of displaying the VR image 5.

[0165] On the other hand, if there is no termination operation (NO), the process returns to step S37. As a result, the acquisition of user actions (step S37) and the generation of VR image 5 (step S38) are repeated until a termination operation is performed. Consequently, VR image 5 is dynamically generated from a viewpoint corresponding to user U's actions, allowing the user to observe the internal structure of the material obtained as an experimental result while immersed in the virtual space.

[0166] <Hardware Configuration> Next, we will describe the hardware configuration of visualization device 2.

[0167] Figure 25 is an example of a hardware configuration diagram of the visualization device 2 according to this embodiment. As shown in Figure 25, the visualization device 2 includes a storage device 101, memory 102, processor 103, USB host controller 104, NIC (Network Interface Card) 105, and media reader 106. These components are interconnected by a bus 107.

[0168] Of these, the storage device 101 is a non-volatile storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores the visualization program 110 according to this embodiment.

[0169] Alternatively, the visualization program 110 may be recorded on a computer-readable recording medium 111, and the processor 103 may read the visualization program 110 via a media reader 106.

[0170] Such recording media 111 include, for example, CD-ROMs (Compact Disc - Read Only Memory), DVDs (Digital Versatile Discs), and USB memory sticks, which are physically portable recording media. Alternatively, semiconductor memory such as flash memory or hard disk drives may also be used as recording media 111. These recording media 111 are not temporary media like carrier waves that do not have a physical form.

[0171] Furthermore, the visualization program 110 may be stored in a device connected to a public network, the internet, or a LAN. In that case, the processor 103 can simply read and execute the visualization program 110.

[0172] On the other hand, memory 102 is hardware that temporarily stores data, such as DRAM (Dynamic Random Access Memory).

[0173] The processor 103 is hardware such as a CPU (Central Processing Unit) and a GPU (Graphical Processing Unit) that controls various parts of the visualization device 2. The processor 103 also works in cooperation with the memory 102 to execute the visualization program 110.

[0174] In this way, the memory 102 and the processor 103 work together to execute the visualization program 110, thereby realizing the control unit 23 (see Figure 16).

[0175] Furthermore, the storage unit 20 (see Figure 16) is realized by the storage device 101 and the memory 102.

[0176] Furthermore, the USB host controller 104 is hardware for connecting the HMD3 to the visualization device 2 via USB communication. The first communication unit 21 (see Figure 16) is realized by the USB host controller 104.

[0177] Furthermore, NIC105 is hardware for connecting the visualization device 2 to the network 30 (see Figure 16). The NIC105 enables the realization of the second communication unit 22 (see Figure 16).

[0178] The media reader 106 is hardware such as a CD drive, DVD drive, and USB interface for reading the recording medium 111. [Explanation of Symbols]

[0179] 1…Visualization system, 2…Visualization device, 3…HMD, 4…Controller, 5…VR image, 7, 8…USB cable, 9…Pointer, 10…Phase boundary surface, 11…Label, 12…Horizontal axis, 13…Vertical axis, 14…Connecting line, 15…Label, 16…Isosurface, 17a, 17b, 17c…Variant, 18…Stress field, 20…Storage unit, 21…First communication unit, 22…Second communication unit, 23…Control unit, 24…Acquisition unit, 25…VR image generation unit, 26…Reception unit, 27…Execution unit, 28...Calculation unit, 29...Display control unit, 30...Network, 31...Display information, 31a, 31b...Sub-information, 35...Virtual space, 40...Selection screen, 40a, 40b, 40c, 40d, 40e, 40f...Text, 41...Highlight display frame, 101...Storage device, 102...Memory, 103...Processor, 104...USB host controller, 105...NIC, 106...Media reader, 107...Bus, 110...Visualization program, 111...Recording medium.

Claims

1. Obtaining user actions, To generate a VR image that visualizes information about the internal structure of the material in stereoscopic vision within the field of view corresponding to the acquired operation, Displaying the aforementioned VR image on a display device, A visualization program to enable a computer to execute a task.

2. The aforementioned information includes predictions or experimental results regarding the internal structure of the material, The visualization program according to claim 1.

3. The computer is then made to run the simulation program that performs the prediction. The aforementioned information is information that shows the execution results of the simulation program. The visualization program according to claim 2.

4. The computer is made to execute the simulation program in real time. The visualization program according to claim 3.

5. The computer is instructed to temporarily suspend the execution of the simulation program. The visualization program according to claim 4.

6. The aforementioned information is information generated by a computer other than the aforementioned computer. The visualization program according to claim 2.

7. To the aforementioned computer, This involves performing additional calculations that were not included in the aforementioned prediction, In generating the aforementioned VR image, the results of the additional calculation are superimposed onto the aforementioned VR image. A visualization program according to claim 2 for further execution of the above.

8. The aforementioned prediction is a prediction of the phase diagram of the material, The aforementioned additional calculation is a calculation to identify the connecting lines in the state diagram. The visualization program according to claim 7.

9. The computer is further made to accept the user's input operation, The aforementioned additional calculation is performed when the aforementioned input operation is received. The visualization program according to claim 7 or 8.

10. The aforementioned input operation is an operation on the operation input device, To the aforementioned computer, The additional calculation is performed based on the position of the point indicated by the operation input device in the VR image. The visualization program according to claim 9.

11. The computer is further instructed to generate a selection screen that prompts the user to select one of a plurality of simulation programs for making the prediction, or one of the results of a plurality of experiments. The visualization program according to claim 2.

12. The aforementioned prediction is a prediction concerning the field in the material, In generating the aforementioned VR image, the VR image is generated in which the field is visualized. The visualization program according to claim 2.

13. In generating the aforementioned VR image, the VR image is generated in which the isosurface or cross-section of the field is visualized. The visualization program according to claim 12.

14. The aforementioned field is one of the following: market price, concentration field, temperature field, magnetic field, electric field, velocity field, stress field, strain field, energy field, potential field, and order field. The visualization program according to claim 12 or claim 13.

15. The aforementioned prediction is a numerical calculation concerning the internal structure of the material. The visualization program according to claim 2.

16. The aforementioned numerical calculations are one of the following: CALPHAD method, phase-field method, density functional theory, molecular orbital method, GW method, Hartree-Fock method, and molecular dynamics method. The visualization program according to claim 15.

17. The aforementioned display device is a head-mounted display. A visualization program according to any one of claims 1 to 16.

18. The aforementioned actions are obtained by acquiring signals indicating the user's position and orientation from the head-mounted display. The visualization program according to claim 17.

19. A unit that acquires user actions, A VR image generation unit generates a VR image that visualizes information about the internal structure of the material in stereoscopic vision within the field of view corresponding to the acquired operation, A display control unit that causes the VR image to be displayed on a display device, A visualization device having the following features.

20. Obtaining user actions, To generate a VR image that visualizes information about the internal structure of the material in stereoscopic vision within the field of view corresponding to the acquired operation, Displaying the aforementioned VR image on a display device, A visualization method performed by a computer.

Images