Information processing apparatus, information processing method, and program

Abstract

To provide an information processing device that outputs an image showing a state in which an external force of a magnitude corresponding to an input from a user is applied to an object.SOLUTION: The information processing device includes a generation unit that generates an output image including an object image indicating a predetermined object, an output unit that outputs the output image, and a reception unit that receives an operation of a user on the object image, wherein the generation unit deforms or moves the object image in the output image based on the operation of the user received by the reception unit, and when the operation of the user satisfies a predetermined condition, the generation unit further generates a simulation image indicating a simulation according to the operation of the user, and the output unit further outputs the simulation image.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, services have been provided that perform simulations of fluids such as liquid cosmetics. Patent Document 1 discloses a makeup simulation method that superimposes a simulation image pattern for a makeup item on an image of a subject's face to form a simulation image of a made-up face. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-257194 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there has been a demand for the development of technology that can reproduce tactile sensations using digital devices. [Means for solving the problem]

[0005] An information processing device according to one aspect of the present invention includes: a first acquisition unit that acquires a plurality of feature parameters; The device includes a second acquisition unit that acquires position information of the moving object, a generation unit that generates, for each of a plurality of feature parameters, an output image showing a sample fluid that is deformed or moves by the moving object in accordance with changes in the position information, or an output image showing a moving object that moves on the sample fluid in accordance with changes in the position information, based on each of the feature parameters, an output unit that outputs the output image, and a reception unit that receives designation of a feature parameter corresponding to a specified object from among the plurality of feature parameters, wherein the generation unit further generates a simulation image showing a simulation using the moving object and fluid based on the feature parameters received by the reception unit, and the output unit further outputs the simulation image. [Effects of the Invention]

[0006] According to the present invention, it is possible to realize an information processing device, an information processing method, and a program that can improve the convenience of users in simulations. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating a schematic configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating a schematic configuration of a terminal device. [Figure 3] FIG. 2 is a diagram illustrating a schematic configuration of a server device. [Figure 4] FIG. 10 is a diagram showing a fluid table. [Figure 5] FIG. 10 shows a sample fluid table. [Figure 6] FIG. 1 is a diagram illustrating a schematic configuration of an edge computer device. [Figure 7] 10 is a flowchart illustrating an example of an overall flow of information processing executed by a server device. [Figure 8] 10 is a flowchart illustrating an example of a calibration process. [Figure 9A] FIG. 10 is a diagram showing a first output image. [Figure 9B]FIG. 10 is a diagram illustrating the movement of a moving object. [Figure 10] FIG. 10 is a diagram showing a second output image. [Figure 11] FIG. 10 is a diagram showing an updated second output image. [Figure 12] 10 is a flowchart showing a flow of a preparatory movement process. [Figure 13A] FIG. 10 is a diagram showing a third output image in the preparatory movement processing. [Figure 13B] FIG. 10 is a diagram showing an example of an updated third output image in the preparatory movement processing. [Figure 14] FIG. 10 is a diagram showing another example of the third output image in the preparatory movement processing. [Figure 15] 10 is a flowchart showing a specific example of a simulation process. [Figure 16A] FIG. 10 is a diagram showing an initial image of a support object. [Figure 16B] FIG. 10 is a diagram showing an initial image of a fluid. [Figure 16C] FIG. 10 shows an updated simulation image. [Figure 16D] FIG. 10 is a diagram showing a simulation image after a moving object is operated. [Figure 17] 10 is a flowchart illustrating an example of a setting process executed by the server device. DETAILED DESCRIPTION OF THE INVENTION

[0008] An information processing device, an information processing method, and a program according to one aspect of an embodiment will be described below with reference to the drawings. However, please note that the technical scope of the present invention is not limited to the embodiment, but extends to the inventions set forth in the claims and their equivalents.

[0009] FIG. 1 is a diagram showing a schematic configuration of an information processing system 1 according to an embodiment.

[0010] As shown in FIG. 1, the information processing system 1 includes, for example, a terminal device 2 and a server device 3. The information processing system 1 may also include an edge computer device 4. The number of terminal devices 2 is not limited to one, and multiple terminal devices 2 may be provided. The terminal devices 2, the server device 3, and the edge computer device 4 are interconnected via a network N. The network N is a wired network such as the Internet or an intranet. The network N may also be a wireless network such as a wireless LAN (Local Area Network).

[0011] FIG. 2 is a diagram showing a schematic configuration of the terminal device 2 according to the embodiment.

[0012] The terminal device 2 is an example of an information processing device. The terminal device 2 is a multi-function mobile phone (a so-called smartphone). The terminal device 2 may be a personal computer, a notebook PC, a tablet PC, a portable game console, or the like. The terminal device 2 has a terminal communication device 201, a terminal input device 202, a terminal display device 203, a terminal storage device 204, an imaging device 205, and a terminal processing device 220. The terminal communication device 201, the terminal input device 202, the terminal display device 203, the terminal storage device 204, the imaging device 205, and the terminal processing device 220 are connected to each other via a CPU (Central Processing Unit) bus or the like.

[0013] The terminal communication device 201 has an antenna for receiving wireless signals and a wireless communication interface circuit conforming to a communication protocol such as wireless LAN, and is communicatively connected to the network N in accordance with the communication standard such as wireless LAN. The terminal communication device 201 sends data received from the server device 3, the edge computer device 4, etc. via the network N to the terminal processing device 220. The terminal communication device 201 also transmits data received from the terminal processing device 220 to the server device 3, the edge computer device 4, etc. via the network N. The terminal communication device 201 may have a wireless communication interface circuit conforming to a communication standard such as LTE (Long Term Evolution) or 5G, and may be communicatively connected to the network N via a base station. The terminal communication device 201 may also be communicatively connected to the network N in accordance with a communication protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol).

[0014] The terminal input device 202 has an input device and an interface circuit that acquires signals from the input device, and outputs an operation signal in response to an input operation by a user. Examples of the input device include a keyboard, a pointing device, a microphone, a camera, a voice input system, a gesture input system, and an eye-gaze input system. The pointing device may be any device used by the user of the terminal device 2 for input operations to specify a position or coordinates on the screen displayed on the terminal display device 203, and the details thereof are not particularly limited. Examples of the pointing device include a touch panel, a mouse, a trackball, and a trackpad.

[0015] In this embodiment, the terminal input device 202 accepts an input operation (sometimes referred to as a coordinate designation operation) by a user of the terminal device 2, for designating a position or coordinates on the screen displayed on the terminal display device 203. The terminal input device 202 outputs an operation signal in response to the input operation, a signal (sometimes referred to as a coordinate designation signal) for designating a position or coordinates on the screen displayed on the terminal display device 203.

[0016] The terminal display device 203 is an example of an output unit. The terminal display device 203 has a display such as a liquid crystal display or an organic EL (Electro-Luminescence) display, and an interface circuit that outputs image data and moving image data to the display, and displays the image data and moving image data on the display.

[0017] The terminal storage device 204 is an example of a storage unit. The terminal storage device 204 includes a memory device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a fixed disk device such as a hard disk, or a portable storage device such as a flexible disk or optical disk. The terminal storage device 204 also stores computer programs, databases, tables, etc. used for various processes of the terminal device 2. The computer programs may be installed into the terminal storage device 204 from a computer-readable portable recording medium using a known setup program, etc. The portable recording medium is, for example, a CD-ROM (Compact Disc-Read Only Memory) or a DVD-ROM (Digital Versatile Disc-Read Only Memory). The computer programs may be stored in a recording medium owned by a predetermined server and installed via the network N.

[0018] The imaging device 205 has an image sensor such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) that is integrated with the terminal device 2, converts the optical signal focused on the imaging surface into an imaged image by photoelectric conversion, and outputs the converted imaged image to the terminal processing device 220.

[0019] The terminal processing device 220 operates based on a program stored in the terminal storage device 204. The terminal processing device 220 is, for example, a CPU. A DSP (Digital Signal Processor), an LSI (Large Scale Integration), an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like may be used as the terminal processing device 220. The terminal processing device 220 is connected to the terminal communication device 201, the terminal input device 202, the terminal display device 203, the terminal storage device 204, the imaging device 205, and the like, and performs overall control of each device. This allows the terminal processing device 220 to perform processing on data received from the server device 3 and the edge computer device 4.

[0020] The terminal processing device 220 acquires an image captured by the imaging device 205 and transmits the acquired image to the server device 3 or the edge computer device 4 via the terminal communication device 201. The image may be a still image or a moving image.

[0021] The terminal processing device 220 may monitor the state of the network N. Based on the state of the network N, the terminal processing device 220 may execute processing to reduce the amount of data of images acquired from the imaging device 205 and / or images transmitted to the outside. For example, when it is detected that the state of the network N has deteriorated below a predetermined state, the terminal processing device 220 executes processing to reduce the resolution of the image or reduce the size of the image. Examples of the state of the network N include latency, bandwidth, and throughput.

[0022] 3 is a diagram showing a schematic configuration of the server device 3. The server device 3 is an example of an information processing device.

[0023] (Overview of server device 3) For the purpose of facilitating understanding of the configuration of the server device 3, in this embodiment, an overview of the server device 3 will be described using an example in which the server device 3 provides a service that presents the state of deformation and / or movement of an object (which may also be referred to as a target or an object) when an external force is applied to the object. More specifically, the server device 3 acquires information indicating an input operation from a user of the terminal device 2. The server device 3 generates information for presenting to the user of the terminal device 2 an image indicating the state of deformation and / or movement of the object when an external force corresponding to the input operation is applied to the object. For example, the server device 3 generates a computer graphic of the object to which an external force of a magnitude corresponding to the input from the user is applied. The server device 3 also outputs the information to the terminal device 2. For example, the server device 3 outputs data of the image to the terminal device 2. The server device 3 may cause the terminal device 2 to execute a process for presenting the image to the user. For example, the server device 3 outputs a command to the terminal device 2 for presenting the image to the user. This allows the user to see how the object is deformed and / or moved as a result of an external force being applied to the object in response to the input operation.

[0024] According to this embodiment, the physical properties of an object (particularly physical properties related to tactile sensation) can be visually reproduced or simulated by adjusting the manner in which the content of an input operation is reflected in an image. Examples of the manner in which the input operation is reflected in an image include (i) the magnitude of the time lag until the input operation is reflected in an image and / or the manner in which the magnitude fluctuates or changes, and (ii) the magnitude of the deviation between the position in the image indicated by the input operation and the position in the image where the content of the input operation is reflected and / or the manner in which the magnitude fluctuates or changes.

[0025] Examples of the physical properties of an object include mechanical properties, electrical properties, thermal properties, and surface properties. Examples of mechanical properties include viscoelasticity, dynamic viscoelasticity, breaking strength, breaking elongation, Young's modulus, and hardness. Examples of surface properties include coefficient of friction, surface tension, surface roughness, and gloss. Examples of thermal properties include dimensional change rate, thermal conductivity, and temperature. The physical properties of an object may be physical properties related to the sense of touch. Examples of physical properties related to the sense of touch include at least one of mechanical properties, thermal properties, and surface properties. The physical properties related to the sense of touch may be at least one of viscoelasticity, dynamic viscoelasticity, coefficient of friction, surface tension, surface roughness, thermal conductivity, and temperature.

[0026] For example, the greater the time lag, the greater the user of the terminal device 2 may perceive the object to have a higher viscosity or dynamic viscosity. The greater the time lag, the greater the user of the terminal device 2 may perceive the object to have a higher coefficient of friction. Similarly, the greater the positional deviation, the greater the user of the terminal device 2 may perceive the object to have a higher viscosity. The greater the positional deviation, the greater the user of the terminal device 2 may perceive the object to have a higher coefficient of friction. Other properties of the object may be perceived in a similar manner.

[0027] The user of the terminal device 2 may be an example of a user of the information processing system 1. The service provided by the server device 3 may be an example of a service provided by the information processing system 1.

[0028] In the present specification, there are cases where a process of outputting an image to the terminal device 2 is executed. The process of outputting an image to the terminal device 2 may be an example of a process for presenting an image or information to a user of the terminal device 2. Examples of the process of outputting an image to the terminal device 2 include (i) a process for outputting image data to the terminal device 2, (ii) a process for generating information used by the terminal device 2 to present an image to the user and / or a process for outputting the information to the terminal device 2, and (iii) a process for causing the terminal device 2 to execute a process for presenting an image to the user.

[0029] (Examples of deformation and / or movement of objects) According to one embodiment of the service, a state of deformation and / or movement of a first object when an external force is applied to the first object due to a collision or contact between the first object and a second object is presented. As described below, the first object and the second object may be different objects. The second object may be placed on the first object.

[0030] When a first object and a second object are in contact with each other and the second object moves, the first object may be deformed or moved due to the interaction between the first object and the second object. The state in which the first object and the second object are in contact with each other may be an example of a state in which an interaction occurs between the first object and the second object.

[0031] Examples of interactions include mechanical interactions, physical interactions, and chemical interactions. Examples of physical interactions include electrical interactions, thermal interactions, and magnetic interactions. Examples of the above interactions include action, reaction, tension, compression, shear, bending, torsion, strain, resistance, repulsion, and adsorption. Examples of forces generated by the above interactions include tension, pressure, shear force, bending force, torsion force, stress, repulsion force, adsorption force, repulsion force, adhesion force, and friction force.

[0032] According to this embodiment, for example, when a person spreads a fluid using a finger, the state of deformation and / or movement of the fluid is presented. For example, the server device 3 acquires information indicating an input operation of the user of the terminal device 2, which corresponds to the finger movement spreading the fluid. For example, the server device 3 outputs to the terminal device 2 an image indicating the state of deformation and / or movement of the fluid in accordance with the finger movement.

[0033] As will be described later, the images may include, for example, an image showing a first object deforming or moving on a third object (which may be referred to as a first image), and / or an image showing a second object moving on a third object (which may be referred to as a second image). In the first image, the first object and the third object may be in contact with each other, or another object may be disposed at least partially between the first object and the third object. Similarly, in the second image, the second object and the third object may be in contact with each other, or another object may be disposed at least partially between the second object and the third object.

[0034] (Examples of deformation and / or movement of objects) According to another embodiment of the above service, a state of deformation and / or movement of a first object when an external force is applied to the first object due to a collision or contact between a first object placed on a third object and a second object is presented. In this embodiment, the state of deformation and / or movement of the first object and the state of deformation and / or movement of the third object may be presented. As described below, the first object, the second object, and the third object may be different objects. The second object may be placed on the first object and the third object.

[0035] When a first object disposed on a third object moves while in contact with a second object, the first object may be deformed and / or moved due to the interaction between the first object, the second object, and the third object. More specifically, the first object may be deformed and / or moved due to the interaction between the first object and the third object and the interaction between the first object and the second object.

[0036] In a case where the deformation and / or movement of a third object is presented, when the third object and the second object are not in contact, the third object may be deformed and / or moved by the interaction between the first object and the third object. On the other hand, when at least a portion of the third object is in contact with the second object, the third object may be deformed and / or moved by the interaction between the first object and the third object and the interaction between the third object and the second object.

[0037] According to this embodiment, for example, when a person uses their fingers to spread a fluid placed on their own or another person's forearm, the deformation and / or movement of the fluid is displayed. The forearm may be an example of skin. According to this embodiment, for example, when a person presses or spreads lipstick protruding from a container held in their hand onto their own or another person's lips, the deformation and / or movement of the lipstick is displayed. The lips may be an example of skin.

[0038] (Examples of deformation and / or movement of objects) According to yet another embodiment of the above service, a state of deformation and / or movement of a third object when an external force is applied to the third object due to a collision or contact between the third object and the second object is presented. This embodiment differs from the above embodiment in that the first object penetrates inside the third object.

[0039] As will be described later, the first object, the second object, and the third object may be different objects. According to this embodiment, for example, when a fluid has permeated a person's forearm and the person strokes the surface of the forearm with the person's fingers, deformation and / or movement of the forearm are displayed.

[0040] According to one embodiment, the deformation and / or movement of the first object and the deformation and movement of the third object are derived independently. According to another embodiment, the third object into which the first object has penetrated is treated as a different type of object having properties different from those of the first object and the third object. In this case, the deformation and / or movement of the different type of object due to collision or contact with the second object is presented. This can significantly reduce the amount of calculation. In this case, the physical properties of the third object into which the first object has penetrated may be different from those of the first object or may be different from those of the third object. Note that even in these cases, the second object may be placed on top of the first object.

[0041] The fluid may be an example of a first object. The first object may be an example of a target object. A finger may be an example of a second object. The second object may be an example of a moving object. Skin may be an example of a third object. The third object may be an example of a supporting object. The first image may be an example of an output image. The second image may be an example of an output image.

[0042] (Image generation process overview) In these embodiments, the deformation and / or movement of the first object and / or the third object is reproduced or simulated, for example, by computer simulation (which may simply be referred to as simulation). For example, the server device 3 includes an image generation unit that generates an image in which the deformation and / or movement of the first object and / or the third object is reproduced or simulated.

[0043] In this embodiment, the image generation unit generates an image showing how an external force corresponding to a change in the position of the second object is applied to the first object, for example. The image generation unit may generate the above-described image by a physical calculation simulation that takes into account at least one of (i) contact or collision between the first object and / or the third object and the second object, (ii) friction between the first object and the third object, (iii) penetration or diffusion of the first object into the third object, and (iv) evaporation of the first object.

[0044] The image generation unit may generate an image showing the state of the first object and / or the third object when an external force corresponding to a change in the position of the second object is applied to the first object. As described above, the position of the second object changes in response to an input operation by the user. In this embodiment, when generating the image, the image generation unit may adjust the manner in which the content of the input operation is reflected in the image. This allows the physical properties of each object (particularly physical properties related to tactile sensation) to be visually reproduced or simulated. Details of the adjustment may be as described above.

[0045] This allows the image generation unit to generate an image (for example, the first image described above) such that the first object and / or the third object deform or move with a delay in response to the user's input. Similarly, the image generation unit can generate an image (for example, the second image described above) such that the second object moves with a delay in response to the user's input (this may be referred to as following).

[0046] In one embodiment, the image generation unit generates the above image by simulating the deformation and / or movement based on various parameters (sometimes referred to as feature parameters) related to the deformation and / or movement that are predetermined for each type of object (e.g., the first object and / or the third object). The image generation unit may acquire various parameters related to each object to be simulated by referring to a storage device (e.g., any database management device) that stores, for each of one or more objects, information indicating the type of the object and the various parameters in association with each other. In this case, the image generation unit may be an example of a physics engine.

[0047] Examples of the various parameters related to deformation and / or movement include parameters related to the magnitude of the various interactions described above, parameters related to the various physical properties described above, etc. Examples of the above parameters include a parameter indicating the ease of deformation, a parameter indicating the ease of separation, a parameter indicating the ease of movement, a parameter indicating the ease of evaporation or penetration, etc.

[0048] The storage device may store some of the parameters related to a specific object in association with information indicating the type of the specific object and information indicating the type of another object in contact with the specific object. For example, the ease of movement of a first object depends on the strength of the bond between the first object and a third object at their interface. The bond may be a mechanical bond, a physical bond, or a chemical bond. In this case, the parameter indicating the ease of movement of the first object may differ depending on the type of the third object supporting the first object. Therefore, the storage device may store information indicating the type of the first object, information indicating the type of the third object supporting the first object, and a parameter indicating the ease of movement of the first object on the surface of the third object in association with each other.

[0049] Similarly, the parameter indicating the ease of penetration of the first object may differ depending on the type of third object supporting the first object. Therefore, the storage device may store information indicating the type of the first object, information indicating the type of the third object supporting the first object, and the parameter indicating the ease of penetration of the first object into the third object in association with each other.

[0050] Similarly, a parameter indicating the ease of deformation of a third object and a parameter indicating the magnitude of friction on the surface of the third object may differ depending on the type and / or amount of a first object that has penetrated the third object. Therefore, the storage device may store information indicating the type of the first object, information indicating the type and / or amount of the first object that has penetrated the third object, and a parameter indicating the ease of deformation of the third object, in association with each other. The storage device may store information indicating the type of the first object, information indicating the type and / or amount of the first object that has penetrated the third object, and a parameter indicating the magnitude of friction on the surface of the third object, in association with each other.

[0051] In another embodiment, the image generator generates the image by inputting various conditions related to the simulation to a simulation execution device that executes the simulation and acquiring the results of the simulation from the simulation execution device. The results of the simulation may be an image that reproduces or simulates the deformation and / or movement of the first object and / or the third object. Examples of the conditions include the type, shape, structure, size, weight, speed, acceleration, and physical properties of each object.

[0052] The above conditions may include various parameters related to deformation and / or movement that are predetermined for each type of first object and / or third object, and information indicating the position or coordinates or their changes indicated by the above-mentioned coordinate designation operation or coordinate designation signal (sometimes referred to as designated position information).

[0053] The simulation execution device executes a simulation of the transformation and / or movement based on various parameters related to the transformation and / or movement that are predetermined for each type of the first object and / or the third object. The simulation execution device may execute the simulation of the transformation and / or movement based on the various parameters and specified position information.

[0054] The simulation execution device may acquire various parameters related to each object to be simulated by referring to a storage device (e.g., any database management device) that stores, for each of one or more objects, information indicating the type of the object and the various parameters described above in association with each other. The simulation execution device may be realized by a part of the server device 3, or may be located outside the server device 3. In this case, the simulation execution device may be an example of a physics engine.

[0055] The simulation execution device reproduces or simulates the deformation or movement of the first object and / or the third object due to the movement of the second object, for example, based on the above parameters and the specified position information. The simulation execution device executes a physics simulation that takes into account at least one of (i) contact or collision between the first object and / or the third object and the second object, (ii) friction between the first object and the third object, (iii) penetration or diffusion of the first object into the third object, and (iv) evaporation of the first object, for example, based on the above parameters and the specified position information.

[0056] In this embodiment, the simulation execution device derives, for example, a region in a physics simulation where a second object applies an external force to a first object and / or a third object. If the external force includes a mechanical force, the region may be a region where the second object contacts the first object and / or the third object, or may be a point that represents the region. The simulation execution device executes a physics calculation simulation to reproduce or simulate the application of the external force to the region. As described above, the external force may be generated by the movement of the second object.

[0057] As described above, in this embodiment, the position or coordinates of the moving object image on the screen are determined based on the position or coordinates of a point (sometimes referred to as a representative point) that represents the moving object image on the screen and the position or coordinates of the specified point described above. The position or coordinates of the moving object image on the screen are then determined so that the representative point of the moving object image matches or follows the movement of the specified point.

[0058] According to this embodiment, the representative position of the above region is determined to match or track the movement of the designated point according to a procedure similar to that of the representative point of the moving object image. For example, the simulation execution device determines the relative positional relationship between the representative position of the above region and the position indicated by the designated position information (i.e., the above-mentioned designated point) based on the above parameters for each object (particularly, parameters for an object that deforms or moves due to an external force). More specifically, the position or coordinates of the above region on the screen are determined so that the representative position of the above region matches or tracks the movement of the designated point. According to this embodiment, by adjusting the degree or manner of tracking, for example, the physical properties of at least one of the first object and the third object can be visually reproduced or simulated.

[0059] The above parameters may be an example of object information, and the designated position information may be an example of position information.

[0060] (Simulation details) The above simulation may be a physics calculation simulation (sometimes referred to as a physics simulation). In the above simulation, the first object and / or the third object may be modeled as a collection of a large number of mass points connected by elastic bodies. The various parameters described above may be expressed by at least one selected from the group consisting of the number of mass points, the mass assigned to each mass point, the strength of the connection between the mass points, the speed and / or acceleration at which the number of mass points changes, the speed and / or acceleration at which the mass assigned to each mass point changes, and the speed and / or acceleration at which the strength of the connection between the mass points changes.

[0061] Examples of the deformation include a change in shape, a change in size, and separation. Examples of the movement include a translational movement and a rotational movement. In the simulation, evaporation of an object and / or penetration or diffusion into another object may be reproduced or simulated. In the simulation, evaporation of an object and / or penetration or diffusion into another object may be represented as a decrease in the mass and / or volume of the object. In the simulation, as one object penetrates or diffuses into another object, parameters related to the other object may be changed.

[0062] In the above simulation, the shape and size of the second object may be constant, or may change due to collision or contact with the first object and / or third object. The change in the shape and size of the second object may be reproduced or simulated, for example, by simulation. The change in the shape and size of the second object may be determined based on various parameters related to deformation and / or movement that are predetermined for each type of second object. The above simulation may be a physics calculation simulation.

[0063] In the above simulation, the second object can be modeled as a collection of multiple mass points connected by elastic bodies. The various parameters described above can be expressed by at least one selected from the group consisting of the number of mass points, the mass assigned to each mass point, the strength of the connection between the mass points, the speed and / or acceleration at which the number of mass points changes, the speed and / or acceleration at which the mass assigned to each mass point changes, and the speed and / or acceleration at which the strength of the connection between the mass points changes.

[0064] The first object is an object (sometimes referred to as a main object) whose deformation or movement due to an external force is presented in the above service, and the details thereof are not particularly limited. The first object may be a real object or a fictional object. The first object may be an object having specific physical properties. The physical properties may be specified by the user or may be determined based on input from the user.

[0065] Examples of main objects include solids and fluids. Examples of fluids include liquids, slurries, emulsions, semi-solids, particle aggregates, and foam aggregates. More specifically, main objects may be skin, artificial skin, hair, artificial hair, organs, artificial organs, cosmetics, food, daily necessities, and the like. As described above, the main object may be skin or artificial skin permeated with liquid or cosmetics. Examples of cosmetics include liquid cosmetics, emulsion cosmetics, cream cosmetics, and powder cosmetics. Examples of food include water, honey, marshmallows, and rice cakes. Examples of daily necessities include containers, container lids, and leather (e.g., the skin of an animal other than a human).

[0066] The second object is an object different from the first object, and may be an object (sometimes referred to as a moving object) that moves within real space in response to the coordinate specification operation by the user, and the details thereof are not particularly limited. Examples of moving objects include an object that can be moved by the user in real space, and an object that can be used to indicate a position or coordinates on the screen displayed on the terminal display device 203.

[0067] More specifically, the moving object may be a part of the user's body or an object that the user can move using a part of their body. Examples of the part of the body include a hand or a finger. Examples of the object include a pen, a spatula, a brush, a cosmetic container, etc. Although the sense of touch is provided on the entire surface of the animal's body, the role played by the hand or fingers in perceiving tactile sensations is significantly greater than that of other parts of the body. Therefore, if the moving object is a part of the user's body or an object that the user can move using a part of their body, the accuracy of reproducing or simulating tactile sensations in the above service may be improved.

[0068] The third object is not particularly limited as long as it is an object different from the first object and the second object and can support the first object on or inside thereof (sometimes referred to as a support object). Examples of the support object include a part of the user's body, an item worn on a part of the user's body, an item whose texture is familiar to the user, or an everyday item. When the moving object and the support object are parts of the user's body, the support object may be another part that can come into contact with the part corresponding to the moving object. For example, when the moving object is the index finger of the user's right hand, the support object may be the forearm of the user's left arm.

[0069] In this embodiment, the server device 3 provides the above-described service by displaying an image of a main object (which may be referred to as a main object image) that deforms and / or moves in accordance with changes in the position of a moving object in real space, on a screen displayed on the terminal display device 203. The server device 3 may further display at least one of an image of a moving object (which may be referred to as a moving object image) and an image of a supporting object (which may be referred to as a supporting object image) on the above-described screen.

[0070] The main object image is, for example, a CG (Computer Graphics) image showing the main object. The main object image may be a CG image showing the appearance of the main object.

[0071] The moving object image is, for example, a CG image showing a moving object. The moving object image may be a CG image showing the appearance of the moving object. This allows the information processing system 1 to provide a VR (Virtual Reality) experience to a user of the information processing system 1. The moving object image may be generated based on an image (sometimes referred to as a captured image) of a moving object captured by any imaging device (for example, the imaging device 205). For example, by presenting a CG image obtained by superimposing the captured image of the moving object on the main object image to the user, the information processing system 1 can provide an AR (Augmented Reality) experience or an MR (Mixed Reality) experience to the user.

[0072] When a captured image of a moving object and a main object image are superimposed, at least one of the following may be adjusted: (i) the timing at which playback of the captured image of the moving object starts; (ii) the playback speed of the captured image of the moving object; (iii) the relative positional relationship between the captured image of the moving object and the main object image (sometimes referred to as a superimposition position); and (iv) the transparency or transparency of the captured image of the moving object and / or the main object image (sometimes referred to as a superimposition parameter). The value of the superimposition parameter is determined based on, for example, the physical properties of at least one of the main object and the supporting object. Adjustment of the superimposition parameter may be an example of adjustment of the manner in which the content of an input operation is reflected in an image.

[0073] For example, the greater the discrepancy between the timing of a moving object's movement in the real world and the timing at which playback of a captured image of the moving object begins, the greater the user of the terminal device 2 may perceive the object's viscosity, kinetic viscosity, coefficient of friction, and other properties as being higher. For example, the slower the playback speed of a captured image of the moving object, the greater the user of the terminal device 2 may perceive the object's viscosity, kinetic viscosity, coefficient of friction, and other properties as being higher. For example, the greater the discrepancy between the superimposed position in the real world and the superimposed position on the screen, the greater the user of the terminal device 2 may perceive the object's viscosity, kinetic viscosity, coefficient of friction, and other properties as being higher. The smaller the transparency or transparency of at least one of the captured image of the moving object and / or the main object image, the more easily the user of the terminal device 2 recognizes the discrepancy. Therefore, the user of the terminal device 2 may perceive the object's viscosity, kinetic viscosity, coefficient of friction, and other properties as being higher. Note that other properties of the object may also be perceived in a similar manner.

[0074] The supporting object image is, for example, a CG image showing the supporting object. The supporting object image may be a CG image showing the appearance of the supporting object. This allows the information processing system 1 to provide a VR (Virtual Reality) experience to a user of the information processing system 1. The supporting object image may be generated based on an image (sometimes referred to as a captured image) of the supporting object captured by any imaging device (for example, the imaging device 205). For example, by presenting a CG image obtained by superimposing the captured image of the supporting object on the main object image to the user, the information processing system 1 can provide an AR (Augmented Reality) experience or an MR (Mixed Reality) experience to the user.

[0075] As will be described later, when a captured image of a supporting object and a main object image are superimposed, at least one of (i) the timing at which playback of the captured image of the supporting object starts, (ii) the playback speed of the captured image of the supporting object, (iii) the relative positional relationship between the captured image of the supporting object and the main object image (sometimes referred to as a superimposition position), and (iv) the transparency or transparency of the captured image of the supporting object and / or the main object image (sometimes referred to as a superimposition parameter) may be adjusted. The value of the superimposition parameter is determined, for example, based on the physical properties of at least one of the main object and the supporting object.

[0076] (Overview of user experience provided by server device 3) As described above, the server device 3 can provide a user with a VR experience, an AR experience, or an MR experience. Furthermore, according to this embodiment, the visual effect of the image presented to the user allows the user to perceive the tactile sensation of the main object and / or the supporting object. Specifically, the server device 3 allows the user to perceive the tactile sensation of the main object and / or the supporting object by (b) adjusting the degree of tracking or delay (sometimes simply referred to as the degree of delay) of at least one of (i) the deformation and / or movement of the moving object image, (ii) the deformation and / or movement of the main object shown in the main object image, and (iii) the deformation and / or movement of the supporting object shown in the supporting object image, in response to (a) a change in the position or coordinates on the screen (sometimes referred to as the position or coordinates of the specified point) indicated by the coordinate specifying operation or coordinate specifying signal described above. This allows the user to perceive the tactile sensation of the main object and / or the supporting object.

[0077] (Moving moving object image) When a moving object image is displayed on a screen presented to a user, the position or coordinates of the moving object image on the screen are determined based on the position or coordinates of a point (sometimes referred to as a representative point) that represents the moving object image on the screen and the position or coordinates of the specified point described above. As described above, the position or coordinates of the specified point change depending on the coordinate specification operation.

[0078] In this embodiment, the position or coordinates of the moving object image on the screen are determined so that the representative point of the moving object image follows the movement of the designated point. As described above, according to this embodiment, by adjusting the degree or manner of following, the physical properties of at least one of the main object and the supporting object can be visually reproduced or simulated.

[0079] In one embodiment, the position or coordinates of the representative point are determined so that the representative point and the designated point coincide with each other. In another embodiment, the position or coordinates of the representative point are determined so that the position or coordinates of the representative point change with a delay relative to the change in the position or coordinates of the designated point. In these embodiments, the degree to which the representative point follows or lags behind the designated point is determined according to, for example, the physical properties of at least one of the main object and the support object. The physical properties of each object may be defined by various parameters related to the above-mentioned transformation and / or movement (sometimes simply referred to as transformation or movement).

[0080] In one embodiment, the degree of delay is determined such that, for example, the greater the viscosity or dynamic viscosity of the main object, the greater the degree of delay. In another embodiment, the degree of delay is determined such that, for example, the greater the magnitude of the interaction between the main object and the supporting object, the greater the degree of delay. The degree of delay may also be determined taking other properties of the object into consideration.

[0081] As described above, examples of interactions include mechanical interactions, physical interactions, and chemical interactions. Examples of physical interactions include electrical interactions, thermal interactions, and magnetic interactions. Examples of the above interactions include action, reaction, tension, compression, shear, bending, torsion, strain, resistance, repulsion, and adhesion. Examples of forces generated by the above interactions include tension, pressure, shear force, bending force, torsion force, stress, repulsion force, adhesion force, repulsion force, adhesion force, and friction force.

[0082] For example, the greater the viscosity or dynamic viscosity of the main object, the greater the degree of delay. Similarly, the greater the surface roughness of the support object, the greater the degree of delay. The relationship between other object properties and the degree of delay can be determined similarly.

[0083] In this way, the physical properties of the main object and / or supporting object can be visually reproduced or simulated depending on the degree of delay, so that the server device 3 can make the user viewing the screen displayed on the terminal display device 203 while performing a coordinate specification operation on the terminal input device 202 feel as if they are perceiving the tactile sensation of the main object and / or supporting object.

[0084] (Transform and / or move the main object) As described above, according to this embodiment, for example, the deformation and / or movement of the main object when an external force is applied to the main object due to a collision or contact between the main object and a moving object is reproduced or simulated. In one embodiment, the server device 3 determines the magnitude of the external force to be applied to the main object based on a change in the position or coordinates of the representative point of the moving object. In another embodiment, the server device 3 determines the magnitude of the external force to be applied to the main object based on a change in the position or coordinates of the representative point of the moving object and the shape and size of the moving object. In these embodiments, the server device 3 may determine the magnitude and direction of the external force.

[0085] As described above, the position or coordinates of the representative point of the moving object image can be determined so that the position or coordinates of the representative point changes in accordance with the change in the position or coordinates of the designated point. Furthermore, the degree of tracking or delay of the representative point relative to the designated point can be determined according to the physical properties of at least one of the main object and the supporting object. This allows the server device 3 to make a user viewing the screen displayed on the terminal display device 203 while performing a coordinate designation operation on the terminal input device 202 feel as if they are sensing the tactile sensation of the main object and / or the supporting object.

[0086] As described above, there may be cases where a moving object image is displayed on the screen presented to the user, and cases where a moving object image is not displayed on the screen presented to the user. Even when a moving object image is not displayed on the screen presented to the user, the magnitude of the external force applied to the main object, etc., may be determined by the same procedure as when a moving object image is displayed on the screen presented to the user.

[0087] In this way, when the server device 3 generates an image including a computer graphic of an object to which an external force of a magnitude corresponding to input from a user is applied, it acquires object information regarding the object to which the external force is applied, acquires position information of the moving object in real space, and, based on the object information, can generate (i) an image (e.g., a first image or a main object image) showing the object deforming or moving on the supporting object due to interaction with the moving object in accordance with changes in the position information, and / or (ii) an image (e.g., a second image or a moving object image) showing the moving object moving on the supporting object that supports the object in accordance with changes in the position information, based on the object information.

[0088] (An example of another embodiment) According to this embodiment, the information processing system 1 has been described in detail using as an example a case where the physical properties of an object (particularly physical properties related to tactile sensation) can be visually reproduced or simulated by adjusting the manner in which the content of an input operation is reflected in an image. However, the information processing system 1 is not limited to this embodiment. In other embodiments, the information processing system 1 may reproduce the physical properties of an object using auditory information in addition to or in addition to the visual changes in the image. When auditory information is reproduced in addition to the visual changes in the image, the sense of immersion is further improved.

[0089] For example, the information processing system 1 stores information about sounds as parameters of each object. The information processing system 1 may store identification information indicating the type of each object, information indicating the circumstances under which the sound is generated, and information about the sound in association with each other. Examples of sound information include information indicating the sound source, volume, timbre, pitch, ADSR (Attack, Decay, Sustain, Release), etc. The sound information may include a URI of the sound data. The sound information may also include parameters for modulating the sound according to the simulation conditions.

[0090] The information processing system 1 (for example, the server device 3) may store, as information about the sound of each object, information about different sounds for each type of other object that comes into contact with the object. As described above, in this embodiment, the main object may penetrate or diffuse into the inside of the support object. A support object in a dry or standalone state and a support object in a state in which another object has penetrated it may be treated as different types of objects.

[0091] When the object is a fluid, examples of situations in which sound is generated include a situation in which the fluid is ejected from a container, a situation in which application of the fluid onto a support object begins, a situation in which a predetermined time has passed since the application began, a situation in which predetermined conditions are met in the application, and a situation in which the application is completed.

[0092] The circumstances under which a sound is generated may be determined for each manner of movement of a moving object, such as a manner in which the moving object glides on a main object or a supporting object, a manner in which the moving object bounces on a main object or a supporting object, a manner in which the moving object is slammed onto a main object or a supporting object, a manner in which the moving object is pushed onto a main object or a supporting object, a manner in which the moving object is pulled up onto a main object or a supporting object, etc.

[0093] The conditions under which the sound is generated may be simulation conditions. If the main object is skin, the simulation conditions may include season, skin type, wetness, skin moisture content, skin viscoelasticity, transepidermal water loss, skin surface roughness, friction characteristics, vibration characteristics, color, gloss, amount of sebum, skin irregularities, pore size, or type of fluid applied in advance.

[0094] The information processing system 1 (for example, the server device 3) may play a sound triggered by the movement of a moving object. For example, a sound is played when the moving object is moving. The information processing system 1 may change parameters related to the sound playback in conjunction with the movement information of the moving object. For example, the information processing system 1 may link the volume with the movement information of the moving object, and adjust the volume so that the faster the moving speed of the moving object is, the louder the volume is.

[0095] Taking the example of a case where the main object is a liquid, the moving object is the user's finger, and the support object is an object that supports the liquid (sometimes referred to as a support), the information processing system 1 may implement the following processing. The sound source to be played is switched depending on the positional relationship between the finger and the liquid and the finger's movement. When the finger is rubbing a non-liquid part of the support, the sound of the support is played. When it is detected that the finger has touched the liquid, the sound of the liquid sliding is played. At this time, sound parameters may be changed in conjunction with the finger movement (speed, etc.). The support sound source and the fluid sound source may be played overlapping. The balance between the volume of the support sound source and the volume of the fluid sound source may be adjusted depending on the characteristics of the fluid. For example, in the case of a highly viscous liquid, the fluid sound source will be dominant. The difference in texture from the start of application to the end of application may be expressed using sound by switching sound sources, modulating sound parameters, modulating ADSR, or synthesizing. The start and end of application may be determined based on the total distance moved by the finger, the time since the start of movement, etc.

[0096] The image generator may be an example of a generator or a physics engine. The simulation execution device may be an example of a physics engine.

[0097] (Overview of each part of the server device 3) The server device 3 includes a server communication device 301, a server storage device 302, and a server processing device 320. The server communication device 301, the server storage device 302, and the server processing device 320 are connected to each other via a CPU bus or the like.

[0098] Server communication device 301 has a wired communication interface circuit that complies with a communication protocol such as TCP / IP. Server communication device 301 is communicatively connected to network N in accordance with a communication standard such as Ethernet (registered trademark). Server communication device 301 sends data received from terminal device 2, etc. via network N to server processing device 320. Server communication device 301 transmits data received from server processing device 320 to terminal device 2, etc. via network N. Server communication device 301 may also have an antenna that transmits and receives wireless signals and a wireless communication interface circuit that complies with a communication protocol such as wireless LAN, and may be communicatively connected to network N in accordance with a communication standard such as wireless LAN.

[0099] The server storage device 302 is an example of a storage unit. The server storage device 302 includes a memory device such as a RAM or a ROM, a fixed disk device such as a hard disk, or a portable storage device such as a flexible disk or an optical disk. The server storage device 302 also stores computer programs, databases, tables, and the like used for various processes of the server device 3. The computer programs may be installed into the server storage device 302 from a computer-readable portable recording medium such as a CD-ROM or a DVD-ROM using a known setup program or the like. The computer programs may be stored in a recording medium owned by a predetermined server and installed via the network N.

[0100] The server storage device 302 also stores various types of information such as a main object image, a moving object image, a support object image, support object feature information, a fluid table T1, a sample fluid table T2, etc. The main object image is an example of a first image.

[0101] In this embodiment, the support object feature information includes one or more items that indicate the physical properties or characteristics of the support object. For example, the support object feature information includes one or more items that indicate the characteristics of the skin of the left arm, which is the support to which the fluid is applied. In this specification, information that indicates the physical properties or characteristics of an object may be referred to as feature information.

[0102] Examples of items of support object characteristic information include the surrounding environment of the support object, the type of support object, and the physical properties of the support object. Examples of the surrounding environment of the support object include season, temperature, and humidity. Examples of the physical properties of the support object include at least one of moisture content, viscoelasticity, amount of moisture evaporation, surface roughness, degree of unevenness, friction characteristics, vibration characteristics, color, gloss, pore size, pore density, amount of sebum, and type of fluid applied in advance. An example of the fluid is cosmetics. When the support object is skin or skin, the type of the support object indicates a classification based on the characteristics of the support object. When the support object is skin or skin, examples of the type of the support object include dry skin, oily skin, combination skin, and sensitive skin.

[0103] The support object characteristic information may be an example of support object information.

[0104] Fig. 4 is a diagram showing a fluid table. As shown in Fig. 4, in the fluid table T1, for each of a plurality of fluids having different characteristics, identification information for identifying each fluid and fluid characteristic information for each fluid are associated with each other.

[0105] The fluid characteristic information includes one or more items that indicate the physical properties or characteristics of the fluid. Examples of the items of the fluid characteristic information include the color, gloss, size of the lump, deformability, spreading speed, evaporation speed, surface tension, viscoelasticity, transparency, deformability of the skin relative to the fluid, deformability of the finger relative to the fluid, or changes over time during deformation, deformability of the supporting object, adhesive properties, spinnability, liquid viscosity, viscosity flow curve, storage modulus, loss modulus, loss tangent, surface tension, density, static friction coefficient, kinetic friction coefficient, mean deviation of the friction coefficient, skin viscoelasticity, finger vibration characteristics, skin vibration characteristics, and changes in these characteristic amounts during the application process (at the time of discharge, at the start of application, during application, and after application).

[0106] As described above, examples of fluids include liquids, slurries, emulsions, semisolids, particle aggregates, and foam aggregates. Fluids may be liquids of any viscosity other than gases and solids, or may be jelly-like colloidal solutions. Colloidal solutions are solutions in which colloidal particles are uniformly dispersed. Fluids may also be fluids in which powders such as slurries and pastes are dispersed. Fluids also include solid cosmetics that can be applied to the skin, such as lipsticks, eye shadows, and powdery foundations. Fluids also include liquid cosmetics (hereinafter simply referred to as "products") such as lotions, emulsions, and perfumes provided or selected by the administrator or user of the information processing system 1.

[0107] The fluid characteristic information may be an example of object information. The fluid may be an example of a first object.

[0108] Fig. 5 is a diagram showing a sample fluid table. As shown in Fig. 5, in the sample fluid table T2, for each of a plurality of sample fluids with different characteristics, identification information for identifying each sample fluid and characteristic parameters of each sample fluid are associated with each other. A sample fluid is a medium that becomes a sample having certain characteristic parameters.

[0109] The characteristic parameters include one or more items indicating the physical properties or characteristics of the sample fluid. Examples of the characteristic parameters of the sample fluid include the color, gloss, size of the lump, ease of deformation, spreading speed, evaporation speed, surface tension, viscoelasticity, transparency, ease of deformation of the skin relative to the fluid, ease of deformation of the finger relative to the fluid, or change over time during deformation, ease of deformation of the support, adhesive properties, spinnability, liquid viscosity, viscosity flow curve, storage modulus, loss modulus, loss tangent, surface tension, density, static friction coefficient, kinetic friction coefficient, mean deviation of the friction coefficient, skin viscoelasticity, finger vibration characteristics, skin vibration characteristics, and changes in these characteristic amounts during the application process (at the time of discharge, at the start of application, during application, and after application).

[0110] A sample fluid may be an example of a sample object. Any storage device that stores a sample fluid table may be an example of a storage device. Identification information for identifying a sample fluid may be an example of information indicating a type of object. Characteristic parameters of the sample fluid may be an example of one or more parameters related to the sample object.

[0111] The server processing device 320 operates based on a program stored in advance in the server storage device 302. The server processing device 320 is, for example, a CPU. The server processing device 320 may be a DSP, an LSI, an FPGA, an ASIC, or the like. The server processing device 320 is connected to the server communication device 301 and the server storage device 302 and controls each device.

[0112] The server processing device 320 reads the computer program stored in the server storage device 302 and operates in accordance with the read computer program. This allows the server processing device 320 to function as a first acquisition unit 321, a second acquisition unit 322, a generation unit 323, a reception unit 324, a calculation unit 325, and a control unit 326.

[0113] The server communication device 301 may be an example of an output unit or a presentation unit. The server storage device 302 may be an example of a storage device. The server processing device 320 may be an example of an information processing device. The first acquisition unit 321 may be an example of a captured image acquisition unit. The second acquisition unit 322 may be an example of a second acquisition unit. The control unit 326 may be an example of an information processing device or a storage device. The control unit 326 may be an example of a reception unit, a first acquisition unit, a second acquisition unit, a generation unit, a presentation unit, a feedback acquisition unit, or a parameter adjustment unit.

[0114] 6 is a diagram showing a schematic configuration of the edge computer device 4. The edge computer device 4 may be an example of an information processing device.

[0115] In this embodiment, the edge computer device 4 is, for example, a distributed computer that is installed near the terminal device 2 and can execute data processing and the like in place of the server device 3 or in a distributed manner together with the server device 3. The edge computer device 4 has, for example, the same configuration as the server device 3. The edge computer device 4 has an edge communications device 401, an edge storage device 402, an edge processing device 420, and the like. The edge communications device 401, the edge storage device 402, and the edge processing device 420 have, for example, the same configuration as the server communications device 301, the server storage device 302, and the server processing device 320 of the server device 3, respectively.

[0116] (Outline of information processing according to the state of network N) If the state of the network N (sometimes referred to as communication quality) deteriorates, the immersive feeling experienced by the user may decrease. Therefore, in this embodiment, the terminal device 2, the server device 3, and / or the edge computer device 4 monitor the state of the network N. Based on the state of the network N, the terminal device 2, the server device 3, and / or the edge computer device 4 may perform processing to change the manner in which the above-mentioned service is provided and / or the respective roles of the multiple information processing devices that provide the service. Examples of the multiple information processing devices include the terminal device 2, the server device 3, and the edge computer device 4. Examples of the state of the network N include latency, bandwidth, and throughput.

[0117] In this embodiment, the terminal device 2, the server device 3 and / or the edge computer device 4 includes, for example, a communication monitoring unit that monitors the state of the network N, and a service management unit that manages the services provided to users in accordance with the state of the network N. The communication monitoring unit and the service management unit may be realized by the same information processing device, or may be realized by different information processing devices. Each of the communication monitoring unit and the service management unit may be realized by a single information processing device, or may be realized by multiple information processing devices.

[0118] In this embodiment, the communication monitoring unit may predict the state of the network N. The communication monitoring unit may predict the future state of the network N, for example, based on the monitoring history of the network N. If the communication monitoring unit determines that the current communication quality of the network N does not satisfy a predetermined standard, or if the communication monitoring unit predicts that the future communication quality of the network N will no longer satisfy the predetermined standard, the communication monitoring unit outputs an abnormality signal indicating an abnormality in the communication quality to the service management unit.

[0119] In this embodiment, when the service management unit receives an abnormality signal from the communication monitoring unit, the service management unit executes processing to manage the services provided to users in accordance with the state of the network N. The service management unit may execute processing in accordance with the degree of latency, in accordance with the degree of bandwidth, or in accordance with the degree of throughput. The service management unit may execute processing in accordance with a combination of at least two of the degree of latency, the degree of bandwidth, and the degree of throughput.

[0120] For example, the information processing for realizing the above-mentioned service is composed of a plurality of steps. In this case, the above-mentioned processing in the service management unit may include: (i) processing for suppressing a decline in service quality due to a deterioration in the state of the network N by executing at least one of the plurality of steps in consideration of the state of the network N (sometimes referred to as a first management processing), (ii) processing for suppressing a significant decline in service quality due to a deterioration in the state of the network N by speeding up at least one of the plurality of steps (sometimes referred to as a second management processing), (iii) processing for suppressing a decline in service quality due to a deterioration in the state of the network N by executing some of the steps executed by the server device 3 in the terminal device 2 and / or the edge computer device 4 (sometimes referred to as a third management processing), and (iv) processing for suppressing a decline in service quality due to a deterioration in the state of the network N via the network N. Examples of management processes include (i) a process for suppressing a decline in service quality due to a deterioration in the state of network N by reducing the amount of data sent and received via the network (sometimes referred to as the fourth management process), (ii) a process for suppressing a decline in service quality due to a deterioration in the state of network N by transmitting redundant image data (sometimes referred to as the fifth management process), (iii) a process for suppressing a decline in service quality due to a deterioration in the state of network N by transmitting data necessary for image generation in advance (sometimes referred to as the sixth management process), and (iv) a process for suppressing a significant decline in service quality due to a deterioration in the state of network N by having terminal device 2 output an alternative image (sometimes referred to as the seventh management process).

[0121] (An example of the first management process) As described above, according to this embodiment, the degree of delay in deformation and / or movement of each object in the image is adjusted in response to a change in the position or coordinates of the specified point. The service management unit may determine the degree of delay in deformation and / or movement of each object, taking into account the degree of latency of the network N.

[0122] For example, the time tt from when a user executes an input operation until the content of the input operation is reflected in the deformation and / or movement of an object in an image presented to the user is derived as the sum of the latency tr of the network N and the degree of delay td in the deformation and / or movement of the object. Therefore, when the measured or predicted value of the latency of the network N exceeds a threshold, the service management unit may reduce the degree of delay in the deformation and / or movement of the object by an amount equivalent to the difference between the measured or predicted value and the threshold.

[0123] This allows the service management unit to suppress degradation of service quality while suppressing degradation of image quality. Examples of image quality include simulation accuracy, drawing accuracy, image resolution, screen resolution, and the number of bits used to represent colors. The process of lowering image quality and the first management process described above may be used in combination.

[0124] Examples of simulation accuracy include the number of mass points used to construct the simulation model, the degree of simplification of the simulation model, the type of method for collision detection, the type of method for transmitting external forces, the type of physical phenomenon to be reproduced, whether or not moving objects are deformed, whether or not interactions between supporting objects and main objects are considered, etc. Examples of rendering accuracy include the degree of simplification of the simulation model, whether ray tracing is turned on or off, whether and / or to what extent light sources are reflected, etc.

[0125] This allows the service management unit to suppress degradation of service quality while suppressing degradation of image quality. Examples of image quality include simulation accuracy, image resolution, screen resolution, and the number of bits used to represent colors. The process of degrading image quality and the above process may be used in combination.

[0126] (An example of the second management process and / or the third management process) For example, by lowering the image quality described above, at least one of the multiple processes can be sped up. This can cancel a portion of the increase in latency of the network N. For example, if the terminal device 2 and / or the edge computer device 4 is equipped with a simulation execution device with lower performance than the simulation execution device equipped with the server device 3, the service management unit controls the server device 3 to execute the simulation process when the measured or predicted latency of the network N is smaller than a threshold value, and controls the terminal device 2 and / or the edge computer device 4 to execute the simulation process when the measured or predicted latency of the network N is larger than the threshold value. This further suppresses a deterioration in service quality due to a deterioration in the state of the network N.

[0127] The type of processing transferred from the server device 3 to the terminal device 2 and / or the edge computer device 4 is not limited to simulation processing. The processing to be transferred may be drawing processing.

[0128] (An example of the fifth management process) For example, when transmitting an image based on the position information of a moving object to the terminal device 2, the server device 3 and / or the edge computer device 4 may transmit an image based on the predicted movement direction and movement speed of the moving object. If the state of the network N deteriorates, the terminal device 2 may output an image based on the predicted movement direction and movement speed until the state of the network N recovers.

[0129] The service management unit may determine whether the above processing is necessary. For example, if it is detected that the state of network N has deteriorated from the first state, or if it is predicted that the state of network N will deteriorate from the second state, the service management unit may determine to start the above processing. If it is detected that the state of network N has recovered to a state better than the third state, or if such recovery is predicted, the service management unit may determine to end the above processing. The first state and the second state may be the same or similar states, and the first state may be a state better than the second state. The third state may be a state better than the first state.

[0130] (Example of information processing according to the state of network N) Details of information processing according to the state of the network N will be explained using specific examples of the operation of each part of the information processing system 1 when a deterioration in the state of the network N is detected or predicted. Note that the subject of each operation is not limited to the specific examples below. Within the scope of technical compatibility, each part of the terminal device 2, the server device 3, or the edge computer device 4 can be the subject of each operation.

[0131] When the terminal communication device 201 of the terminal device 2 detects a delay in downloading simulation data from the server device 3, the terminal processing 201 transmits a data amount reduction command to the server device 3. Upon receiving the command, the server communication device 301 of the server device 3 commands the server processing device 320 to reduce the data amount during simulation. The server processing device 320 (control unit 326) optimizes the amount of expression data for texture / skin / texture and skin according to the simulation content and reduction amount to be expressed on the terminal device 2, and commands the generation unit 323.

[0132] For example, in order to reduce gloss information in texture representation, the control unit 326 controls normal vector information of the CG image and issues a command to the generation unit 323. To simplify the representation of unevenness of the skin, the control unit 326 controls bump mapping parameters and issues a command to the generation unit 323. The control unit 326 controls the 3D / 2D representation of the CG image of each object and issues a command to the generation unit 323. The control unit 326 also controls the image resolution and bit-level of image color information of each object and issues a command to the generation unit 323.

[0133] In the AR mode, when the server communication device 301 of the server device 3 detects a delay in uploading image information from the imaging device 205 of the terminal device 2, the functions of detecting moving objects from the captured image and detecting coordinates within the image, which were previously performed by the second acquisition unit 322 of the server device 3, are changed to be performed by the edge processing device 420 of the edge computer device 4.

[0134] The simulation function is restricted due to the download delay 0196 or the upload delay 0197. For example, to restrict the simulation function when multiple cosmetic items are used, the cosmetic item selection log from the terminal storage device 204 is obtained from the terminal processing device 220, and the user is prompted to select the cosmetic items to be experienced (left) on the terminal display device 203. The terminal input device 202 obtains information about the cosmetic items selected by the user and transmits the information from the terminal communication device 201 to the server communication device 301. Based on the received cosmetic item information, the control unit 326 of the server processing device 320 obtains the necessary parameters from the cosmetic item information from the server storage device 302 and commands the calculation unit 325 to recalculate. The generation unit 323 is then commanded to generate the recalculated texture object, which generates an image. The generated image is transmitted from the server communication device 301 to the terminal communication device 201 and displayed on the terminal display device 202. In limiting the simulation function based on application time, in order to limit simulation of application times that involve a large amount of image generation data, the terminal processing device 220 acquires the application time log selected through experience from the terminal storage device 204, and if the acquired application time log corresponds to the restriction, displays a message on the terminal display device 203 requesting reselection of the application time. Application time information received from the terminal input device 202 is transmitted from the terminal communication device 201 to the server communication device 301. The control unit 326 of the server processing device 320 acquires necessary parameters from the application time information from the server storage device 302, commands texture object generation, and generates an image in the generation unit 323. The generated image is transmitted from the server communication device 301 to the terminal communication device 201 and displayed on the terminal display device 202.

[0135] If the processing performance on the user's terminal device 2 side cannot be ensured, the terminal processing device 220 forcibly terminates unrelated background applications, thereby securing processing resources and realizing stable performance.

[0136] When the server device 3 is offline, the terminal device 2 does not acquire the user's actions, and the server device 3 does not generate a simulation, but instead displays a CG image making a predetermined movement that has been defined in advance. If communication from the server device 3 is completely cut off in the terminal communication device 201, the terminal processing device 220 commands the terminal display device 203 to notify the user of a change to offline mode. Thereafter, the terminal processing device 220 commands the terminal display device 203 to call up a CG image stored in the terminal storage device 204 and display it on the terminal display device 203. Thereafter, the terminal display device 203 instructs the user to make a predetermined movement.

[0137] FIG. 7 is a flowchart showing an example of the overall flow of information processing executed by the server device 3.

[0138] 7, the flow of information processing executed by the server device 3 will be described below. Note that the operation flow described below is executed mainly by the server processing device 320 of the server device 3 in cooperation with each element of the server device 3, based on a program stored in advance in the server storage device 302 of the server device 3.

[0139] First, the control unit 326 determines an operation mode (step S1). Specifically, the control unit 326 transmits a predetermined operation mode selection screen to the terminal device 2 via the server communication device 301 to allow the user to select an operation mode. The terminal processing device 220 of the terminal device 2 displays the operation mode selection screen received via the terminal communication device 201 on the terminal display device 203. The terminal processing device 220 allows the user to select, via the terminal input device 202, an operation mode for operation in a virtual space of AR (Augmented Reality) / VR (Virtual Reality) via the operation mode selection screen displayed on the terminal display device 203. The terminal processing device 220 transmits the operation mode selected by the user to the server device 3 via the terminal communication device 201. The control unit 326 determines the operation mode received from the terminal device 2 via the server communication device 301 as the operation mode to be used in the server device 3 thereafter.

[0140] Examples of the operation mode include a slider operation mode, a touch panel operation mode, a moving object operation mode, a moving object image operation mode, etc. The operation mode selection screen is configured, for example, to display a plurality of selectable operation modes.

[0141] The slider operation mode is a mode that enables a coordinate specification operation using, for example, a slider arranged on a screen displayed on the terminal display device 203. The coordinate specification operation may be accepted via any input device. In the slider operation mode, the screen displayed on the terminal display device 203 includes, for example, a first area in which an image including a main object image is displayed, and a second area for accepting a coordinate specification operation by the user. In the second area, for example, an image used for the coordinate specification operation (for example, the slider described above) and an image for indicating the boundary of the second area are arranged. The first area and the second area may not overlap, or may overlap at least partially. In the slider operation mode, the moving object image may be a CG image. The slider operation mode may be an example of the first operation mode.

[0142] The touch panel operation mode is a mode that enables, for example, a coordinate specification operation using a finger moving on the touch panel of the terminal display device 203. As described above, the object used in the coordinate specification operation (which may be referred to as a moving object as described above) is not limited to a finger. In the touch panel operation mode, the moving object image may or may not be displayed on the screen. Even if the moving object image is not displayed on the screen in the touch panel operation mode, the visual effect between the moving object existing in real space and the main object image displayed on the screen may make the user feel as if they are perceiving the tactile sensation of the main object and / or supporting object. The touch panel operation mode may be an example of the second operation mode.

[0143] The moving object operation mode is a mode that enables, for example, a coordinate specification operation by gesture input. In this case, the user moves the moving object within the real space without touching the input device. The terminal input device 202 detects the movement of the moving object using, for example, the imaging device 205. This allows the user's coordinate specification operation to be accepted. The moving object operation mode may be an example of a third operation mode.

[0144] The moving object image operation mode, like the moving object operation mode, is a mode that enables coordinate specification operations by gesture input. In this case, too, the user moves the moving object within real space without touching the input device. The terminal input device 202 detects the movement of the moving object using, for example, the imaging device 205, and displays an image of the moving object on the terminal display device 203. This allows the display of a moving object image linked to the gesture input movement, and also allows the user's coordinate specification operation to be accepted. The moving object image operation mode may be an example of a fourth operation mode.

[0145] In the slider operation mode, the second acquisition unit 322 detects the coordinates on the screen of the slider displayed on the terminal display device 203 as position information of the moving object. In the touch panel operation mode, the second acquisition unit 322 detects the coordinates indicated by a finger pressing the touch panel on the terminal display device 203 as position information of the moving object. Note that the UI (User Interface) for operating the moving object is not limited to a slider, and may be a UI such as a virtual joystick. In the case of a virtual joystick, the amount and time of finger dragging is detected to determine the position information of the moving object.

[0146] In the moving object operation mode and the moving object image operation mode, the second acquisition unit 322 detects a moving object from an image of the moving object by using known image processing techniques such as pattern matching, inter-frame difference, computer vision technology, hand tracking technology, motion recognition, and posture estimation technology, and detects the coordinates of the detected moving object in the image as position information of the moving object. In this embodiment, the following describes an example in which the moving object image operation mode is selected, but the slider operation mode, touch panel operation mode, or moving object operation mode may also be selected.

[0147] Next, the control unit 326 executes a calibration process (step S2). The calibration process will be described in detail later.

[0148] Next, the control unit 326 selects a fluid (step S3). The control unit 326 transmits a predetermined fluid selection screen to the terminal device 2 via the server communication device 301 to allow the user to select a desired fluid. The terminal communication device 201 of the terminal device 2 receives data indicating the fluid selection screen and displays the fluid selection screen based on the received data on the terminal display device 203. The terminal processing device 220 transmits information indicating the fluid selected by the user via the fluid selection screen to the server device 3 via the terminal communication device 201. Note that the terminal processing device 220 may display the product brand name, product category, product price, skin type, past purchase history, preferred texture, skin care category, etc. on the fluid selection screen displayed on the terminal display device 203, and allow the user to select a desired fluid from among them. The control unit 326 selects the fluid indicated in the information received from the terminal device 2 via the server communication device 301.

[0149] Next, the control unit 326 determines whether or not the simulation mode has been selected (step S4). The simulation mode includes, for example, a basic experience mode and a partial experience mode.

[0150] The basic trial mode is a basic mode that allows users to try a series of simulations, including "discharging" the fluid selected by the user, "during application" where the fluid is actually applied to the skin and spread with the fingers, and "after application" where the fluid spreads and penetrates the skin, and shows changes in the skin's texture such as moistness, smoothness, friction, elasticity, softness, and stickiness, as well as changes in appearance such as skin color, gloss, and texture. The partial trial mode is a mode that allows users to try only some of the simulations of "discharging," "during application," and "after application."

[0151] The control unit 326 transmits a predetermined simulation mode selection screen to the terminal device 2 via the server communication device 301 to allow the user to select a simulation mode. The terminal processing device 220 of the terminal device 2 displays the simulation mode selection screen received via the terminal communication device 201 on the terminal display device 203, and allows the user to select the basic experience mode or the partial experience mode via the simulation mode selection screen. The terminal processing device 220 transmits a signal indicating the simulation mode selected by the user to the server device 3 via the terminal communication device 201. Furthermore, on the simulation mode selection screen, the user can additionally select options such as a skin type mode that identifies dry or oily skin, a skin texture experience mode that allows the user to try out the skin experience after long-term use of a fluid, a seasonal mode that reflects seasonal skin conditions, a multiple use mode for when multiple cosmetics are used in layers, etc.

[0152] If the user has not selected either the basic trial mode or the partial trial mode (step S4: NO), the server device 3 repeats the processes of steps S3 and S4. On the other hand, if the user has selected either the basic trial mode or the partial trial mode (step S4: YES), the control unit 326 executes a preparatory movement process (step S5). The details of the preparatory movement process will be described later.

[0153] Next, the control unit 326 executes a process for executing a simulation (hereinafter referred to as a "simulation process") (step S6). The simulation process will be described in detail later.

[0154] Next, the control unit 326 determines whether or not an end instruction has been received (step S7). The control unit 326 transmits a predetermined end screen to the terminal device 2 via the server communication device 301. The terminal processing device 220 of the terminal device 2 displays the end screen received via the terminal communication device 201 on the terminal display device 203, and transmits an end signal to the server device 3 when the user selects the end button on the end screen. If the control unit 326 does not receive an end signal from the terminal device 2 (step S7: NO), the control unit 326 repeats the processes of steps S3 to S7. On the other hand, if the control unit 326 receives an end signal from the terminal device 2 via the server communication device 301 (step S7: YES), the control unit 326 terminates the entire process.

[0155] In this embodiment, the details of the information processing in the information processing system 1 have been described using as an example a case where the information processing described in relation to Fig. 7 is executed in the server device 3. However, the information processing system 1 is not limited to this embodiment. In other embodiments, at least a part of the information processing in the server device 3 described in relation to Fig. 7 may be executed by another device (for example, the terminal device 2, the edge computer device 4, etc.).

[0156] (An example of another embodiment) In this embodiment, the details of the information processing system 1 have been described using an example in which the simulation mode includes a basic experience mode and a partial experience mode. However, the information processing system 1 is not limited to this embodiment. In other embodiments, the simulation mode may include a mode for searching for a texture that the user likes (sometimes referred to as a preference search mode). The preference search mode is used, for example, when the user has not yet decided on an object (e.g., a product) that the user wants to experience using, and the user searches for a product based on their own preference (e.g., the product's texture).

[0157] For example, if the above-mentioned product is a cosmetic product, the information processing system 1 executes information processing to realize the following service. First, the user determines the range for searching for products with the desired texture via a user interface presented by the information processing system 1. For example, the user specifies the dosage form and the texture of the skin after application, and the information processing system 1 determines the above-mentioned range. The dosage form is a cosmetic category such as lotion, emulsion, or cream.

[0158] Next, the user selects a trial mode via a user interface presented by the information processing system 1. Examples of trial modes include the basic trial mode and partial trial mode described above. Note that if the user specifies the texture of the skin after application, the selection of the trial mode may be omitted. Next, the user evaluates the textures presented by the information processing system 1 via the user interface presented by the information processing system 1. The information processing system identifies the parameters of the user's preferred texture based on the evaluation results.

[0159] An example of an evaluation method is paired comparison. The user experiences two CG images with different textures and selects their preferred one. Next, another texture different from the selected one is presented, and the user selects the preferred texture again. This process can be repeated multiple times to adjust each parameter used to select the preferred texture. For example, the parameters for which preferences are to be investigated, such as viscosity, penetration speed, spreadability, and transparency, are set. A single liquid is presented. After the presentation, the user is asked to respond to a questionnaire for each parameter, such as very much increased, increased, just right, decreased, or very decreased. The liquid is then presented with the parameters changed according to the instructions. The sample selected as just right remains unchanged. The process ends when all parameters are just right, or when the user selects to end. The parameters of the final CG image can then be linked to the user and saved as their preferred parameters. The similarity between these parameters and those of registered products is then calculated, and several products with high similarity are presented.

[0160] (Calibration process) As described above, in this embodiment, the control unit 326 executes a calibration process. The calibration process is a process for reducing the difference between the sensation the user gets when actually moving a moving object in the moving object image operation mode and the sensation the user gets from the movement of the moving object image displayed on the terminal display device 203.

[0161] (Calibration process overview) In this embodiment, the control unit 326 adjusts parameters used to reproduce or simulate deformation and / or movement of an object due to an external force. The parameters are adjusted, for example, by the following procedure.

[0162] The control unit 326 acquires one or more parameters related to a sample object, which is an object used for adjustment. The control unit 326 acquires position information of a moving object in real space. The control unit 326 generates (i) a first image showing the state of the sample object deforming or moving on the support object due to interaction between the sample object, the moving object, and the support object that supports the sample object, and / or (ii) a second image showing the state of the moving object moving on the support object that supports the sample object, based on the change in the position information and the one or more parameters.

[0163] The type of object adopted as the sample object is not particularly limited. In one embodiment, when tactile sensations are classified into multiple types and a representative object is defined for each tactile sensation type, the representative object may be adopted as the sample object. Representative objects for two or more types may be adopted as the sample object. In another embodiment, the control unit 326 may adopt as the sample object an object whose probability of being known to the user is greater than a predetermined value. The control unit 326 may derive the probability based on, for example, nationality, age or generation, gender, hobbies, product purchase history, and website browsing history.

[0164] The control unit 326 executes processing for presenting the first image and / or the second image to the user. Specifically, the control unit 326 outputs the first image and / or the second image to the terminal device 2 and causes these images to be displayed on the terminal device 2. The control unit 326 also executes processing for presenting to the user a screen for prompting the user, who has viewed the image (which, as described above, is a moving image generated to visually reproduce or simulate the tactile feel of an object), to input the impression he or she had of the tactile feel of the sample object or the supporting object.

[0165] The control unit 326 acquires feedback information including information indicating the impression that the user, who has been presented with the first image and / or the second image, has about the tactile feel of the sample object or the supporting object. For example, when the user inputs the impression into the terminal device 2, the terminal device 2 transmits the feedback information to the server device 3.

[0166] The control unit 326 adjusts at least one of the parameters used to reproduce or simulate the deformation and / or movement of the object due to the external force based on the feedback information. The parameters to be adjusted may include at least one of one or more parameters related to the sample object.

[0167] Examples of the process of adjusting a parameter include a process of changing the value of the parameter, a process of deriving a correction coefficient for correcting the value of the parameter, a process of deriving a correction value of the parameter using the correction coefficient, etc. In this embodiment, it should be noted that the parameter is adjusted even at a stage where the value of the parameter has not been changed or the correction value of the parameter has not been derived.

[0168] In one embodiment, the control unit 326 derives a correction coefficient for each parameter type based on the feedback information. In another embodiment, one or more parameters may be classified into a plurality of categories in advance. The control unit 326 derives a correction coefficient for each category based on the feedback information. In yet another embodiment, the tactile sensation may be classified into a plurality of types in advance. Each of the one or more parameters may be associated with one of the plurality of types in advance. A single parameter may also be associated with a plurality of types.

[0169] As described above, the control unit 326 receives position information of the moving object as a user input, and generates a first image in which the sample object is deformed or moved with a delay from the input, and / or generates a second image in which the moving object is moved with a delay from the input.

[0170] In the above process, the control unit 326 may determine the type of the sample object. The control unit 326 may acquire one or more parameters associated with the type of sample object by referring to a storage device that stores, for each of one or more types of objects, information indicating the type of the object and one or more parameters used to reproduce or simulate deformation and / or movement of the object due to an external force, in association with each other.

[0171] In the above process, the storage device may store, for each of one or more types of objects, information indicating the type of the object, parameter identification information for identifying each of one or more parameters related to the object, and value information indicating the values ​​of the parameters, in association with each other. In the above process, the control unit 326 may adjust the value of at least one parameter.

[0172] (Calibration process details) 8 is a flowchart showing an example of the calibration process. According to this embodiment, first, the control unit 326 performs initial settings (step S201). For example, initial setting items include position settings such as the orientation of the camera of the imaging device 205 of the terminal device 2, geometry settings indicating the shape of the moving object image, settings for the skin type of the target to which the liquid cosmetic will be applied, settings for volume adjustment, settings for screen brightness and color tone, etc.

[0173] Next, the generation unit 323 generates a first output image and transmits it to the terminal device 2 (step S202). The first output image shows a moving object. The generation unit 323 reads the moving object image from the server storage device 302, generates a first output image including the moving object image, and transmits it to the terminal device 2 via the server communication device 301. The terminal processing device 220 displays the first output image received via the terminal communication device 201 on the terminal display device 203.

[0174] Fig. 9A is a diagram showing a first output image displayed on a terminal device, and Fig. 9B is a diagram showing the movement of a moving object (finger) of a user.

[0175] 9A, the first output image G1 shows the user's right hand as a moving object image g11 (broken line). The generation unit 323 may use the user's right hand, which is actually captured by the imaging device 205 of the terminal device 2, as the moving object.

[0176] Next, as shown in FIG. 9B, the second acquisition unit 322 acquires a first movement amount D1 and a first movement direction of the moving object Ob1 (step S203). The first movement amount D1 and the first movement direction are the actual movement amount and movement direction of the moving object Ob1. The second acquisition unit 322 periodically acquires position information of the moving object Ob1 by the method shown in step S1 of FIG. 7. The second acquisition unit 322 acquires the movement amount and movement direction of the acquired position information of the moving object Ob1 within a certain period as the first movement amount D1 and the first movement direction.

[0177] Next, the calculation unit 325 calculates a second movement amount D2 of the moving object image g11 included in the first output image G1 (step S204). The calculation unit 325 calculates the second movement amount D2 of the moving object image g11 representing the moving object Ob1, which corresponds to the first movement amount D1, based on the movement parameter α using the following equation (1), where α is a movement parameter (coefficient) and satisfies 1>α>0. D2=α×D1………………………………………………………………………………(1)

[0178] For example, when the first movement amount D1 is 10 cm and the movement parameter α is 0.25, the second movement amount D2 is 2.5 cm.

[0179] The generation unit 323 sets the movement speed of the moving object image g11 in the first output image G1 so that the movement start timing and movement end timing of the moving object Ob1, which moves by the first movement amount D1, coincide with the movement start timing and movement end timing of the moving object image g11, which moves by the second movement amount D2 in the first output image G1. This allows the server device 3 to match the real feeling the user gets when moving the moving object Ob1 with the feeling the user gets visually when the moving object image g11 moves in the first output image G1. As a result, when the user visually observes the movement of the moving object image g11, the user can feel as if the movement of the moving object image g11 is linked to the movement of the moving object Ob1.

[0180] Next, the generation unit 323 generates (updates) a first output image G1 (step S205). The generation unit 323 generates the first output image G1 by moving the moving object image g11 in a second movement direction corresponding to the first movement direction by a second movement amount D2. The second movement direction is the same direction as the first movement direction.

[0181] Next, the generation unit 323 outputs the updated first output image G1 by transmitting it to the terminal device 2 via the server communication device 301 (step S206).

[0182] The terminal processing device 220 displays the first output image G1 received via the terminal communication device 201 on the terminal display device 203. The user visually checks the first output image G1 displayed on the terminal display device 203, and if the user feels that there is a difference (deviation) between the movement of the moving object Ob1 and the movement of the moving object image g11 in the first output image G1, the user can change the value of the movement parameter α. When the user specifies a new movement parameter α using the terminal input device 202, the terminal processing device 220 transmits the specified movement parameter α to the server device 3 via the terminal communication device 201.

[0183] Next, the reception unit 324 waits until an instruction is specified by the user (step S207). If an instruction is specified by the user, the reception unit 324 determines whether a new movement parameter α has been specified (step S208). If the reception unit 324 has not received the movement parameter α from the terminal device 2 via the server communication device 301, the reception unit 324 repeats the processes of steps S203 to S207 using the movement parameter α. On the other hand, if the reception unit 324 has received the movement parameter α from the terminal device 2 via the server communication device 301 (step S208: YES), the reception unit 324 sets the current (latest) value of the movement parameter α by storing it in the server storage device 302 (step S209).

[0184] Next, the generation unit 323 generates a second output image and transmits it to the terminal device 2 (step S210). The second output image shows a plurality of different sample fluids. Each sample fluid shown in the second output image is predetermined so that the user can select a sample fluid that resembles a predetermined object in the process described below. The predetermined object is a liquid or solid with a widely known texture, such as water or honey, or a substance that exhibits an intermediate state between liquid and solid. In the second output image, the generation unit 323 displays each sample fluid so as to represent the characteristic parameters of each sample fluid stored in the sample fluid table T2, particularly the color, gloss, mass size, viscoelasticity, etc. of each sample fluid.

[0185] For example, if the predetermined object is honey, the generation unit 323 identifies multiple types of sample fluids having characteristic parameters similar to the characteristic information of honey as sample fluids to be shown in the second output image. The generation unit 323 transmits the generated second output image to the terminal device 2 via the server communication device 301. The terminal processing device 220 receives the second output image via the terminal communication device 201 and displays it on the terminal display device 203.

[0186] FIG. 10 is a diagram showing the second output image that is initially displayed on the terminal device.

[0187] In the example shown in FIG. 10, the second output image G2 shows two types of sample fluid images g12 and g13.

[0188] The two sample fluid images g12 and g13 are displayed with different appearances depending on their respective characteristic parameters, such as the color, gloss, size of the lump, and viscoelasticity of the liquid. At least some of the characteristic parameters of the sample fluid images g12 and g13 are different. Therefore, the degree of deformation, the degree of movement, etc. of the sample fluid images g12 and g13 differs depending on the differences in the characteristic parameter items (ease of deformation, spreading speed, evaporation speed, surface tension, viscoelasticity, transparency, etc.) corresponding to the sample fluid images g12 and g13.

[0189] Next, the first acquisition unit 321 acquires a plurality of feature parameters (step S211). The first acquisition unit 321 acquires the feature parameters of each of the sample fluid images g12 and g13 included in the second output image G2 by reading them from the sample fluid table T2.

[0190] Next, the second acquisition unit 322 acquires the position information of the moving object (step S212). The second acquisition unit 322 acquires the position information of the moving object by the method shown in step S1 of FIG. 7. For example, the terminal processing device 220 transmits a captured image of the user's right hand (including fingers) as a moving object captured by the imaging device 205 to the server device 3 via the terminal communication device 201. The second acquisition unit 322 receives the captured image via the server communication device 301 and detects the position information of the moving object from the captured image. The second acquisition unit 322 calculates the amount of movement and the direction of movement of the acquired position information of the moving object over a certain period of time as a change in the position information.

[0191] Next, the generation unit 323 generates (updates) the second output image G2 (step S205). The generation unit 323 generates a second output image G2s by deforming or moving the sample fluid images g12 and g13 and / or moving the moving objects in accordance with changes in the feature parameters and position information of the moving objects corresponding to the sample fluid images g12 and g13 from the latest second output image G2.

[0192] If the current second output image G2 does not contain a moving object, the generation unit 323 places the moving object at a position in the second output image G2 that corresponds to the position information of the detected moving object. On the other hand, if the current second output image G2 contains a moving object, the generation unit 323 moves the moving object in the second output image G2 and deforms or moves the sample fluid images g12 and g13.

[0193] For example, the server device 3 stores a relational expression or a table showing the relationship between changes in the characteristic parameters of each sample fluid and the position information of the moving object and the degree of deformation or movement of each sample fluid in the server storage device 302. The generation unit 323 refers to the relational expression or the table stored in the server storage device 302, identifies the degree of deformation or movement corresponding to the changes in the characteristic parameters of the sample fluid and the position information of the moving object, and deforms or moves the sample fluid image in the first output image according to the identified degree.

[0194] Furthermore, the generation unit 323 may use a physics simulator to simulate the deformation or movement of an object based on the movement of the moving object. The physics simulator receives as input the movement of the moving object (e.g., speed, direction, acceleration, etc.) and the physical properties of the sample fluid (e.g., mass, hardness, shape, etc.), and calculates how the sample fluid will deform or move based on the input. Based on the calculation result, the generation unit 323 deforms or moves a sample fluid image of the sample fluid and generates the result as a second output image. That is, the generation unit 323 can use the physics simulator to generate a first image showing how an external force corresponding to a change in position information of the moving object is applied to the sample fluid, and output the second output image including the first image.

[0195] The server device 3 also stores a relational expression or table in the server storage device 302 that indicates the relationship between changes in the characteristic parameters of each sample fluid and the position information of the moving object, and the movement direction and movement amount of the moving object on each sample fluid. The generation unit 323 refers to the relational expression or table stored in the server storage device 302 and identifies the movement direction and movement amount corresponding to the changes in the characteristic parameters of the sample fluid and the position information of the moving object. The generation unit 323 moves the moving object image shown in the current second output image G2 according to the acquired movement amount and movement direction.

[0196] The generation unit 323 may also use a physics simulator to simulate the deformation or movement of the moving object based on the physical properties of the sample fluid. The physics simulator receives as input the movement of the moving object (e.g., velocity, direction, acceleration, etc.) and the physical properties of the sample fluid (e.g., mass, hardness, shape, etc.), and calculates how the moving object will deform or move based on the input. Based on the calculation result, the generation unit 323 deforms or moves the image of the moving object and generates the result as the second output image G2.

[0197] The generation unit 323 may deform or move the sample fluid image using a trained model that has been pre-trained to output a sample fluid image deformed or moved by a moving object when a sample fluid image representing the sample fluid, feature parameters of the sample fluid, and changes in the position information of the moving object are input. This trained model is pre-trained using a neural network, support vector machine, or the like, using a combination of the sample fluid image, changes in the feature parameters of the sample fluid, and the position information of the moving object, and the sample fluid image deformed or moved by the moving object as training data. The generation unit 323 inputs the sample fluid images g12 and g13 included in the current second output image G2, the feature parameters of the sample fluid images g12 and g13, and changes in the position information of the moving object into the trained model, and obtains an image output from the trained model. The generation unit 323 replaces the sample fluid images g12 and g13 included in the current second output image G2 with the obtained images p11 and p12.

[0198] Furthermore, the generation unit 323 may move the moving object using a trained model that has been pre-trained to output the movement amount and movement direction of the moving object moving on the sample fluid images g12 and g13 when changes in the feature parameters of the sample fluid images g12 and g13 and the position information of the moving object are input. This trained model is pre-trained using a neural network, a support vector machine, or the like, using as training data a combination of changes in the feature parameters and position information of the moving object of the sample fluid images g12 and g13 and the movement amount and movement direction of the moving object moving on the sample fluid images g12 and g13. The generation unit 323 inputs the changes in the feature parameters and position information of the moving object of the sample fluid images g12 and g13 included in the current second output image G2 into the trained model and obtains the movement amount and movement direction output from the trained model. The generation unit 323 moves the moving object shown in the current second output image G2 according to the acquired movement amount and movement direction.

[0199] Furthermore, the generation unit 323 may deform or move the supporting object using a trained model that has been pre-trained to output a supporting object image deformed or moved by the moving object when a supporting object image indicating the supporting object, characteristic information of the supporting object, and changes in the positional information of the moving object are input. This trained model is pre-trained using a neural network, a support vector machine, or the like, using as training data a combination of the supporting object image, the characteristic information of the supporting object, changes in the positional information of the moving object, and the supporting object image deformed or moved by the moving object. The generation unit 323 inputs the supporting object image included in the current simulation image, the characteristic information of the supporting object selected by the user, and changes in the positional information of the moving object into the trained model and obtains an image output from the trained model. The generation unit 323 replaces the supporting object image included in the current simulation image with the image obtained from the trained model.

[0200] In this way, the generation unit 323 generates, for each of the multiple feature parameters, a second output image showing a sample fluid that is deformed or moved by a moving object in accordance with changes in the position information of the moving object, or a moving object that moves on the sample fluid in accordance with changes in the position information of the moving object, based on each feature parameter.

[0201] Next, the generation unit 323 transmits the updated second output image G2s to the terminal device 2 via the server communication device 301 (step S214). The terminal processing device 220 displays the second output image G2s received via the terminal communication device 201 on the terminal display device 203.

[0202] FIG. 10 is a diagram showing the updated second output image G2s. As shown in FIG. 10, in the updated second output image G2s, each sample fluid image p11, p12 is deformed or moved. The updated second output image G2s also includes text TX requesting the user to select a sample fluid from each sample fluid image p11, p12 that has a texture similar to that of a predetermined object. The user can visually view the updated second output image G2s and select a sample fluid from each sample fluid image p11, p12 that feels similar to the predetermined object. The generation unit 323 may change the feature parameters of the two sample fluid images p11, p12 and repeat the process of updating the second output image G2s multiple times. For example, the generation unit 323 may first generate two sample fluid images p11 and p12 with different ease of deformation and allow the user to compare them, and then, secondly, generate two sample fluid images p11 and p12 that have the ease of deformation of the sample fluid selected in the first run but with different spreading speeds and allow the user to compare them.

[0203] Next, the receiving unit 324 determines whether or not a designation of a feature parameter has been received (step S207). When a user selects any sample fluid included in the second output image G2s updated using the terminal input device 202, the terminal processing device 220 of the terminal device 2 transmits a selection signal indicating the selected sample fluid to the server device 3 via the terminal communication device 201. The receiving unit 324 receives the selection signal via the server communication device 301. The receiving unit 324 identifies the sample fluid indicated in the received selection signal, and accepts the feature parameters of the identified sample fluid as a designation of feature parameters corresponding to a specific object designated by the user.

[0204] If the reception unit 324 has not received the designation of the characteristic parameter (step S207: NO), the reception unit 324 repeats the processes of steps S204 to S206.

[0205] On the other hand, when the receiving unit 324 receives the designation of the feature parameters (step S207: YES), it sets the fluid feature information based on the feature parameters (step S208). The receiving unit 324 sets the fluid feature information of a fluid having feature information similar to the feature information of a predetermined object based on the feature parameters of the sample fluid designated by the user. The fluid having feature information similar to the feature information of the predetermined object is set in advance by an administrator. A fluid having feature information whose similarity to the feature information of the predetermined object is equal to or greater than a threshold may be set as a fluid having feature information similar to the feature information of the predetermined object. The similarity may be normalized cross-correlation or cosine similarity, etc. Alternatively, the receiving unit 324 may set the feature parameters of the sample fluid designated by the user as the fluid feature information as is. Alternatively, the receiving unit 324 may set a value obtained by multiplying or dividing each parameter value of the feature parameters by a predetermined coefficient and / or adding or subtracting a predetermined offset to or from the parameter value of the feature parameters as the fluid feature information. The predetermined coefficient or predetermined offset may be set based on the degree of similarity between the characteristic information of the predetermined object and the characteristic information of the fluid for which the fluid characteristic information is set. In other words, the receiving unit 324 can set the fluid characteristic information so that the user's sense of the movement of the moving object and the sense of the degree of deformation or movement of the sample fluid are similar. Thereafter, when the user specifies the end of the calibration process using the terminal input device 202, the terminal processing device 220 transmits an instruction to end the calibration process to the server device 3 via the terminal communication device 201.

[0206] With the above, the reception unit 324 ends the calibration process. Note that the server device 3 may perform the calibration process by setting only either the movement parameter α (step S209) or the fluid characteristic information (step S216).

[0207] (Preliminary action processing) 12 is a flowchart showing the preparatory movement processing. As described above, in this embodiment, the control unit 326 executes the preparatory movement processing. The preparatory movement processing is a processing that encourages the user to create a sensation (illusion) by, for example, deforming or moving a main object image showing a predetermined main object displayed on the terminal display device 203 in accordance with the movement of a moving object by the user.

[0208] The preparatory movement processing may be processing for acclimatizing the user's eyes in advance so that even if there is a slight difference between the sensation the user gets from the movement of the moving object by the user and the sensation the user gets from the deformation or movement of the main object image after the calibration processing, the difference is no longer perceptible. The preparatory movement processing may be processing for practicing acclimatizing the user's eyes before actually performing a simulation processing in the AR / VR virtual space, or may be processing for improving the user's sense of immersion. The preparatory movement processing may be processing for preparing the user to be immersed in the experience.

[0209] (Outline of preliminary operation processing) In this embodiment, the control unit 326 generates an output image including an object image representing a predetermined object. The control unit 326 outputs the output image. The control unit 326 accepts a user's operation on the object image.

[0210] According to this embodiment, the control unit 326 executes processing for deforming or moving a predetermined object image within the output image based on an operation received by the control unit 326. The object may include the main object described above. The object may include the main object described above and at least one of the moving object and supporting object described above.

[0211] When executing the process for deforming or moving the above-mentioned object image, the control unit 326 is configured to be able to adjust at least one of (i) the degree of delay between the timing at which the control unit 326 receives a user operation and the timing at which the object image is deformed or moved within the output image, and (ii) the manner in which the appearance of the object is expressed.

[0212] According to this embodiment, the control unit 326 executes a process for presenting to the user a screen for prompting the user to perform a predetermined operation or a screen designed to have the user perform the predetermined operation. When executing the process for presenting the above screen to the user, the control unit 326 may change at least one of the degree of delay and the expression mode during at least a part of the period during which the control unit 326 is accepting the predetermined operation.

[0213] In the above process, if the user's operation matches a predetermined operation, the control unit 326 may further generate a simulation image showing a simulation according to the user's operation. The control unit 326 may output the simulation image.

[0214] In the above process, the predetermined operation may include at least one of the following: (i) an operation of continuing an input operation of the same or similar type or pattern for a period of at least a predetermined length, (ii) an operation of repeating an input operation of the same or similar type or pattern at least a predetermined number of times, and (iii) an operation of continuing an input operation for specifying a position or coordinates on the screen until the degree of change in the position or coordinates meets a predetermined condition. An example of the predetermined condition is a condition that the degree of change in the position or coordinates is equal to or greater than a predetermined degree.

[0215] During at least a portion of the period during which the above operation is accepted, at least one of the above-described degree of delay and the representation manner is changed, so that the user feels as if the object is deforming or moving as intended. Examples of the representation manner include the degree of realism, the degree of deformation, the degree of blurring, the resolution, and the degree of appearance similarity to the user's hand or finger.

[0216] As a result, in various experience modes, when a discrepancy occurs between the user's coordinate designation operation and the movement of the moving object, the user may feel as if they are perceiving a tactile sensation corresponding to the discrepancy. In particular, when the object includes a moving object, and the representation of the moving object becomes more realistic during the above-mentioned period, the user may begin to feel the moving object as if it were his or her own hand, as the degree of similarity in appearance between the moving object and the user's hand or fingers increases.

[0217] In one embodiment, the degree of delay at a first point in time included in a period during which the control unit 326 accepts a predetermined operation is adjusted to be smaller than the degree of delay at a second point in time included in the period. The first point in time is a point in time later than the second point in time.

[0218] In another embodiment, the representation mode at a first time point included in a period during which the control unit 326 accepts a predetermined operation is adjusted to be more realistic than the representation mode at a second time point included in the period. The first time point is a time point after the second time point. The degree of realism may increase as the degree of blurring of the image of the object decreases. The degree of realism may increase as the resolution of the image of the object increases. If the object is a moving object, the degree of realism may increase as the degree of external similarity between the object and the user's hand or finger increases.

[0219] (Calibration process details) First, the generation unit 323 reads out a main object image indicating a predetermined main object from the server storage device 302 (step S501). Next, the generation unit 323 generates a third output image including the read out main object image and outputs it by transmitting it to the terminal device 2 via the server communication device 301 (step S502). The third output image is an example of an output image. The terminal processing device 220 receives the third output image via the terminal communication device 201 and displays it on the terminal display device 203.

[0220] Next, the receiving unit 324 receives an input operation from the user (step S503). The second acquisition unit 322 receives position information of the moving object by the method shown in step S1 of FIG. 7. For example, the terminal processing device 220 transmits, to the server device 3 via the terminal communication device 201, a captured image of an operation of the moving object to deform or move the main object image included in the third output image, captured by the imaging device 205. The receiving unit 324 receives the captured image via the server communication device 301 and detects position information of the moving object from the captured image. The receiving unit 324 calculates the amount of movement and the direction of movement of the acquired position information of the moving object over a certain period as a change in position information. The receiving unit 324 receives the calculated direction and amount of movement as an input operation by the user for the object.

[0221] Examples of user input operations include a click operation (including a double-click), a drag operation, a flick operation, a swipe operation, a pinch-in operation, a pinch-out operation, etc. User input operations are not limited to the above operations, and any operation can be employed, such as an operation of depressing (pressing) a mochi object, an operation of rotating the cap on a plastic bottle object, a rotation operation, an operation of moving a slider, an operation of solving a maze, an operation of foaming liquid, an operation of opening a door, an operation of checking one's own skin condition, an operation of putting an item into a basket, an operation of grasping a container, an operation of opening a container and discharging the liquid contained inside the container, etc.

[0222] Next, the generation unit 323 generates (updates) the third output image by deforming or moving the main object image within the third output image based on the user's input operation, i.e., the movement direction and movement amount, received by the reception unit 324 (step S504).

[0223] The server device 3 stores a relational expression or a table indicating the relationship between the direction and amount of movement of the moving object and the amount of deformation or movement of the main object in the server storage device 302. The generation unit 323 refers to the relational expression or table stored in the server storage device 302 and specifies the amount of deformation or movement of the main object that corresponds to the direction and amount of movement of the moving object. The generation unit 323 deforms or moves the main object image in the third output image according to the specified amount of deformation or movement.

[0224] The generation unit 323 may deform or move the main object image using a trained model that has been trained in advance to output a main object image deformed or moved by a moving object when a main object image indicating the main object and a movement direction and movement amount of the moving object are input. This trained model is trained in advance using a neural network, a support vector machine, or the like, using as training data a combination of the main object image, the movement direction and movement amount of the moving object, and the main object image deformed or moved by the moving object. The generation unit 323 inputs the main object image, the movement direction and movement amount of the moving object included in the current third output image into the trained model and obtains an image output from the trained model. The generation unit 323 replaces the main object image included in the current third output image with the image obtained from the trained model.

[0225] The generation unit 323 transmits a third output image including the transformed or moved predetermined main object image to the terminal device 2 via the server communication device 301. The terminal processing device 220 displays the third output image received via the terminal communication device 201 on the terminal display device 203.

[0226] 13A and 13B are diagrams showing an example of a third output image in the preparatory movement processing, and an example of an updated third output image in the preparatory movement processing.

[0227] 13A and 13B, the main object image g14 is a rice cake, and the moving object Obg1 is a finger. As shown in Fig. 13A and 13B, when the user moves his / her finger up and down, a depression is formed in the main object image g14, g14s in the third output images P3, P3s.

[0228] By visually checking the third output images P3 and P3s, the user can visually experience the change in the main object images g14 and g14s. This allows the server device 3 to give the user the sensation (illusion) of changing the main object in accordance with the operation of the moving object. As a result, the user's eyes can become accustomed to the illusion, and the user can become accustomed to imagining tactile sensations from the visual information of the moving object and the main object.

[0229] FIG. 14 is a diagram showing another example of the third output image in the preparatory movement processing.

[0230] 14, the main object image g15 is a cap, the main object image g16 is a plastic bottle body, and the moving object Obg1 is a finger. When the user rotates his / her finger, the main object image g15 of the cap rotates relative to the main object image g16 of the plastic bottle body in the third output image P4.

[0231] By visually checking the third output image P4, the user can visually experience the change in the main object image g15. This allows the server device 3 to give the user the sensation (illusion) of changing the main object in accordance with the operation of the moving object. As a result, the user's eyes can become accustomed to the illusion, and the user can become accustomed to imagining tactile sensations from the visual information of the moving object and the main object.

[0232] Next, the control unit 326 determines whether the user's input operation satisfies a predetermined condition (step S505).

[0233] For example, if the main object is a rice cake, the predetermined condition is that the moving object moves up and down by a predetermined amount or more, a predetermined number of times or more, or a predetermined time or more. Also, if the main object is a plastic bottle and a cap, the predetermined condition is that the cap separates from the plastic bottle, or the cap joins with the plastic bottle, or the amount of movement related to the rotation of the moving object is a predetermined amount or more.

[0234] If the user's input operation does not satisfy the predetermined condition (step S505: NO), the control unit 326 repeats the processes of steps S503 and S504.

[0235] On the other hand, if the user's input operation satisfies a predetermined condition (step S505: YES), the control unit 326 ends the preparatory movement process.

[0236] The generation unit 323 may generate a third output image including a main object image that resembles, for example, a dial or knob as the main object, and a moving object image that resembles the user's finger or cursor as the moving object Obg1. In this case, when the user performs a rotation operation by rotating his or her finger or cursor, the main object image is displayed in the third output image performing a predetermined rotational movement. This rotation operation corresponds to, for example, turning a volume control dial. By visually checking the third output image, the user can visually experience the movement of the main object image. In this way, the server device 3 gives the user the sensation (illusion) of moving the main object in accordance with the operation of the moving object, and allows the user's eyes to become accustomed to the illusion.

[0237] The generation unit 323 may also generate a third output image including a main object image that simulates, for example, a slider or bar as the main object, and a moving object image that simulates the user's finger or cursor as the moving object Obg1. In this case, when the user performs a slide operation by sliding their finger or cursor up and down or left and right, the main object image is displayed in the third output image performing a predetermined slider movement. This slide operation corresponds to, for example, moving a brightness adjustment slider. By visually checking the third output image, the user can visually experience the movement of the main object image. In this way, the server device 3 gives the user the sensation (illusion) that the main object is moving in accordance with the operation of the moving object, allowing the user's eyes to become accustomed to the illusion.

[0238] The generation unit 323 may also generate a third output image including, for example, a main object image in which a maze is the main object, and a moving object image in which the user's finger or cursor is the moving object Obg1. In this case, when the user operates his / her finger or cursor to move within the maze, the moving object image moves within the main object image in the third output image, displaying the maze being solved. The operation for solving this maze corresponds to the operation performed in an actual maze game. By visually checking the third output image, the user can visually experience the change in the main object image. In this way, the server device 3 gives the user the sensation (illusion) of changing the main object in accordance with the operation of the moving object, allowing the user's eyes to become accustomed to the illusion.

[0239] Furthermore, the generation unit 323 may generate a third output image including, for example, a main object image that resembles honey as the main object, and a moving object image that resembles the user's finger as the moving object Obg1. In this case, when the user performs a movement operation of touching the honey with his / her finger and moving it left or right, the third output image displays the main object image of honey changing in shape. By visually checking the third output image, the user can visually experience the change in the main object image. In this way, the server device 3 gives the user the sensation (illusion) of changing the main object in accordance with the operation of the moving object, and allows the user's eyes to become accustomed to the illusion.

[0240] Furthermore, the generation unit 323 may generate a third output image including, for example, a main object image that resembles a facial cleanser as the main object, and a moving object image that resembles the user's hand as the moving object Obg1. In this case, when the user moves their hand in a circular motion, the main object image of the facial cleanser is displayed foaming in the third output image. By visually checking the third output image, the user can visually experience the change in the main object image. In this way, the server device 3 gives the user the sensation (illusion) of changing the main object in accordance with the operation of the moving object, allowing the user's eyes to become accustomed to the illusion.

[0241] Furthermore, the generation unit 323 may generate a third output image including, for example, a main object image that represents a product list as a main object, and a moving object image that represents the user's finger as a moving object Obg1. In this case, when the user performs a slide operation by sliding their finger up and down, the main object image of the product list is displayed scrolling within the third output image. By visually checking the third output image, the user can visually experience the change in the main object image and simultaneously select a product and perform a preparatory movement.

[0242] Furthermore, the generation unit 323 may generate a third output image including, for example, a first main object image that resembles a predetermined product as the main object, a second main object image that resembles a shopping cart as the main object, and a moving object image that resembles the user's finger as the moving object Obg1. In this case, when the user performs a drag-and-drop operation by dragging the first main object image of the product with his / her finger and dropping it on the second main object image of the shopping cart, the third output image displays the first main object image of the product moving to the second main object image of the shopping cart. By visually checking the third output image, the user can visually experience the changes in the first and second main object images and can simultaneously select a product and perform a preparatory movement.

[0243] FIG. 15 is a flowchart showing the simulation process.

[0244] The simulation process is a process for providing a user who wants to know how a fluid spreads when applied to the skin with a finger with a visual representation of the deformation and movement of the fluid in a virtual space, allowing the user to have a simulated experience. In other words, the simulation process is a process for allowing the user to have a simulated experience, through the display screen of the terminal display device 203, of how the fluid will deform or move as it penetrates the skin, how the fingers will move, and what kind of tactile sensation will be produced, when the user applies a selected fluid to the skin of their left arm and then tries to spread it by moving the fingers of their right hand.

[0245] First, the first acquisition unit 321 acquires the simulation mode information selected in step S4 of Fig. 7. The first acquisition unit 321 identifies whether the simulation mode is the basic experience mode or the partial experience mode based on the simulation mode information.

[0246] Next, the first acquisition unit 321 acquires fluid characteristic information related to the fluid selected in step S3 of Fig. 7 (step S601). The first acquisition unit 321 acquires fluid characteristic information related to the fluid that was set in advance in the server storage device 302 in step S208 of the calibration process or fluid characteristic information related to the fluid that was pre-stored in the server storage device 302 as a default, by reading it from the server storage device 302 in order to execute the simulation process.

[0247] Next, the first acquisition unit 321 acquires support object characteristic information relating to the physical properties of the support object by reading it from the server storage device 302 (step S602). However, the first acquisition unit 321 does not need to acquire the support object characteristic information if the fluid is not affected by the support object when it is applied, such as when only fluid discharge is selected in the partial experience mode.

[0248] Next, the first acquisition unit 321 acquires the movement parameter α set in step S215 of the calibration process by reading it from the server storage device 302 (step S603).

[0249] Next, the generation unit 323 generates a simulation image and transmits it to the terminal device 2 (step S604). The control unit 326 reads out the supporting object image from the server storage device 302, generates a simulation image in which the read supporting object image is arranged at a predetermined position, and transmits it to the terminal device 2 via the server communication device 301. The terminal processing device 220 displays the simulation image received via the terminal communication device 201 on the terminal display device 203.

[0250] Fig. 16A is a diagram showing a simulation image initially displayed on a terminal device, and Fig. 16B is a diagram showing a simulation image including a fluid.

[0251] In the example shown in FIG. 16A, the simulation image S11 displayed first displays a support object image g21 of the left arm to which the fluid is to be applied.

[0252] Next, the generation unit 323 updates the simulation image by synthesizing the main object image of the fluid with the supporting object image g21, and transmits the updated simulation image to the terminal device 2 (step S605). The generation unit 323 identifies, in the fluid table T1, fluid feature information corresponding to the fluid selected by the user.

[0253] As shown in Figure 16B, the generation unit 323 generates an updated simulation image S12 by superimposing the main object image g22 of the fluid on top of the support object image g21 in the simulation image S11 to represent the identified fluid characteristic information, particularly the corresponding color, gloss, lump size, viscoelasticity, etc.

[0254] The generation unit 323 transmits the updated simulation image S12 to the terminal device 2 via the server communication device 301. The terminal processing device 220 receives the updated simulation image S12 via the terminal communication device 201 and displays it on the terminal display device 203. Note that if the user has selected the partial experience mode as the simulation mode, the processing of step S605 may be omitted.

[0255] In the example shown in FIG. 16B, the updated simulation image S12 shows a state in which a main object image g22 of fluid is ejected onto a support object image g21.

[0256] Next, the second acquisition unit 322 acquires the position information of the moving object (step S606). The second acquisition unit 322 acquires the position information of the moving object by the method shown in step S1 of FIG. 7. For example, the terminal processing device 220 transmits a captured image of the moving object captured by the imaging device 205 to the server device 3 via the terminal communication device 201. The second acquisition unit 322 receives the captured image via the server communication device 301 and detects the position information of the moving object from the captured image. The second acquisition unit 322 calculates the amount of movement and the direction of movement of the acquired position information of the moving object over a certain period of time as a change in the position information.

[0257] Next, the generation unit 323 generates (updates) a simulation image based on the change in the position information of the moving object, and outputs the generated simulation image by transmitting it to the terminal device 2 via the server communication device 301 (step S607).

[0258] When the current simulation image S12 does not contain a moving object image, the generation unit 323 places the moving object image Obg1 at a position in the simulation image S12 that corresponds to the position information of the detected moving object. On the other hand, when the current simulation image S12 contains the moving object image Obg1, the generation unit 323 moves the moving object image Obg1 within the simulation image S12 and also deforms or moves the main object image g22 of the fluid.

[0259] For example, the server device 3 stores a relational expression or a table indicating the relationship between changes in fluid feature information, supporting object feature information, and position information of each fluid, and the degree of deformation or movement of each fluid, in the server storage device 302. The generation unit 323 refers to the relational expression or table stored in the server storage device 302 and specifies the degree of deformation or movement corresponding to changes in the fluid feature information, supporting object feature information, and position information of the moving object of the fluid selected by the user. The generation unit 323 deforms or moves the main object image representing the fluid in the simulation image S12 according to the specified degree.

[0260] The server device 3 also stores in the server storage device 302 a relational expression or table indicating the relationship between the fluid feature information, supporting object feature information, and changes in the position information of the moving object and / or the movement parameter α of each fluid, and the movement direction and movement amount of the moving object on each fluid. The generation unit 323 refers to the relational expression or table stored in the server storage device 302 and identifies the movement direction and movement amount of the moving object image corresponding to the fluid feature information, supporting object feature information, changes in the position information of the moving object and / or the movement parameter α of the fluid selected by the user. The generation unit 323 moves the moving object image shown in the current simulation image according to the acquired movement amount and movement direction.

[0261] The generation unit 323 may deform or move the fluid using a trained model that has been pre-trained to output a main object image representing the fluid deformed or moved by the moving object when a main object image representing the fluid, fluid characteristic information of the fluid, support object characteristic information, and changes in the position information of the moving object are input. This trained model is pre-trained using a neural network, support vector machine, or the like, using as training data a combination of the main object image representing the fluid, fluid characteristic information of the fluid, support object characteristic information, changes in the position information of the moving object, and the main object image representing the fluid deformed or moved by the moving object. The generation unit 323 inputs the main object image of the fluid included in the current simulation image, the fluid characteristic information of the fluid selected by the user, support object characteristic information, and changes in the position information of the moving object into the trained model and obtains an image output from the trained model. The generation unit 323 replaces the main object image of the fluid included in the current simulation image with the image obtained from the trained model.

[0262] The generation unit 323 may also move the moving object using a trained model that has been pre-trained to output the amount of movement and direction of movement of the moving object moving on the main object image representing the fluid when a main object image representing the fluid, fluid characteristic information of the fluid, support object characteristic information, a change in the position information of the moving object, and / or a movement parameter α are input. This trained model is pre-trained using a neural network, a support vector machine, or the like, using as training data a combination of the main object image representing the fluid, fluid characteristic information of the fluid, support object characteristic information, a change in the position information of the moving object, and / or the movement parameter α, and the amount of movement and direction of movement of the moving object image moving on the main object image representing the fluid. The generation unit 323 inputs the main object image of the fluid included in the current simulation image, the fluid characteristic information of the fluid selected by the user, support object characteristic information, a change in the position information of the moving object, and / or the movement parameter α into the trained model, and obtains the amount of movement and direction of movement of the moving object image output from the trained model. The generation unit 323 moves the moving object image shown in the current simulation image according to the obtained amount of movement and direction of movement.

[0263] Furthermore, the generation unit 323 may deform or move the support object image using a trained model that has been pre-trained to output a support object image deformed or moved by a moving object when a support object image indicating the support object, characteristic information of the support object, and changes in the position information of the moving object are input. This trained model is pre-trained using a neural network, a support vector machine, or the like, using as training data a combination of the support object image, characteristic information of the support object, changes in the position information of the moving object, and the support object image deformed or moved by the moving object. The generation unit 323 inputs the support object image included in the current simulation image, characteristic information of the support object selected by the user, and changes in the position information of the moving object into the trained model and obtains an image output from the trained model. The generation unit 323 replaces the support object image shown in the current simulation image with the image obtained from the trained model.

[0264] In this way, based on the fluid characteristic information selected by the user, the generation unit 323 synthesizes a first image showing a fluid that is deformed or moved by a moving object on a support object in accordance with changes in the position information of the position object, and a second image showing a moving object moving on a support object that supports the fluid, to generate a simulation image as a third image.

[0265] The control unit 326 transmits the updated simulation image to the terminal device 2 via the server communication device 301. The terminal device 2 displays the updated simulation image received via the terminal communication device 201 on the terminal display device 203.

[0266] FIG. 16C is a diagram showing a simulation image updated based on changes in the position information of the moving object.

[0267] As shown in FIG. 16C, in the simulation image S13, the fluid object image g22a is deformed while being painted out in accordance with the movement of the moving object Obg1.

[0268] In this way, the generation unit 323 generates a simulation image S13 showing the simulation result obtained by superimposing the moving object image Obg1 on the fluid object image g22a based on the fluid characteristic information received by the reception unit 324 during the calibration process, and outputs the generated image to the terminal device 2.

[0269] Next, the control unit 326 determines whether or not an end instruction has been received (step S608). The control unit 326 transmits a predetermined end screen to the terminal device 2 via the server communication device 301. The terminal processing device 220 of the terminal device 2 displays the end screen received via the terminal communication device 201 on the terminal display device 203, and transmits an end signal to the server device 3 when the user selects the end button on the end screen. If the control unit 326 does not receive an end signal from the terminal device 2 (step S608: NO), the control unit 326 repeats the processes of steps S606 to S608. On the other hand, if the control unit 326 receives an end signal from the terminal device 2 via the server communication device 301 (step S608: YES), the control unit 326 terminates the simulation process.

[0270] FIG. 16D is a diagram showing a simulation image after the movement of the moving object has been repeatedly executed.

[0271] 16D, in the simulation image S14, the fluid object image g22b is further spread and penetrates into the support object image g21 in accordance with the movement of the moving object image Obg1. By visually checking the simulation image S12, the user can visually and intuitively experience how the fluid object image g22a is spread on the support object image g21 and then penetrates into the skin. Furthermore, by visually checking the simulation image S14, the user can visually and intuitively experience changes in the skin's texture, such as its moistness, smoothness, friction, elasticity, softness, and adhesiveness, as well as in its appearance, such as its color, gloss, and texture.

[0272] In this way, by visually checking the simulation image S14, the user can intuitively experience the process of how the fluid object image g22b is spread on the skin and penetrates, as if they were actually experiencing it. The server device 3 can provide a simulated experience that makes the user feel as if they were actually experiencing it, thereby improving the user's convenience in the simulation.

[0273] FIG. 17 is a flowchart illustrating an example of the setting process executed by the server device.

[0274] 17, an example of the setting process executed by the server device 3 will be described below. Note that the operation flow described below is executed mainly by the server processing device 320 of the server device 3 in cooperation with each element of the server device 3, based on a program stored in advance in the server storage device 302 of the server device 3.

[0275] The setting process is a process for deriving fluid object characteristic information. The server device 3 executes the setting process before executing the overall process. The server device 3 may execute the setting process at any timing.

[0276] The control unit 326 identifies the fluid for which the user wishes to derive fluid characteristic information (step S11). The control unit 326 transmits a predetermined fluid selection screen to the terminal device 2 via the server communication device 301. The terminal processing device 220 displays the fluid selection screen received via the terminal communication device 201 on the terminal display device 203. The terminal processing device 220 transmits information indicating the fluid selected by the user on the fluid selection screen to the server device 3 via the terminal communication device 201. The control unit 326 identifies the fluid selected by the user based on the information received via the server communication device 301.

[0277] Next, the control unit 326 acquires captured images of the fluid (step S12). The terminal processing device 220 of the terminal device 2 transmits captured images of the fluid actually to be applied onto the support, captured by the imaging device 205, and captured images of the fluid after application, to the server device 3 via the terminal communication device 201. The control unit 326 receives the captured images of the fluid via the server communication device 301 and stores them in the server storage device 302.

[0278] Next, the control unit 326 determines whether the user's input operation satisfies a predetermined condition (step S13). For example, the predetermined condition may be that the operation of spreading the fluid applied to the user's arm with a finger has been performed a predetermined amount or more, that a predetermined time has elapsed, or that an instruction to end has been received from the user. For example, the control unit 326 detects position information of moving objects from the captured image using the method shown in step S1. When the total amount of movement of the acquired position information of the moving objects is equal to or greater than a predetermined amount, the control unit 326 determines that the operation of spreading the fluid applied to the user's arm with a finger has been performed a predetermined amount or more.

[0279] Next, if the user's input operation does not satisfy a predetermined condition (step S13: NO), the control unit 326 repeats the processes of steps S12 to S13. On the other hand, if the predetermined condition is satisfied (step S3: YES), the control unit 326 derives the value of each item of the fluid characteristic information (step S14). The server device 3 stores in the server storage device 302 a relational expression or a table indicating the relationship between the amount of change of the fluid and the speed of the moving object and the values ​​of each item of the fluid characteristic information (ease of deformation, spreading speed, evaporation speed, etc., as well as the finger movement speed, degree of change in the skin, time required for change, etc.). The control unit 326 detects the fluid and the moving object from each acquired captured image using the above-mentioned well-known image processing technology, and calculates the amount of change of the detected fluid and the speed of the moving object. The control unit 326 refers to the relational expression or the table stored in the server storage device 302 and identifies the value of each item of the fluid characteristic information corresponding to the calculated amount of change of the fluid and the speed of the moving object.

[0280] When a plurality of consecutive captured images of a fluid and a moving object are input, the control unit 326 may identify the value of each item of the fluid characteristic information using a trained model that has been pre-trained to output the value of each item of the fluid characteristic information of the fluid contained in the captured images. This trained model is pre-trained using a neural network, a support vector machine, or the like, using combinations of a plurality of consecutive captured images of a fluid and a moving object and the values ​​of each item of the fluid characteristic information of the fluid as training data. The control unit 326 inputs each acquired captured image into the trained model and identifies the information output from the trained model as the value of each item of the fluid characteristic information of the fluid contained in the captured images.

[0281] The control unit 326 may derive more accurate fluid characteristic information using a physical simulation. In the physical simulation, the physical behavior of the fluid is modeled based on the fluid characteristic information acquired from the captured image, the characteristic information of the moving object, and the characteristic information of the supporting object, and parameters for reproducing that behavior are inversely estimated. This inverse estimation makes it possible to more accurately identify the physical properties of the fluid, such as viscosity, density, and surface tension.

[0282] The control unit 326 uses a physics simulation engine implemented in the server processing device 320 to perform inverse estimation processing based on the physics simulation. This physics simulation engine includes an optimization algorithm for numerically analyzing the behavior of the fluid and finding optimal parameters between the actual captured image and the simulation results. The control unit 326 uses the optimized parameters to update the values ​​of each item of the fluid characteristic information to reflect more realistic characteristics. Finally, the control unit 326 stores the updated fluid characteristic information, including the parameters obtained by the physics simulation, in the server storage device 302, and ends the setting process in step S15. This allows the behavior of the fluid actually experienced by the user to be more accurately predicted based on the fluid characteristic information and utilized in simulations and analyses.

[0283] The control unit 326 can also estimate the values ​​of each item of the fluid characteristic information using AI (artificial intelligence). Specifically, the AI ​​uses a trained model to estimate the fluid characteristic information of a fluid from a captured image. This trained model is pre-trained using AI techniques such as deep learning and reinforcement learning, using a combination of multiple consecutive captured images of a fluid and a moving object and the values ​​of each item of the fluid characteristic information of that fluid as training data. The control unit 326 inputs each captured image into the trained model and identifies the information output from the trained model as the values ​​of each item of the fluid characteristic information of the fluid contained in the captured image. This enables more accurate estimation of fluid characteristic information. Through these processes, the control unit 326 can derive more accurate fluid characteristic information and identify parameters by integrating the captured images, physical simulations, and text information from sensory evaluations.

[0284] Next, the control unit 326 stores the values ​​of each item of the derived fluid characteristic information in the server storage device 302, thereby setting them as fluid characteristic information (step S15), and ends the setting process.

[0285] The control unit 326 may identify the value of each item of the fluid characteristic information based on the liquid physical properties of the fluid (surface tension, rheological properties (viscoelasticity), tribology (friction, vibration), appearance (color, transparency), and evaporation rate). The control unit 326 stores, in the server storage device 302, a relational expression or a table indicating the relationship between the liquid physical properties of the fluid and the value of each item of the fluid characteristic information. The control unit 326 receives a designation of the liquid physical properties of the fluid from a user. The control unit 326 refers to the relational expression or the table stored in the server storage device 302 and identifies the value of each item of the fluid characteristic information corresponding to the liquid physical properties received from the user. The control unit 326 may identify the value of each item of the fluid characteristic information using a fluid characteristic information trained model that has been trained in advance to output the value of each item of the fluid characteristic information of the fluid when the liquid physical properties of the fluid are input. This trained model is trained in advance using a neural network, a support vector machine, or the like, using combinations of the liquid physical properties of the fluid and the values ​​of each item of the fluid characteristic information of the fluid as training data. The control unit 326 inputs the liquid properties received from the user into the trained model, and identifies the information output from the trained model as the values ​​of each item of the fluid characteristic information of the fluid.

[0286] The control unit 326 may also derive values ​​for each item of the fluid characteristic information based on the fluid prescription. Examples of the prescription include at least one of the total amount of water, the type of surfactant, the high molecular weight, the amount of moisturizer, the amount of oil, and the amount of powder. The control unit 326, for example, stores a relational expression or a table in the server storage device 302 indicating the relationship between the fluid prescription and the value of each item of the fluid characteristic information. The control unit 326 receives a designation of the fluid prescription from a user. The control unit 326 references the relational expression or table stored in the server storage device 302 and identifies the value of each item of the fluid characteristic information corresponding to the fluid prescription received from the user. Note that the generation unit 323 may identify the value of each item of the fluid characteristic information using a trained model that has been pre-trained to output the value of each item of the fluid characteristic information when the fluid prescription is input. This trained model is pre-trained using a neural network, a support vector machine, or the like, using a combination of the fluid prescription and the value of each item of the fluid characteristic information of the fluid as training data. The control unit 326 inputs the prescription received from the user into the trained model and identifies the information output from the trained model as the values ​​of each item of the fluid characteristic information of that fluid.

[0287] The control unit 326 may identify various characteristic parameters based on a sensory evaluation of the fluid. Examples of sensory evaluation include moist, smooth, silky, chewy, fresh, rich, and moisturizing. The control unit 326 stores a relational expression or a table in the server storage device 302 that indicates the relationship between the sensory evaluation of the fluid and the value of each item of the fluid characteristic information. The control unit 326 receives a fluid prescription from a user. The control unit 326 references the relational expression or table stored in the server storage device 302 and identifies the value of each item of the fluid characteristic information corresponding to the sensory evaluation received from the user. Note that, when a sensory evaluation of a fluid is input, the generation unit 323 may identify the value of each item of the fluid characteristic information using a trained model that has been pre-trained to output the value of each item of the fluid characteristic information of that fluid. This trained model is pre-trained using a neural network, a support vector machine, or the like, using a combination of the sensory evaluation of the fluid and the value of each item of the fluid characteristic information of that fluid as training data. The control unit 326 inputs the sensory evaluation received from the user into the trained model, and identifies the information output from the trained model as the value of each item of the fluid characteristic information of the fluid.

[0288] <Other embodiments> It should be understood that those skilled in the art can make various changes, substitutions, and alterations to the present invention without departing from the spirit and scope of the present invention. The above-described embodiments and modifications may be implemented in any suitable combination within the scope of the present invention.

[0289] In the above embodiment, the case has been described in which the server processing device 320 of the server device 3 executes all of the calibration processing, preparatory movement processing, and simulation processing, but the present invention is not limited to this, and all or part of these processes may be executed by the terminal processing device 220 of the terminal device 2 or the edge processing device 420 of the edge computer device 4. Furthermore, the server device 3 and the edge computer device 4 may execute the calibration processing, preparatory movement processing, and simulation processing in a distributed manner. Similarly, the terminal device 2 and the edge computer device 4 may execute the calibration processing, preparatory movement processing, and simulation processing in a distributed manner. Similarly, the server device 3 and the terminal device 2 may execute the calibration processing, preparatory movement processing, and simulation processing in a distributed manner.

[0290] In the above embodiment, the server processing device 320 of the server device 3 executes the calibration process, the preparatory operation process, the simulation process, and the setting process. However, the server processing device 320 may execute only one of the calibration process, the preparatory operation process, the simulation process, and the setting process, and omit the other processes.

[0291] In the above embodiment, the server processing device 320 performs the basic trial mode in the simulation process, but the server processing device 320 may perform only a part of the simulation process of discharging, applying, and after applying the fluid in a partial trial mode.

[0292] In the above embodiment, an example was described in which the information processing system 1 is applied to an AR (Augmented Reality) / VR (Virtual Reality) virtual space, but the information processing system 1 may also be applied to an XR (Extended Reality / Cross Reality) virtual space.

[0293] (Guide function) In this embodiment, the details of the information processing system 1 have been described using as an example a case where the information processing system 1 is provided with a preliminary operation mode in addition to a normal experience mode, thereby improving the user's sense of immersion. However, the information processing system 1 is not limited to this embodiment. In other embodiments, the information processing system 1 may further include a function for guiding the user's operation.

[0294] The guide function may function in the normal experience mode, the preliminary operation mode, or the calibration mode, thereby suppressing unexpected actions by the user and allowing the user to maintain a high level of immersion.

[0295] In this embodiment, the information processing system 1 provides the user with information (sometimes referred to as guide information) regarding the operation the user should perform through visual, auditory, and / or tactile feedback. For example, if the speed or acceleration of the user's input operation is very high, it may be possible that the way the main object is transformed by the input operation is not properly represented. By presenting the guide information to the user, it is expected that the user will perform the input operation at an appropriate speed or acceleration.

[0296] Furthermore, if the relative positions of the main object and the moving object are not appropriate, it is conceivable that the way the main object is transformed by the user's input operation will not be properly represented.Similarly, if the relative positions of the support object and the main object and / or the moving object are not appropriate, it is conceivable that an appropriate image will not be generated.

[0297] Therefore, the position of the user's movement may be guided. For example, if the moving object is the user's own finger, when the finger is about to move out of the camera's field of view, the movement position of the finger may be guided so that the finger will fit within the camera's field of view. Similarly, if the support object is the user's own arm, when the arm is about to move out of the camera's field of view, the movement position of the arm may be guided so that the arm will fit within the camera's field of view.

[0298] An example of visual feedback is a process for displaying information indicating the content, trajectory, or goal of an action to be performed by the user on the display device of the terminal device 2. The information indicating the content, trajectory, or goal of an action to be performed by the user may be text, a marker, a diagram, a photograph, an animation, or a video. The content of the action may include the type of action.

[0299] Examples of auditory feedback include processing for outputting information indicating the content or goal of the action that the user should perform to the audio output device of the terminal device 2. Specifically, examples include processing for playing a predetermined sound when it is determined that the user has performed the correct action, processing for playing a sound at the timing when the user performs the action, and processing for playing a sound at a constant rhythm.

[0300] An example of a tactile guide is a process for outputting information indicating the content or goal of an action to be performed by the user to a vibration device of the terminal device 2 or a wearable device worn by the user. Examples of the content include the type of movement of a moving object, the speed or acceleration of the action, and the rhythm of the action. Examples of the goal include a setting value related to the content. The action may be a tactile action. When a unit action is repeated multiple times, the speed may be the average speed of the unit action, and the rhythm may be the cycle at which the unit action is repeated.

[0301] If the moving object is the user's own finger, examples of ways to move the moving object include moving the finger back and forth, drawing a circle with the finger, lightly pressing the main object or supporting object with the finger, and pulling up the main object or supporting object with the finger.

[0302] (Practice mode with guide function) The guide function may be used to help the user practice a service. For example, by enabling the guide function in the calibration mode or the preliminary operation mode, the user can become familiar with the operations of the service.

[0303] In the practice mode, for example, it is confirmed whether the range in which the user moves their finger is in the center of the camera's angle of view. For example, the information processing system 1 indicates the touch position using an arrow or text display so that the range of finger movement is in the center of the camera's angle of view. The distance from the camera is estimated from the displayed size of the finger, and instructions are given to keep the distance appropriate.

[0304] In the practice mode, for example, it is confirmed whether the hand can be recognized normally. For example, the information processing system 1 checks whether another hand is simultaneously captured, whether it is dark, whether there is an obstruction, etc.

[0305] In the practice mode, for example, the information processing system 1 checks whether the sound is being played back correctly, such as whether the sound is being played back in accordance with the user's movements, whether the device is muted, and whether the volume is within an appropriate range.

[0306] In the practice mode, for example, the information processing system 1 checks whether the display device is moving based on an image captured by a camera or information from an acceleration sensor of a smartphone.

[0307] In the practice mode, for example, ambient light source information is acquired. For example, the information processing system 1 instructs the user to capture an image of the surroundings with a camera on the device to acquire light source information of the experiential environment. The information processing system 1 may instruct the user to change the orientation of their arms or hands. The information processing system 1 may analyze the captured image to check the degree of reflection of light on the user's arms or hands. The information processing system 1 may analyze the captured image to estimate the color and / or position of the light source. This improves the realism of the CG reproduction.

[0308] (An example of another embodiment) In the preparatory operation mode, effects to enhance the sense of immersion may be added. As described above, in the preparatory operation mode, an image is generated so that the degree of reality of at least one object gradually increases from a low state, or an image is generated so that the degree of linkage between the input operation and the movement of the moving object gradually increases from a low state. This encourages the user to become immersed in the experience. Examples of effects to enhance the sense of immersion include the following:

[0309] (A performance in which movements gradually interlock) When moving a moving object, the synchronization is intentionally made poor at first, and then gradually increased to create a sense of belonging and encourage immersion in the experience. Poor synchronization refers to situations where the movement is not smooth, the object does not move despite being moved, or there is a delay in the movement. The above-mentioned triggers gradually synchronize the movement.

[0310] (Image of implementation when opening and closing a plastic bottle cap) Instruct the moving object to rotate near the cap of a plastic bottle. At first, the moving object will not move or will move slowly. Once it has moved a certain distance, it will gradually rotate in conjunction with the movement of your finger, and eventually the cap will come off.

[0311] (The realistic appearance of moving objects gradually increases) At first, the display is gray, with no surface texture or gloss. As the finger moves further, color gradually appears and the texture becomes more realistic. Finally, light source information is included in the reproduction, enhancing realism and promoting immersion.

[0312] (The realism of the main object gradually increases) At first, it appears as a gray, textureless shape that does not deform when touched. Once the moving object is moved over a certain distance while touching the main object, the visual realism gradually increases. This increases the visual realism, including color, texture, and light source information. Gradually, the moving object will also cause the main object to deform.

[0313] (Sound gradually plays) At first, no sound is played. Once the finger is moved a certain distance, a sound is played. When the sound is played, the volume is low at first, but gradually increases. A different pattern to the increasing volume pattern is to have the sound always play at a constant volume at first, but gradually the volume becomes linked to the movement speed.

[0314] Examples of triggers for the effect include the length of time the user is performing the preparatory movement, the total distance the moving object moves during the preparatory movement, the degree of coordination with the guide's movement, and the user's concentration level. The distance between the guide object and the moving object may be monitored, and if the distance is below a threshold for a certain period of time, it may be determined that the user's movement is coordinated with the guide's movement. Whether the user's face is facing the screen may be sensed, and if the user is looking at the monitor for a certain period of time, it may be determined that the user is concentrating.

[0315] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. Furthermore, matters described for a particular embodiment can be applied to other embodiments to the extent that they are not technically inconsistent. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention. Furthermore, when a description is made such as "Examples of X include A, B, C, etc.", it is apparent that X may be at least one of A, B, and C.

[0316] When a device, system, program, method, etc. (sometimes referred to as a device, etc.) is described in the claims, specification, and drawings, the order of execution of each process, such as operations, procedures, steps, and stages, in the device, etc. is not specifically stated as "before," "prior to," etc., and it should be noted that the process can be performed in any order unless there are special circumstances, such as when it is technically not feasible unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specification, and drawings is described using terms such as "first," "next," etc. for convenience, this does not mean that the operational flow must be performed in that order.

[0317] The present specification discloses, for example, the following: The subject of operations in each step of the following information processing method may be a computer. A computer-readable medium storing the following program may be provided. The computer-readable medium may be a non-transitory computer-readable medium. The computer-readable medium may be a computer-readable recording medium.

[0318] [Item A-1] An information processing device that generates an image including a computer graphic of an object to which an external force of a magnitude corresponding to an input from a user is applied, a first acquisition unit that acquires object information related to an object to which the external force is applied; a second acquisition unit that acquires position information of a moving object in real space; a generation unit that generates (i) a first image showing the state of the object that is deformed or moved on the support object due to the interaction between the object, the moving object, and the support object that supports the object, and / or (ii) a second image showing the state of the moving object that is moving on the support object that supports the object, based on the change in the position information and the object information; An information processing device comprising: [Item A-2] The object includes a liquid, a slurry, an emulsion, a semi-solid, a particle aggregate, and a foam aggregate. The information processing device according to item A-1. [Item A-3] a receiving unit that receives position information of the moving object as the input from the user, The generating unit generates the first image so that the object deforms or moves with a delay from the input, and / or generates the second image so that the moving object moves with a delay from the input. The information processing device according to item A-1. [Item A-4] the generation unit generates the first image showing a state in which an external force corresponding to a change in the position information is applied to the object through a physical calculation simulation. The information processing device according to item A-1. [Item A-5] the generation unit generates the first image by a physical calculation simulation that takes into consideration at least one of (i) contact or collision between the target object and the moving object, (ii) friction between the target object and the supporting object, (iii) penetration or diffusion of the target object into the supporting object, and (iv) evaporation of the target object. The information processing device according to item A-4. [Item A-6] The generation unit inputting the object information and the position information into a physics engine that executes the physics calculation simulation; causing the physics engine to output the first image; The physics engine is deriving a region in which the moving object applies the external force to the object in the physics simulation based on the object information and the position information; executing the physical calculation simulation to reproduce or simulate the application of the external force to the region of the object; The information processing device according to item A-4. [Item A-7] The physics engine is determining a relative positional relationship between a representative position of the region and a position indicated by the position information based on the object information; deriving a representative position of the region based on the position indicated by the position information and the relative positional relationship; The information processing device according to item A-6. [Item A-8] the generation unit calculates a degree of deformation or a degree of movement of the moving object based on the movement of the moving object and the target object information, generates the second image showing the moving object moving on the support object based on the calculation result, and synthesizes the first image and the second image to output a third image. The information processing device according to item A-1. [Item A-9] The first acquisition unit further acquires supporting object information indicating characteristics or physical properties of the supporting object; the generation unit generates the first image and / or the second image further based on the supporting object information. The information processing device according to item A-1. [Item A-10] An information processing method for generating an image including computer graphics of an object to which an external force of a magnitude corresponding to an input from a user is applied, comprising: acquiring object information relating to the object to which the external force is applied; Acquire position information of a moving object in real space; Based on the object information, (i) a first image showing the object deforming or moving on a support object due to an interaction with the moving object in accordance with a change in the position information, and / or (ii) a second image showing the moving object moving on the support object supporting the object in accordance with a change in the position information, is generated. An information processing method comprising: [Item A-11] A program for generating an image including computer graphics of an object to which an external force of a magnitude corresponding to an input from a user is applied, acquiring object information relating to the object to which the external force is applied; Acquire position information of a moving object in real space; Based on the object information, (i) a first image showing the object deforming or moving on a support object due to an interaction with the moving object in accordance with a change in the position information, and / or (ii) a second image showing the moving object moving on the support object supporting the object in accordance with a change in the position information, is generated. A program that causes a computer to execute the following:

[0319] [Item B-1] a first acquisition unit that acquires a plurality of feature parameters; a second acquisition unit that acquires position information of a moving object; a generating unit that generates, for each of the plurality of feature parameters, an output image showing the sample fluid deformed or moved by the moving object in accordance with a change in the position information, or an output image showing the moving object moving on the sample fluid in accordance with a change in the position information, based on each of the feature parameters; an output unit that outputs the output image; a reception unit that receives designation of a feature parameter corresponding to a predetermined object from among the plurality of feature parameters, the generating unit further generates a simulation image showing a simulation using the moving object and the fluid, based on the feature parameters received by the receiving unit; The output unit further outputs the simulation image. 1. An information processing device comprising: [Item B-2] The information processing device described in item B-1, wherein the characteristic parameters are the color, gloss, lump size, deformability, spreading speed, evaporation speed, skin deformability, finger deformability, and change over time during deformation of the fluid. [Item B-3] the second acquisition unit further acquires a first movement amount of the moving object; a calculation unit that calculates a second movement amount of a moving object image representing the moving object, the second movement amount corresponding to the first movement amount, based on a movement parameter; the generation unit further generates a second output image showing the moving object image moved by the second movement amount; the output unit further outputs the second output image; the accepting unit accepts the change in the movement parameter; the generation unit generates the simulation image further based on the movement parameters accepted by the acceptance unit. The information processing device according to item B-1. [Item B-4] An information processing device that adjusts parameters used to reproduce or simulate deformation and / or movement of an object due to an external force, a first acquisition unit that acquires one or more of the parameters related to a sample object that is an object used in the adjustment; a second acquisition unit that acquires position information of a moving object in real space; a generating unit that generates (i) a first image showing the state of the sample object deforming or moving on the support object due to the interaction between the sample object, the moving object, and the support object supporting the sample object, and / or (ii) a second image showing the state of the moving object moving on the support object supporting the sample object, based on the change in the position information and one or more of the parameters; a presentation unit that executes a process for presenting the first image and / or the second image to a user; a feedback acquisition unit that acquires feedback information including information indicating an impression that the user, who has been presented with the first image and / or the second image, has about the tactile feel of the sample object or the support object; a parameter adjustment unit that adjusts at least one of the parameters used to reproduce or simulate the deformation and / or movement of the object due to the external force based on the feedback information; An information processing device comprising: [Item B-5] a receiving unit that receives position information of the moving object as an input from the user; The generation unit generates the first image so that the sample object deforms or moves with a delay relative to the input, and / or generates the second image so that the moving object moves with a delay relative to the input. The information processing device according to item B-4. [Item B-6] The first acquisition unit Determine the type of the sample object. acquiring, for each of one or more types of objects, the one or more parameters associated with the type of the sample object by referring to a storage device that stores information indicating the type of the object in association with the one or more parameters used to reproduce or simulate the deformation and / or movement of the object due to the external force; The information processing device according to item B-4. [Item B-7] the storage device stores, for each of the one or more types of objects, information indicating the type of the object, parameter identification information for identifying each of the one or more parameters related to the object, and value information indicating values ​​of the parameters, in association with each other; The parameter adjustment unit adjusts the value of at least one parameter. The information processing device according to item B-6. [Item B-8] Obtaining multiple feature parameters; Obtain the position information of the moving object, generating, for each of the plurality of feature parameters, an output image showing the sample fluid deformed or moved by the moving object in response to a change in the position information based on the feature parameters, or an output image showing the moving object moving on the sample fluid in response to a change in the position information based on the feature parameters; Output the above output image, accepting designation of a feature parameter corresponding to a predetermined object from among the plurality of feature parameters; further generating a simulation image showing a simulation using the moving object and the sample fluid based on the feature parameters whose designation has been accepted; Further outputting the simulation image. Including 1. An information processing method comprising: [Item B-9] 1. An information processing method for adjusting parameters used to reproduce or simulate deformation and / or movement of an object due to an external force, comprising: a first acquisition step of acquiring one or more of said parameters for a sample object that is an object used in said adjustment; a second acquisition stage of acquiring position information of the moving object in real space; a generation step of generating, based on the change in position information and one or more of the parameters, (i) a first image showing the state of the sample object deforming or moving on the support object due to interaction between the sample object, the moving object, and the support object supporting the sample object, and / or (ii) a second image showing the state of the moving object moving on the support object supporting the sample object; a presentation step of performing a process for presenting the first image and / or the second image to a user; a feedback acquisition step of acquiring feedback information including information indicating an impression that the user, who has been presented with the first image and / or the second image, has had about the tactile feel of the sample object or the support object; a parameter adjustment step of adjusting at least one of the parameters used to reproduce or simulate the deformation and / or movement of the object due to the external force based on the feedback information; An information processing method comprising: [Item B-10] A program for causing a computer to function as the information processing device according to any one of items B-1 to B-7.

[0320] [Item C-1] a generation unit that generates an output image including an object image showing a predetermined object; an output unit that outputs the output image; a reception unit that receives a user's operation on the object image; and the generation unit transforms or moves the object image within the output image based on the user's operation accepted by the acceptance unit, and when the user's operation satisfies a predetermined condition, further generates a simulation image showing a simulation in accordance with the user's operation; The output unit further outputs the simulation image. 1. An information processing device comprising: [Item C-2] the receiving unit receives, as the user's operation, an operation to rotate the object image; the predetermined condition is that the object image is separated from a second object image representing a predetermined second object, or that the object image is combined with the second object image; The information processing device according to item C-1. [Item C-3] the accepting unit accepts an operation of pressing the object image as the operation of the user, The predetermined condition is that the object image is pressed a predetermined amount, a predetermined number of times, or for a predetermined period of time or more. The information processing device according to item C-1. [Item C-4] a generation unit that generates an output image including an object image showing a predetermined object; an output unit that outputs the output image; a reception unit that receives a user's operation on the object image; and The generation unit When executing a process for transforming or moving the predetermined object image within the output image based on the operation received by the reception unit, at least one of (i) a degree of delay between the timing at which the reception unit receives the operation from the user and the timing at which the object image is transformed or moved within the output image, and (ii) a representation mode of the appearance of the object is adjustable; When the output unit executes a process for presenting to the user a screen for prompting the user to perform a predetermined operation or a screen designed to cause the user to perform a predetermined operation, the output unit changes at least one of the degree of delay and the expression mode during at least a part of a period during which the reception unit is receiving the predetermined operation. Information processing device. [Item C-5] the generating unit further generates a simulation image showing a simulation according to the user's operation when the user's operation matches the predetermined operation; The output unit further outputs the simulation image. The information processing device according to item C-4. [Item C-6] The predetermined operation includes at least one of the following: (i) an operation of continuing an input operation of the same or similar type or pattern for a period of time equal to or longer than a predetermined length; (ii) an operation of repeating an input operation of the same or similar type or pattern a predetermined number of times or more; and (iii) an operation of continuing an input operation for specifying a position or coordinates on a screen until the degree of change in the position or coordinates meets a predetermined condition. The information processing device according to item C-4. [Item C-7] a degree of the delay at a first point in time included in a period during which the reception unit receives the predetermined operation is smaller than a degree of the delay at a second point in time included in a period during which the reception unit receives the predetermined operation; The first time point is a time point later than the second time point. The information processing device according to item C-4. [Item C-8] The expression mode at a first time point included in a period during which the receiving unit receives the predetermined operation is more realistic than the expression mode at a second time point included in a period during which the receiving unit receives the predetermined operation. The first time point is a time point later than the second time point. The information processing device according to item C-4. [Item C-9] generating an output image including an object image showing the predetermined object; Output the above output image, Accepting a user's operation on the predetermined object, transforming or moving the predetermined object image within the output image based on the user's operation, and if the user's operation satisfies a predetermined condition, further generating a simulation image showing a simulation according to the user's operation; Further outputting the simulation image. An information processing method comprising: [Item C-10] a generating step of generating an output image including an object image showing a predetermined object; an output stage for outputting the output image; a receiving step of receiving a user's operation on the object image; and The generation step includes: When a process for transforming or moving the predetermined object image within the output image is executed based on the operation accepted in the accepting step, a procedure for adjusting at least one of (i) the degree of delay between the timing at which the user's operation is accepted in the accepting step and the timing at which the object image is transformed or moved within the output image, and (ii) the expression mode of the appearance of the object; a step of changing at least one of the degree of delay and the expression mode during at least a part of a period during which the predetermined operation is being accepted, when a process for presenting to the user a screen for prompting the user to perform a predetermined operation or a screen designed to allow the user to perform the predetermined operation is executed; Including, Information processing methods. [Item C-11] A program for causing a computer to function as the information processing device according to any one of items C-1 to C-8. [Explanation of symbols]

[0321] 2. Terminal Device 3. Server equipment 4. Edge computing equipment 220 Terminal Processing Device 302 Server Storage Device 320 Server Processing Unit 321 First acquisition part 322 Second Acquisition Department 323 Generation part 324 Reception Department 325 Calculation Unit 326 Control Unit 420 Edge Processing Equipment

Claims

1. a first acquisition unit that acquires a plurality of feature parameters; a second acquisition unit that acquires position information of the moving object; a generating unit that generates, for each of the plurality of feature parameters, an output image showing the sample fluid deformed or moved by the moving object in accordance with a change in the position information, or an output image showing the moving object moving on the sample fluid in accordance with a change in the position information, based on each of the plurality of feature parameters; an output unit that outputs the output image; a reception unit that receives designation of a feature parameter corresponding to a predetermined object from among the plurality of feature parameters, the generating unit further generates a simulation image showing a simulation using the moving object and the fluid, based on the feature parameters received by the receiving unit; The output unit further outputs the simulation image.

1. An information processing device comprising:

2. The information processing device according to claim 1 , wherein the characteristic parameters are color, gloss, lump size, deformability, spreading speed, volatilization speed, skin deformability, finger deformability, and time change during deformation of the fluid.

3. the second acquisition unit further acquires a first movement amount of the moving object; a calculation unit that calculates a second movement amount of a moving object image representing the moving object, the second movement amount corresponding to the first movement amount, based on a movement parameter; the generating unit further generates a second output image showing the moving object image moved by the second movement amount; the output unit further outputs the second output image; the accepting unit accepts a change to the movement parameter; the generation unit generates the simulation image further based on the movement parameters accepted by the acceptance unit. The information processing device according to claim 1 .

4. An information processing device that adjusts parameters used to reproduce or simulate deformation and / or movement of an object due to an external force, a first acquisition unit that acquires one or more of the parameters related to a sample object that is an object used for the adjustment; a second acquisition unit that acquires position information of a moving object in real space; a generating unit that generates, based on the change in the position information and one or more of the parameters, (i) a first image showing the state of the sample object deforming or moving on the support object due to the interaction between the sample object, the moving object, and the support object that supports the sample object, and / or (ii) a second image showing the state of the moving object moving on the support object that supports the sample object; a presentation unit that executes a process for presenting the first image and / or the second image to a user; a feedback acquisition unit that acquires feedback information including information indicating an impression that the user, who has been presented with the first image and / or the second image, has had about the tactile feel of the sample object or the support object; and a parameter adjustment unit that adjusts at least one of the parameters used to reproduce or simulate the deformation and / or movement of the object due to the external force based on the feedback information; An information processing device comprising:

5. a receiving unit that receives position information of the moving object as an input from the user; the generation unit generates the first image so that the sample object is deformed or moved with a delay from the input, and / or generates the second image so that the moving object is moved with a delay from the input. The information processing device according to claim 4 .

6. The first acquisition unit determining a type of the sample object; referencing a storage device that stores, for each of one or more types of objects, information indicating the type of the object and one or more parameters used to reproduce or simulate the deformation and / or movement of the object due to the external force, in association with each other, and acquiring the one or more parameters associated with the type of the sample object; The information processing device according to claim 4 .

7. the storage device stores, for each of the one or more types of objects, information indicating the type of the object, parameter identification information for identifying each of the one or more parameters related to the object, and value information indicating values ​​of the parameters, in association with each other; the parameter adjustment unit adjusts the value of at least one parameter. The information processing device according to claim 6 .

8. Obtaining multiple feature parameters; Obtain the position information of the moving object, generating, for each of the plurality of feature parameters, an output image showing the sample fluid deformed or moved by the moving object in response to a change in the position information based on the feature parameters, or an output image showing the moving object moving on the sample fluid in response to a change in the position information based on the feature parameters; outputting the output image; accepting designation of a feature parameter corresponding to a predetermined object from among the plurality of feature parameters; further generating a simulation image showing a simulation using the moving object and the sample fluid based on the feature parameters whose designation has been accepted; further outputting the simulation image; Including An information processing method comprising:

9. 1. An information processing method for adjusting parameters used to reproduce or simulate deformation and / or movement of an object due to an external force, comprising: a first acquisition step of acquiring one or more of the parameters for a sample object that is an object used in the adjustment; a second acquisition step of acquiring position information of the moving object in real space; a generation step of generating, based on the change in position information and one or more of the parameters, (i) a first image showing the state of the sample object deforming or moving on the support object due to interaction between the sample object, the moving object, and the support object supporting the sample object, and / or (ii) a second image showing the state of the moving object moving on the support object supporting the sample object; a presentation step of performing a process for presenting the first image and / or the second image to a user; a feedback acquisition step of acquiring feedback information including information indicating an impression that the user, who has been presented with the first image and / or the second image, has had about the tactile feel of the sample object or the support object; a parameter adjustment step of adjusting at least one of the parameters used to reproduce or simulate the deformation and / or movement of the object due to the external force based on the feedback information; An information processing method comprising:

10. A program for causing a computer to function as the information processing device according to any one of claims 1 to 7.

Images