Method and system for interactive sonification of materials

Abstract

The present invention relates to a method for interactive sonification of a physical object with which an image or video representative of the physical object is associated, in which the following steps are performed : A. Projecting the image or video in the form of a graphic texture onto a virtual mesh (200) of connected points to which at least one physical property is associated by means of a dynamic network model of damped elastic masses, where the network parameters are node mass, elasticity, and damping of damped elastic masses; B. Based on an interaction (320) of the user on one or more regions of the image (101), through one or more controllers, perturbing said network of damped elastic masses; C. Based on the displacement (333), velocity (332), acceleration (331) nodes and duration (334) of the B-phase perturbation, generating (340) or retrieve from a memory one or more corresponding sounds; D. Sending said one or more corresponding sounds to a sound-emitting device; E. Emitting (350) said one or more corresponding sounds by said sound emitting device. The invention also concerns a system that implements the method of the invention.

Description

[0001] Method and system for interactive Bonification of materials

[0002] Name: Politecnico di Bari

[0003] Inventors: Michele Fiorentino, Fabio Vangi, Marina

[0004] Ricci, Angela Campana, Alessandra Scarcelli, Annalisa Di Roma

[0005] The present invention relates to a method and system for interactive Bonification of materials.

[0006] Status of the technique

[0007] In modern teleconferencing and e-commerce systems, the user passively looks at images and models without being involved in the experience.

[0008] There are examples in the literature of visual and auditory simulation of materials for greater user involvement, but such simulations are not integrated as a means of exploring materials in a realistic or evocative way.

[0009] Patent document US 2022 / 0261334 Al describes a continuous Bonification method of data but not of physical product characteristics.

[0010] Patent document US 9,918,679 B2 describes a Bonification method applied to physiological parameters in the biomedical field but is not interactive; the sounds are solely representative of a predetermined physiological state.

[0011] Patent document US 2022 / 0061695 Al describes a sonification method that is intended to provide the user with warnings regarding breathing during an exercise.

[0012] Patent document US20150201889A1 describes a sonification method applied to the diagnostic field, in which one diagnosis is distinguished from another by corresponding sounds, without user interaction.

[0013] The publication Martin, R., Weinmann, M., & Hullin, M. B. (2018). A Study of Material Sonification in Touchscreen Devices. ISS 2018 - Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces, 305-310. https: / / doi.org / 10.1145 / 3279778.3281455 describes a sonification methodology, which is unable to distinguish different regions of an image.

[0014] The publication Furfaro, Enrico, et al. "Sonification of virtual and real surface tapping: evaluation of behaviour changes, surface perception and emotional indices." IEEE MultiMedia

[0015] (2015).https: / / hal.science / hal-01572620, although it explores the general principle of sonification of materials, it does so limited to a specific tapping (touch) system using a finger-mounted accelerometer with a ring, so it cannot differentiate different areas of an image.

[0016] The Barrett publication, Natasha. "Interactive spatial sonification of multidimensional data for composition and auditory display." Computer Music Journal 40.2 (2016): 47- 9.https: / / direct .mit.edu / comj / articleabstract / 40 / 2 / 47 / 94535 / Interactive-Spatial- Sonification-of?redirectedFrom=fulItext presents a system for generating sounds corresponding to generic data sets, without user interaction.

[0017] There remains a need to provide a method and system for sonification of physical materials (e.g., product components) through a user's interaction with an image of the physical product, in its parts.

[0018] Purpose and object of the invention

[0019] The purpose of the present invention is to be used in the area of technical, and commercial communication of materials and products, serving both materials research, B2B presentations (trade shows, showrooms, meetings between industry professionals) and B2C e-commerce and sales (on various channels such as websites, metaverse, video games, etc.). The added value concerns complex materials and products and sophisticated finishes (e.g., luxury, made in Italy, custom craftsmanship), where multi-sensory helps to describe the characteristics of materials, finishes, and values.

[0020] The main function is to enhance the user's mode of interaction by integrating beyond just displaying images (state of the art) to the resultant integrated and novel combination of visual deformation of the material and, simultaneously, realistic or evocative sonification of the material of interest. From the point of view of the experience creator (e.g., marketing designer, metaverse creator), this technology offers in addition to the surprise effect, archival, educational, and hedonic aspects to increase the perception, involvement, and recollection of the materials. In addition, it is possible to both follow the driven realism using standardized and reproducible methodologies in acquisition, and that of evocation, driven by creative, anomalous, and unexpected experiences. From the point of view of the end user, professional or not, through the naturalness of single or multiple touch, the technology increases the emotional aspect in addition to the technical aspect of perception, overcoming the current limitations aspect of the visual and static nature of interfaces .

[0021] The subject matter of the present invention is the method and system as described in the attached claims.

[0022] Detailed description of examples of implementation of the invention

[0023] Four implementation examples are presented. A first example is a teleconference of some researchers who need to discuss the outcome of some materials research while not having a sample of actual materials, but having the ability to explore them remotely, getting an indicative and informative response of the outcome. Current teleconferencing tools do not have tools for materials exploration but could provide it by integrating "the sound of touch" as an expansion. In this case, material acquisition must be rigorous, following certified and standardized protocols.

[0024] A second example is B2B in an internal sales context from manufacturing company to retailer or wholesaler (e.g., clothing). In this case the users are experts and need a very in-depth level of knowledge and information about the materials. For the sake of saving time and money, the showroom could take place via metaverse, video or teleconferencing tools and the materials that make up the garments, could become interactive with the "sound of touch" system. In this scenario, the system could be integrated with the retailer's virtual showroom and each buyer user could explore garments and\or finishes not currently available in person. Again, the simulation must be based on certified and realistic systems.

[0025] The system will make it possible to reduce or even eliminate in-person meetings and showrooms (often required to perceive materials and the associated costs) in terms of time, resources, and pollution.

[0026] A third area of application is B2C e-commerce. Compared to the techniques implemented by major marketplaces where the user only sees images in a passive and boring way (e.g., Amazon, eBay, etc.) with this invention the products are explorable, e.g., a shoe or a bag can be explored in its different materials and behaviors. In this case the creator could follow a purely evocative path. The user is presented with the function and informed that the effects are not realistic and invited to participate. Leather parts could be simulated by a low elastic mesh and reproduce the wrinkled surface texture in the form of noise evocative of real leather, metal parts made rigid visually but emit a metallic and jarring sound evocative of swords used in movies. Soft fabric parts can be rendered by accentuated mesh deformation, supplemented with evocative Bonification's reminiscent of extremely soft and emotional materials such as a pillow or wind sounds.

[0027] The last scenario concerns education and training. Both children and adults could use the system on their own or be integrated with other platforms to acquire skills and information about materials that could not be transmitted without real-life testing. An example is in the food industry where the texture and consistency of some recipes is critical. The "sound of touch" system could serve at an early stage and then a real-life trial can follow, reducing waste and dissatisfaction.

[0028] List of figures

[0029] The invention will now be described by way of illustration but not by limitation, with special reference to the drawings in the attached figures, in which:

[0030] - Figure 1 shows a rectangle representing the user's visual space and a smaller interaction zone of closed shape;

[0031] - Figure 2 shows a 3d Mesh (200), consisting of the connected nodes and a perturbation volume (201);

[0032] - Figure 3 shows a working diagram of the method of the invention according to an embodiment;

[0033] - Figure 4 shows a flowchart of an implementation of the interaction method according to the invention.

[0034] It is specified here that elements of embodiments can be combined to provide additional embodiments without limitation respecting the technical concept of the invention, as readily understood by the skilled person based on the description. This description also refers to the known technique for its implementation with respect to nondescript detailed features, such as minor elements usually used in the known technique in solutions of the same type.

[0035] When introducing an element, it is always meant that it can be "at least one" or "one or more."

[0036] When listing a list of elements or features in this description it is understood that the invention according to the invention "includes" or alternatively "consists of" such elements.

[0037] When listing features within the same sentence or bullet list, one or more of the individual features may be included in the invention without connection to the other features in the list.

[0038] Two or more of the parts (elements, devices, systems) described above may be loosely associated and considered as part kits according to the invention.

[0039] Embodiments

[0040] The invention involves transforming the experience of a passive user into a multisensory shopping experience. Specifically, the invention engages the sense of touch interactively, in addition to the sense of vision typically involved in in-store shopping and in online shopping.

[0041] There are examples of visual and auditory simulation of materials in the literature, but they are not integrated as a means of exploring materials in a realistic or evocative way.

[0042] Through interactive sonification and deformation of materials, this important limitation can be reduced. Three touch modes of roughness, elasticity and thickness have been identified. These qualities turn out to be the most important ones when evaluating and purchasing a product according to literature.

[0043] From the interaction with real surfaces, it is possible to capture these qualities in terms of sound (recorded with a microphone and\or generated at the time of interaction) and implement them in the visualization space broadly understood as space on a web page or other forms such as virtual or mixed reality (Fig. 1, 100) on the web or metaverse using HTML tools or by generating or processing them with digital algorithms.

[0044] Therefore, through sonification and visual deformation of materials, it was possible to inform the user about the qualities of the materials by enhancing their online shopping experience, making it the most interesting experience, and more or less similar to the real thing, by acting on the evocative and playful power of sounds, which are not realistic but give feelings of roughness, softness or elasticity and thickness.

[0045] These parameters may or may not adopt standardized and calibrated systems that exert a known force, photograph the result, and capture the sound emitted by precision microphones. This system is easily standardized and automated and thus scalable to complex and numerous material samples.

[0046] By increasing product perception, the system can increase the sense of purchase confidence and can reduce returns and related environmental problems. Referring to Fig. 1, in an embodiment, a rectangle 100 represents the user's visual space (web page or other virtual visual field or mixed reality type forms), with a closed-form interaction zone 101. Associated with this last specific region representing a region of a product are interactive features that combine in an integrated way sounds that correspond to characteristics or sensations that one wants to associate with the chosen material along with a relative simulated deformation thereof by means of an algorithm.

[0047] Referring to Fig. 2, the image or video associated with a representative image of the physical object is represented in the form of a graphical texture on a deformable virtual mesh of connected nodes 200 to which a physical attribute is associated by means of a dynamic network model of damped elastic masses with parameters appropriately sized according to the type of material simulated. In the case of video, the modeling is carried out for each frame or for a subset of frames. In the following we will refer only to the image, but it should be understood that what is said also extends to video frames.

[0048] The parameters of the network are node mass, elasticity and damping, which are diversified depending on the material based on achieving a realism or conversely an evocative effect of the same.

[0049] Referring to the flow diagrams in Figs. 3 and 4 that illustrate an embodiment of the method 300 of the invention, the user, depending on the type of platform, will have a mouse, or touch screen or three-dimensional VR system (or other virtual pointer system), and will have the ability, at the beginning 310 of the process to interact in step 320 with the material by associating with the single pointer (or multiple in the case of multitouch, generally one or more controllers) a three- dimensional element of varying shape that will rigidly perturb the network of masses creating a perturbation volume 201.

[0050] At the same time, the algorithm, based on the nodes' displacement, speed and acceleration will provide a sound output, whether as prerecorded sounds or digital processing of the sounds themselves or creation of digital sounds from algorithms (Fig. 3). The sounds can be, in step 345, a simulation of real materials or evocative sounds such as wind blowing, dry leaves crackling, straw, scraped sandpaper, a crushed pillow, etc. Such sounds, once generated, will be sent in 350 to an appropriate sound emitting device that will emit them.

[0051] In addition, two types of predefined sensations 360 were chosen to be used: roughness and softness (or, alternatively or in addition to softness, elasticity and thickness) .

[0052] Roughness is associated with dynamic interaction 361, that is, when the point of application (e.g., the center of gravity of the interaction zone in Fig. 1, 101) undergoes a displacement over time. The sound generator will use as input not only as the displacement value but also, its processing such as velocity and acceleration.

[0053] Softness or elasticity or thickness is associated in 362 with static interaction, i.e., when the point of application (e.g., the center of gravity of the interaction zone in Fig. 1, 101) remains almost unchanged while remaining active.

[0054] Once the user experiences in 370 the process is finished in 360.

[0055] Such a flowchart is implemented in a computer processor configured to perform its steps and by user interaction means and a sound-emitting device.

[0056] Since the system is based on one or more pointers, it is scalable from single-touch to multi-touch, is crossplatform and thus usable on PCs and smartphones as well as VR and AR systems. The invention is easily implemented and scalable because the grid can be adapted to a particular platform.

[0057] The methodology of the invention is web-integrable, portable to various platforms, does not require onerous equipment costs, and the content is easy to implement.

[0058] An example of a possible embodiment of the invention is an e-commerce site, on-screen or in metaverse, where a user can interact with a dedicated area with a mouse or controller, and get a realistic or evocative or playful sensory experience so as to make the shopping experience memorable, fun, and engaging. The experience creator can intervene on the many parameters such as the image, node parameters, sound generator, sound type, etc.

[0059] The system for interactive sonification of a physical object with which a representative image of the physical object is associated comprises according to one aspect of the invention: - A user interface equipped with a 2D, or 3D pointer configured to select one or more regions of said image;

[0060] - A computerized system configured to carry out an interaction with said image modeled as a virtual mesh of connected nodes to which at least one physical property is associated by means of a dynamic network model of damped elastic masses, where the network parameters are node mass, elasticity, and damping of damped elastic masses, according to the following steps:

[0061] • based on a user interaction on one or more regions of the image, perturbing a network of damp elastic masses;

[0062] • based on the displacement, velocity, node acceleration and duration of the perturbation of step B, generating one or more corresponding sounds;

[0063] • sending one or more corresponding sounds to a sound-emitting device;

[0064] - A sound-emitting device configured to emit said one or more corresponding sounds.

[0065] In the illustrative case of video conferencing capable of realizing a sharing of physical properties of a physical object, the described system used, in which the user interface and sound-emitting device is on one or more clients, and the electronic processor is on a server.

[0066] In the foregoing, preferred embodiments have been described, and variants of the present invention have been suggested, but it is to be understood that those skilled in the art may make modifications and changes without thereby departing from the relevant scope of protection as defined by the appended claims.

Claims

CLAIMS1. Method (300) for interactive sonification of a physical object with which a representative image or video (100) of the physical object is associated, in which the following steps performed:A. projecting the image or video in the form of a graphical texture onto a virtual mesh (200) of connected nodes to which at least one physical property is associated by means of a dynamic network model of damped elastic masses, wherein the network parameters are node mass, elasticity, and damping of damped elastic masses;B. based on user interaction (320) on one or more regions of the image (101) or video, through one or more controllers, perturbing said network of damped elastic masses;C. based on the displacement (333), velocity (332), acceleration (331) of the nodes and duration (334) of the perturbation of phase B, generating or retrieving from a memory (340) one or more corresponding sounds;D. sending said one or more corresponding sounds to a sound-emitting device;E. emitting (350) said one or more corresponding sounds by said sound emitting device.

2. Method according to claim 1, where in step B the type (360) of interaction, static or dynamic, is recognized and:- if the interaction is static, said one or more corresponding sounds of the C phase arerepresentative of one or more among softness, elasticity and thickness (362) of the physical object in said one or more regions of the image;- If the interaction is dynamic, said one or more corresponding sounds of the C phase are representative of a roughness (361) of the object.

3. A system for interactive Bonification of a physical object to which is associated an image or video representative of the physical object, the system comprising :- a user interface equipped with a 2D or 3D pointer configured to select one or more regions of said image or at least one frame of said video;- a computer processor configured to perform an interaction with said image or video modeled as a virtual mesh of connected nodes to which at least one physical property is associated by means of a dynamic network model of damped elastic masses, wherein the parameters of the network are node mass, elasticity and damping of the damped elastic masses, according to steps A to D of the method of any of the preceding claims;- a sound emitting device configured to perform step E of the method of any of the preceding claims.

4. Teleconferencing system configured for sharing physical properties of a physical object, in which the system of claim 3 is used, in which the user interface and sound emitting device is on one or more clients and the electronic processor is on the server.

Images