Method and system for enabling inspection of industrial robot simulations at critical virtual time intervals

By using device sound simulators and machine learning algorithms in the CAD system, real industrial robot motion sounds are generated, which solves the problem that the existing technology cannot truly simulate industrial sounds, and improves the user experience and the authenticity of the simulated environment.

CN118946876APending Publication Date: 2024-11-12SIEMENS INDUSTRY SOFTWARE LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280093945.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-03-22
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing computer-aided design (CAD) systems and their virtual reality (VR)/Augmented Reality (AR) tools cannot realistically simulate the motion sound of industrial robots in factories, lack of real simulation of the sound of industrial equipment, affecting the simulation of users Experience.

Method used

By receiving virtual receiver position and device motion task data, the device sound simulator is used to generate the sound generated during the virtual device motion, and train functions through machine learning algorithms to provide real industrial sound simulation, combining virtual devices and receivers The mutual positioning data processing between them generates motion sounds detected at the receiver position.

Benefits of technology

It realizes the same appearance and sound experience as the real factory in a simulation environment, and enhances the user's simulation experience. Especially in VR and AR tools, it can truly reflect the virtual sound of robot tasks and meet the verification and optimization of industrial professionals. and virtual debugging requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118946876A_ABST
    Figure CN118946876A_ABST
Patent Text Reader

Abstract

Systems and methods for generating motion sounds for a mobile virtual industrial device. Data regarding a location of the virtual receiver is received. Data is received regarding a virtual device that is to generate a sports sound. Data about a motion task of a device is received, wherein the motion task data includes data about a source location and data about a target location. A device sound simulator is applied to input data to obtain an output motion sound that simulates a motion sound of a virtual device moving between a source location and a target location. The motion sound detected at the location of the receiver is determined by processing the output motion sound using differential positioning data based on mutual positioning between the virtual device and the virtual receiver.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to computer-aided design, visualization and manufacturing ("CAD") systems, product lifecycle management ("PLM") systems, product data management ("PDM") systems, production environment simulations, and similar systems (collectively referred to as "product data management" systems or PDM systems) that manage data for products and other items. More specifically, the present disclosure relates to production environment simulations. Background Art

[0002] Software applications for industrial robot simulation for validation, optimization and virtual commissioning purposes.

[0003] Examples of such robotic simulation platforms and systems include, but are not limited to, Computer Assisted Robotics (“CAR”) tools, Process Simulation (a product of the Siemens Group), robotic software simulation tools, software applications for industrial robot simulation, and other systems and virtual stations for industrial robot simulation.

[0004] The Robot Simulation Platform enables simulation engineers to simulate robotic operations performed by multiple industrial robots on a simulated scene on a shop floor. The Robot Simulation Platform is capable of simulating the industrial activities of robots and other mobile industrial equipment in a factory.

[0005] Utilizing virtual reality (“VR”) and augmented reality (“AR”) tools, the CAR system enables users to see the robotic factory in real size, and they get a VR / AR experience of visiting a real factory.

[0006] Professionals working in robotic factories are accustomed to the sounds that factory robots and other mobile equipment produce during their manufacturing operations.

[0007] The current CAR system and its VR / AR tools do not provide users with industrial equipment sounds that realistically simulate the sounds of mobile industrial equipment. Improved techniques for simulating industrial sounds are desired. Summary of the invention

[0008] Various disclosed embodiments include methods, systems, and computer-readable media for generating motion sounds of mobile virtual industrial equipment; the sounds are detectable by a virtual receiver located at a location within a virtual environment of an industrial simulation that includes multiple mobile virtual equipment. A method includes receiving data about the location of the virtual receiver. The method includes receiving data about a virtual device for which motion sounds are to be generated. The method includes receiving input data about a motion task of the device, wherein the motion task data of the device includes data about a source location and data about a target location. The method includes applying a device sound simulator to the input data to obtain an output motion sound that simulates the motion sound of a virtual device moving between a source location and a target location. The method includes determining the motion sound detected at the location of the receiver by processing the output motion sound using differential positioning data based on a mutual positioning between the virtual device and the virtual receiver.

[0009] Various disclosed embodiments include methods, systems, and computer-readable media for generating a training function for motion sounds of moving virtual industrial equipment. A method includes receiving input training data; wherein the input training data includes data about a motion task of the equipment, and the motion task data of the equipment includes data about a source position and data about a target position. The method includes receiving output training data; wherein the output training data includes data about output motion sounds of industrial equipment moving between a source position and a target position; wherein the output training data is related to the input training data. The method includes training a function based on the input training data and the output training data through a machine learning ("ML") algorithm. The method includes providing a training function for modeling an equipment sound simulator.

[0010] The features and technical advantages of the present disclosure have been outlined quite broadly above so that those skilled in the art may better understand the detailed description that follows. Additional features and advantages of the present disclosure that form the subject matter of the claims will be described hereinafter. Those skilled in the art will recognize that they can easily use the disclosed concepts and specific embodiments as a basis for modifying or designing other structures for achieving the same purpose of the present disclosure. Those skilled in the art will also recognize that such equivalent constructions do not depart from the spirit and scope of the present disclosure in its broadest form.

[0011] Before proceeding to the following detailed description, it may be helpful to set forth definitions of certain words or phrases used in this patent document: the terms "include" and "comprise," and their derivatives, mean including but not limited to; the term "or" is inclusive, meaning and / or; the phrases "associated with" and "associated with," and their derivatives, may mean including, included within, interconnected with, containing, contained within, connected to or connected with, coupled to or coupled with, communicable with, cooperating with, interlaced, juxtaposed, proximate, bound to or bound with, having, having the property of, and the like; and the term "controller" refers to any device, system, or portion thereof that controls at least one operation, whether such device is implemented in hardware, firmware, software, or in some combination of at least two identical devices. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether local or remote. Definitions of certain words and phrases are provided throughout the patent document, and one of ordinary skill in the art will understand that such definitions apply in many, if not most, instances to prior and future uses of such defined words and phrases. While some terms may include a variety of implementations, the appended claims may expressly limit these terms to specific implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a more complete understanding of the present disclosure and advantages thereof, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like objects, and wherein:

[0013] Figure 1 A block diagram of a data processing system is shown in which embodiments may be implemented.

[0014] Figure 2 A flow chart for generating motion sounds of mobile virtual industrial equipment according to an embodiment is shown.

[0015] Figure 3 A block diagram for generating motion sounds of three mobile robots according to an embodiment is schematically shown.

[0016] Figure 4 The motion task data of the robot according to the embodiment is schematically shown.

[0017] Figure 5 The motion task data of the joints of the robot according to the embodiment are schematically shown.

[0018] Figure 6 A block diagram for generating motion sounds of a mobile device according to an embodiment is schematically shown.

[0019] Figure 7 A block diagram of a sound simulator for training a device according to an embodiment is schematically shown.

[0020] Figure 8 A block diagram for generating a sound simulator based on a mapping table according to an embodiment is schematically shown. DETAILED DESCRIPTION

[0021] Discussed below Figures 1 to 8 The various embodiments used to describe the principles of the present disclosure in this patent document are merely illustrative and should not be construed as limiting the scope of the present disclosure in any way. It will be appreciated by those skilled in the art that the principles of the present disclosure can be implemented in any appropriately arranged device. The numerous innovative teachings of the present application will be described with reference to exemplary non-limiting embodiments.

[0022] The features, advantages or alternative embodiments herein may be assigned to the objects of other claims, and vice versa. In addition, hereinafter, a solution according to the present embodiment will be described, regarding a method and system for generating motion sounds of mobile virtual industrial equipment, and regarding a method and system for providing a training function for generating motion sounds of mobile industrial equipment. The features, advantages or alternative embodiments herein may be assigned to the objects of other claims, and vice versa.

[0023] Previous technologies could not realistically generate the motion sounds of moving virtual industrial equipment.

[0024] The embodiments disclosed herein provide numerous technical benefits, including but not limited to the following embodiments.

[0025] Embodiments provide simulation users with an industrial "feel" experience to accompany the industrial "look" experience known for industrial simulation systems.

[0026] Embodiments provide users with the same "look and feel" as in a real robotic factory.

[0027] Embodiments enable the generation of industrial sounds through artificial intelligence to improve the user experience for simulation users, especially when they utilize VR and / or AR tools.

[0028] Embodiments enable a VR system to play virtual sounds of robotic tasks that realistically reflect real factory sounds.

[0029] Embodiments enable the generation of mobile industrial equipment in a simulated environment.

[0030] Embodiments enable the generation of industrial sounds perceptible by a virtual receiver based on data from an industrial simulation.

[0031] Embodiments are able to pick up a point in a simulated scene and determine the sounds that are receivable at that pick up point.

[0032] Embodiments enable industrial professionals to realistically check industrial simulations and perform necessary adjustments to industrial simulations for validation, optimization, and virtual commissioning purposes. In embodiments, simulation adjustments performed in a virtual environment are in turn performed in a real industrial environment.

[0033] Figure 1 A block diagram of a data processing system 100 is shown in which an embodiment may be implemented, for example, as a PDM system that is specifically configured by software or otherwise to perform processing as described herein, and in particular as each of a plurality of interconnect and communication systems as described herein. The data processing system 100 shown may include a processor 102 connected to a secondary cache / bridge 104, which in turn is connected to a local system bus 106. The local system bus 106 may be, for example, a peripheral component interconnect (PCI) architecture bus. In the example shown, a main memory 108 and a graphics adapter 110 are also connected to the local system bus. The graphics adapter 110 may be connected to a display 111.

[0034] Other peripheral devices such as a local area network (LAN) / wide area network / wireless (e.g., WiFi) adapter 112 may also be connected to the local system bus 106. An expansion bus interface 114 connects the local system bus 106 to an input / output (I / O) bus 116. The I / O bus 116 connects to a keyboard / mouse adapter 118, a disk controller 120, and an I / O adapter 122. The disk controller 120 may be connected to a storage device 126, which may be any suitable machine-usable or machine-readable storage medium, including, but not limited to: non-volatile, hard-coded type media, such as read-only memory (ROM) or electrically erasable programmable read-only memory (EEPROM), magnetic tape storage devices; and user-recordable type media, such as a floppy disk drive, a hard drive, and a compact disk read-only memory (CD-ROM) or digital versatile disk (DVD) and other known optical, electrical, or magnetic storage devices.

[0035] In the example shown, an audio adapter 124 is also shown connected to I / O bus 116 to which speakers (not shown) may be connected for playing sounds. A keyboard / mouse adapter 118 provides a connection for a pointing device (not shown) such as a mouse, trackball, trackpointer, touch screen, or the like.

[0036] Those of ordinary skill in the art will recognize that Figure 1The hardware shown can be changed for a specific implementation. For example, other peripheral devices such as optical disk drives, etc. can be used in addition to or instead of the hardware shown. The examples shown are provided for illustrative purposes only and are not meant to imply architectural limitations regarding the present disclosure.

[0037] The data processing system according to the embodiment of the present disclosure may include an operating system using a graphical user interface. The operating system allows multiple display windows to be presented simultaneously in the graphical user interface, and each display window provides an interface to different applications or different instances of the same application. The cursor in the graphical user interface can be operated by the user through a pointing device. The position of the cursor can be changed and / or an event such as clicking a mouse button can be generated to initiate a desired response.

[0038] One of a variety of commercial operating systems, such as versions of Microsoft Windows, produced by Microsoft Corporation of Redmond, Wash., may be employed if suitably modified. An operating system may be modified or created in accordance with the present disclosure as described.

[0039] LAN / WAN / wireless adapter 112 can be connected to network 130 (not part of data processing system 100), which can be any public or private data processing system network or combination of networks known to those skilled in the art, including the Internet. Data processing system 100 can communicate through network 130 with server system 140, which is also not part of data processing system 100, but can be implemented as, for example, a separate data processing system 100.

[0040] Figure 2 A flow chart of a method for generating motion sounds of mobile virtual industrial equipment according to the disclosed embodiment is shown. Such a method can be, for example, Figure 1 The present invention is performed by the system 100, but the "system" in the following process may be any device configured to perform the described process.

[0041] As used herein, the term industrial equipment refers to any industrial object whose motion produces sound detectable by a human at a receiver location. Examples of industrial equipment include robots; conveyors; turntables; subunits of complex industrial equipment (e.g., joints of a robot); moving elements of motion equipment; and other types of industrial mobile equipment.

[0042] The motion sounds may be detected by a virtual receiver located at a location within an industrial simulated virtual environment including a plurality of mobile virtual devices.

[0043] At action 205, data regarding the location of the virtual receiver is received. In an embodiment, the receiver location may be obtained by a VR / AR tool. In an embodiment, the receiver location is obtained by selecting (e.g., manually or automatically selecting) a specified location in a virtual space within the virtual environment.

[0044] At act 210, data about a virtual device for which motion sounds are to be generated is received. Examples of virtual device data include, but are not limited to, the type of industrial equipment, its model and vendor, the type of tool, and the type of operation and process to be performed by the device.

[0045] At action 215, input data is received. The input data includes data of a motion task of the device; the motion task data of the device includes data about a source position and data about a target position. In an embodiment, the device motion task data is preferably received by a simulation system.

[0046] At action 220, an equipment sound simulator is applied to the input data to obtain an output motion sound that simulates the motion sound of a virtual equipment moving between a source location and a target location. In an embodiment, other sounds of the equipment can be added based on the input equipment operation data. The term "equipment operation" or simply "operation" as used herein refers to an industrial process performed by the equipment. Examples of equipment operations include, but are not limited to, welding, laser cutting, water jetting, and other noisy operations performed by industrial equipment.

[0047] At act 225, the motion sound detected at the location of the receiver is determined by processing the output motion sound using differential positioning data based on the mutual position between the virtual device and the virtual receiver. In an embodiment, the differential positioning data includes the distance between the receiver location and the industrial equipment location, and data about any obstacles present in the virtual space between the receiver location and the virtual industrial equipment location that have an effect on the detected sound. In an embodiment, the differential positioning data is received by a simulation system.

[0048] In an embodiment, the user performs changes in the simulated virtual space by listening to the detected motion sounds, whereby the performed changes are applied in the real industrial environment.

[0049] In an embodiment, the device sound simulator may be configured to output simulated motion sounds by heuristically selecting motion sounds based on stored motion sounds (e.g., stored in a mapping table) according to received input data. The stored motion sounds may be obtained from recorded motion sounds or by synthesized sounds stored in a sound module that includes a fixed mapping table accessible via a heuristic function.

[0050] In an embodiment, the device sound simulator may be obtained by a function trained via a machine learning algorithm.

[0051] In an embodiment, actions 210 - 225 may be performed for a group of mobile virtual devices, and the combined industrial sound at the receiver location is calculated as a combination of the motion sounds of the group of determined corresponding devices.

[0052] In an embodiment, for input data, the data of the source S and the target T may be conveniently selected from the group consisting of:

[0053] -S / T coordinate positions, S / T motion parameters and configuration of S / T equipment;

[0054] -S / T coordinate positions and configuration of S / T equipment;

[0055] -S / T device posture and S / T motion parameters;

[0056] -S / T equipment position.

[0057] In an embodiment, the sound simulator may advantageously be device specific, and the device may be selected from the group consisting of:

[0058] -General purpose robots;

[0059] - Specific types of robots;

[0060] - a robot without tools;

[0061] - A robot with a tool;

[0062] - rotating table;

[0063] - conveyors;

[0064] -Joints of the robot;

[0065] - Moving elements of sports equipment.

[0066] In an embodiment, a trained function for generating motion sounds of moving virtual industrial equipment is provided by:

[0067] - receiving input training data; wherein the input training data comprises data about a motion task of the device; the motion task data of the device comprises data about a source position and data about a target position;

[0068] - receiving output training data; wherein the output training data comprises data regarding output motion sounds of industrial equipment moving between a source location and a target location; wherein the output training data is related to the input training data;

[0069] - Train the function using a machine learning algorithm based on input training data and output training data;

[0070] - Provides training functions for modeling industrial sound simulators.

[0071] Of course, those skilled in the art will recognize that certain steps in the above-described processes may be omitted, performed simultaneously or sequentially, or performed in a different order unless specifically indicated or required by the order of operations.

[0072] Figure 3 A block diagram for generating motion sounds of three mobile robots according to an embodiment is schematically shown.

[0073] A simulation platform, such as, for example, the CAR system of Process Simulation (a software product of the Siemens Group), simulates a virtual environment of a robot factory 301 including three robots r1 , r2 , r3 .

[0074] The receiver position is the position of the virtual user 316 in the virtual environment. In an embodiment, the position of the virtual receiver 316 in the digital factory is determined by manually selecting 336 or by automatically selecting a corresponding position in the virtual environment. In other embodiments, the position of the virtual receiver is obtained 336 from a VR and / or AR tool (not shown) used by the real user 306. In summary, the receiver position 316 reflects 336 the VR / AR tool positioning of the real user 306 and can be moved via VR or by picking / selecting directly on the CAR tool - manually selecting 336 or automatically pre-determined (not shown).

[0075] In an embodiment, the real user 306 is provided with industrial sounds 327 of mobile devices that realistically simulate the digital factory 301 according to the virtual location 316 of the real user 306 in the simulated virtual environment.

[0076] The equipment sound module 302 generates motion sounds 322 based on input motion task data 321 of three industrial equipment, namely three robots r1, r2, r3. The input motion task data 321 includes data describing the motion task of each robot, especially data about its source position and target position (not shown).

[0077] In an embodiment, the device sound module 302 is composed of multiple device sound modules 352 dedicated to specific types of equipment. Examples of equipment types include robots of a given manufacturer (e.g., KUKA or ABB), "naked" (i.e., without tools) robots, robots with tools, conveyors, joints of robots, rotating tables, or other types of industrial equipment that generate motion sounds. The corresponding selectable sound (sub) modules 352 are shown. In an embodiment, these modules are modeled by functions trained via ML algorithms.

[0078] In embodiments, the generated sound may relate to all mobile devices of the virtual environment, or it may be for only a pre-selected subset of mobile devices.

[0079] The operation device sound module 304 generates an operation sound 324 based on the type of operation performed by the robot. In addition, given that some material properties may have an impact on the generated operation sound, the operation type can be classified according to the characteristics of the material (e.g., its thickness and its material type). In an embodiment, the corresponding operation sound with / without material characteristics is stored and made retrievable. A corresponding selectable sound submodule 354 is shown.

[0080] At the sound mixer 303 , the device sounds for the motion tasks and for the processing type for each device are mixed together and sent 325 to the calculator module 305 .

[0081] The calculator module 305 additionally receives differential positioning data 326 from the simulation platform 301. For each robot r1, r2, r3, the received data 326 includes its respective distance d1, d2, d3 in the virtual environment from the virtual receiver 316. The sound calculator 305 attenuates the sound received at the receiver position 316 for each motion sound of each robot by taking into account the distance d1, d2, d3 from the sound source r1, r2, r3 and obstacles (not shown).

[0082] As will be readily appreciated by those skilled in the art, in an embodiment, the attenuation of the motion sound is dynamically calculated by taking into account time-related information about the motion device and about the motion receiver.

[0083] The sound module receives data about device activity - such as robot motion tasks and robot operations - at defined time intervals and returns the relative sound perceptible at the receiver's location.

[0084] It is easy for a person skilled in the art to understand that the real user 306 hears an analog converted version of the digital version of the industrial generated sound 327 through a D / A converter (the D / A converter is not shown in the figure).

[0085] In an embodiment, the user 306 is provided with a final processed sound that realistically simulates the industrial sounds experienced by the virtual user at the receiver location 316 in the virtual factory. Those skilled in the art will readily appreciate that such a final processed sound may preferably include a wide selection of additional noises and effects, such as background noise, factory noise, stereo effects, echo effects, and other types of noises and effects.

[0086] In summary, in an embodiment, the industrial sound generator module (not shown) can be viewed as a meta-module (not shown) that combines the sound generation modules 302, 303, 304, 305 with other industrial sounds and sound effects (not shown). The meta-module (not shown) receives the following information as input data: information about industrial motion equipment and operations that generate dynamic sounds; corresponding relative receiver position data, such as distance from the equipment, orientation of the receiver, and orientation of the sound source; obstacles present in the sound path; and other relevant information.

[0087] In an embodiment, the sound modules 302, 304 and their sub-modules 352, 354 are grouped in a sound simulator meta-module (not shown). In an embodiment, any sound (sub)module 302, 304, 352, 354 or meta-module (not shown) can be advantageously provided in the cloud and / or as software as a service.

[0088] In embodiments, the sound simulator module may be pre-generated and provided to the end user. In other embodiments, the user may perform their own ML training or generate a fixed mapping.

[0089] In an embodiment, the device sound simulator is for a specific type of device. In other embodiments, the device sound simulator is universal for general devices to suit a wide range of different types of devices.

[0090] In an embodiment, the sound perceived at the receiver location (e.g., VR user location) is generated by performing one or more of the following steps:

[0091] a. Use the prepared sound module

[0092] b. For each time during the simulation,

[0093] i. Send each robot a task, such as movement + optional operation

[0094] ii. Calculate the correct sound for each robot movement and operation

[0095] iii. Calculate relative sound based on the mutual distance between the user and the robot

[0096] iv. Provide the user with a mix of all relative sounds.

[0097] Example Implementation: Generating a Group of Mobile Robot Sounds

[0098] Figure 4The motion task data of the robot according to the embodiment is schematically shown. An exemplary robot r having six joints (j1, j2, j3, j4, j5, j6) and a gun tool performs a robot motion task between a source position S and a target position T.

[0099] In an embodiment, the motion data of the robot including data of source and target positions to be input to the robot sound generator includes source S and target T position data and the configuration of the robot at the source S and target T.

[0100] Preferably, motion S / T parameters are also provided - such as motion type, speed, acceleration, area, etc. - for example in order to realistically simulate the sound given that characteristics like speed and acceleration have a direct impact on the robot sound.

[0101] In an embodiment, the input data regarding the S / T position may preferably be given as 3D Cartesian coordinates.

[0102] In an embodiment, the input data regarding the S / T position may be given as a robot pose with joint values ​​(j1, j2, j3, j4, j5, j6).

[0103] Figure 5 Schematically illustrates motion task data of joints of robot r1 according to an embodiment. In joint mode, S / T positions can be input via joint jog "GUI to enter joint values ​​numerically" or via movable cursor to enter interconnected steering poses or via a combination thereof.

[0104] Figure 6 A block diagram for generating motion sounds of a mobile device according to an embodiment is schematically shown. Assume that the mobile device is a robot r, which is similar to Figure 4 The robot shown.

[0105] In an embodiment, the sound module Ms 601 - a sound simulator of the device - receives as input data 602 motion data about a motion task (S, T) of the robot and outputs a simulation of the sound 603 generated by the device moving along the path of the motion task.

[0106] In an embodiment, the sound module of the robot receives as input data the source pose and the target pose of the robot at a given point in time, and outputs the digital sound of the robot generated by the mobile robot at the same point in time. Further sound processing includes processing the sound based on the mutual positioning information between the sound source and the receiver (e.g., relative position and relative direction) and the potential obstacles present along the sound path in the factory. In an embodiment, the final sound detectable at the receiver position is obtained by processing the sound, for example, by attenuation and by applying other sound effects.

[0107] In an embodiment, the sound module MS 601 transmits the signal to the user through f ML The training algorithm trains the function ML to model.

[0108] In an embodiment, the sound simulator 601 of the device is a module Mx obtained by mapping or function based on stored sounds (eg, sounds from recordings or synthetically generated sounds).

[0109] In an embodiment, the output of the sound module Ms 601 is a list of sounds performed by the device during movement, ie, sounds and times (eg, milliseconds) at sampling time points (eg, the same sampling interval used during ML training).

[0110] Figure 7 A block diagram of a sound simulator for training a device according to an embodiment is schematically shown.

[0111] ML Module M ML 701 generates a training function 704 from training data sets 702 and 703.

[0112] In an embodiment, the input training data 702 includes a data set about a motion task of a device. Assuming that the device is a robot of a given type, the input training data includes data about the source position and target position of the robot.

[0113] The output training dataset 703 is obtained by acquiring the sound of the device along the path at a defined time point (e.g., every 1 millisecond) for each input training dataset. In an embodiment, the sound of the device is generated by digitally recording the sound of a real device (e.g., a robot with or without tools).

[0114] The implementation of the ML training algorithm includes the following steps:

[0115] - receiving input training data 702 about a motor task of the device;

[0116] - receiving output training data 703 comprising motion sounds generated by industrial equipment moving between a source location and a target location (S, T) at defined points in time;

[0117] - Train the ML algorithm 701 to obtain a training function f for modeling a device motion sound simulator ML 704.

[0118] In an embodiment, the training data set—both input and output—can be prepared synthetically by generating a list of random robot (S, T) position pairs, by moving the robot between the random position pairs, and by recording the sounds generated by the mobile robot along the path at defined points in time.

[0119] In an embodiment, the input training data set includes a set of data including a source position S with its motion parameters, a target position T with its motion parameters, and a robot configuration at the source position and the target position (S, T).

[0120] In other embodiments, motion parameters are not included, and thus the input training data set includes a set of data including a source position S, a target position T, plus the robot configuration at the source position and the target position (S, T).

[0121] In an embodiment, the source and target (S, T) may be given as Cartesian coordinates (see Table 1) or as robot poses (see Table 2).

[0122] In Table 1 below, there is a really simple pseudocode example where <source position S, target position T>, where (S, T) is given as a Cartesian position with configuration and motion parameters.

[0123]

[0124] Table 1: (S, T) given as Cartesian positions

[0125] In an embodiment, the configuration of the robot can be given as one of the following configuration strings: for example, J5-J6-OH-, J5+J6-OH-, J5-J6-OH+, J5+J6-OH+, etc.

[0126] In Table 2 below, there is a really simple pseudocode example, where <source robot pose Sr, source robot pose Tr>, where (Sr, Tr) are given as joint values.

[0127]

[0128] Table 3: (Sr, Tr) is given as the robot poses of the source and target.

[0129] In an embodiment, the output training data set includes a list of sounds recorded at determined time points from a robot moving between a list of target and source locations, with / without motion parameters and with / without configuration. The recorded sounds are digitally sampled, and the determined time points may be downsampled subsets of the digitally recorded sounds, or may be upsampled by reconstruction techniques.

[0130] In an embodiment, the robot or device is considered as a single unit, and the recorded sounds and corresponding training functions relate to the entire device unit.

[0131] In other embodiments, the device is considered as a collection of moving units or elements. For example, a robot can be considered as a collection of six joints. Sound is recorded for each individual joint, for example by also using noise cancellation techniques when needed. In an embodiment, each element of the device is considered as a device and a specific sound module is trained for the unit / joint.

[0132] In summary, the motion device can be considered as a single unit of the whole or a combination of joints, and thus a single module or several joint modules of the robot can be trained using input motion tasks and output sound and time.

[0133] As those skilled in the art readily appreciate, the sound of the entire robot is the accumulation of the sounds of all its joints over time.

[0134] In an embodiment, the robot is divided into individual units, for example into six joints.

[0135] In Table 3 below, there is a pseudocode example of how to generate the input training dataset for training the sound module for each robot joint. For example, the motion task data of a joint is given not only in terms of source position Sj, target position Tj, but also in terms of intermediate poses Mi,j. The output training data is a list of sounds of motion along the relevant joint.

[0136]

[0137] Table 3: The motion tasks of the input joints are given as joint values ​​and the robot poses.

[0138] In an embodiment, during a training phase in which the ML training sound simulator module is generated using training data, the training function may adapt to new environments, and may detect and infer patterns.

[0139] Typically, the parameters of the training function can be adaptively adjusted by training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning and / or active learning can be used. In addition, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of the training function can be iteratively adaptively adjusted through several steps of training.

[0140] In particular, the training function may include a neural network, a support vector machine, a decision tree and / or a Bayesian network, and / or the training function may be based on k-means clustering, Q-learning, a genetic algorithm and / or association rules.

[0141] In particular, the neural network can be a deep neural network, a convolutional neural network or a convolutional deep neural network. In addition, the neural network can be an adversarial network, a deep adversarial network and / or a generative adversarial network.

[0142] In embodiments, the device sound simulator may be modeled as a set of selectable ML training modules, which may be modeled as a set of heuristically selectable stored motion sounds, and / or may be modeled as any combination thereof.

[0143] For example, in an embodiment, the stored movement sounds to be selected heuristically may be part of a fixed pre-stored mapping table obtained from real or from synthetic robot sounds.

[0144] Figure 8 A block diagram for generating a sound simulator based on a fixed mapping table according to an embodiment is schematically shown.

[0145] In an embodiment, the input mapping motion data 802 includes a data set about the motion task of the device. Assuming that the device is a robot of a given type, the input data includes data about the source position and target position of the robot.

[0146] The output mapping dataset 803 is obtained by recording the sound of the device along the path at defined time points (e.g., every 1 millisecond) for each input training dataset. In an embodiment, the sound of the device is generated by recording the sound of a real device (e.g., a robot with or without tools).

[0147] An embodiment for generating a sound simulator 804Mx with pre-stored tables or functions includes the following steps:

[0148] - receiving mapped motion data 802, the data comprising data on a motion task of the device, the data on the motion task of the device having data on a source position and a target position (S, T);

[0149] - receiving output data comprising motion sounds generated by industrial equipment moving between a source location and a target location (S, T) at defined points in time;

[0150] - Generate a fixed mapping table or function Mx 804 for heuristically modeling the device motion sound simulator.

[0151] In an embodiment, a real physical robot is provided for recording sounds to generate a specific mapping table or mapping function for associating tasks of the robot with / without the motion parameters of the robot with the sounds of the robot. During use, the stored function Mx is able to select the closest sound entry in the mapping for each upcoming robot task. In an embodiment, the stored recorded sounds can be selected based on close joint values, close position values ​​and / or close speeds. In an embodiment, the robot can be a bare robot or with tools. In an embodiment, sounds of robot actions or robot operating activities can be added.

[0152] Although an exemplary embodiment for a conventional robot having six joints has been described in detail, one skilled in the art will be able to implement embodiments for other types of robots or other types of mobile industrial equipment.

[0153] Those skilled in the art will recognize that, for simplicity and clarity, the complete structure and operation of all data processing systems applicable to the present disclosure are not shown or described herein. Instead, only data processing systems that are unique to the present disclosure or are necessary for understanding the present disclosure are shown and described. The remaining structure and operation of the data processing system 100 can comply with any of the various current implementations and practices known in the art.

[0154] It is important to note that while the present disclosure includes a description in the context of a fully functional system, those skilled in the art will recognize that at least portions of the present disclosure can be distributed in the form of instructions contained in any of a variety of forms of machine-usable, computer-usable or computer-readable media, and that the present disclosure applies equally regardless of the particular type of instruction or signal-bearing media or storage device media used to actually implement the distribution. Examples of machine-usable / readable or computer-usable / readable media include: non-volatile, hard-coded type media, such as read-only memory (ROM) or electrically erasable programmable read-only memory (EEPROM); and user-recordable type media, such as floppy disk drives, hard disk drives, and compact disk read-only memory (CD-ROM) or digital versatile disks (DVD).

[0155] Although exemplary embodiments of the present disclosure have been described in detail, those skilled in the art will understand that they may make various changes, substitutions, variations, and alterations herein without departing from the spirit and scope of the disclosure in its broadest form.

[0156] Nothing in the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included in the claims scope: the scope of patented subject matter is limited only by the allowed claims.

Claims

1. A method of generating, by a data processing system, motion sounds of mobile virtual industrial equipment; the motion sounds being detectable by a virtual receiver positioned at a location within a virtual environment of an industrial simulation including a plurality of mobile virtual equipment, The method comprises the following steps: - receiving data about the location of the virtual receiver; - receiving data about a virtual device for which motion sounds are to be generated; - receiving input data about a motion task of the device; the motion task data of the device comprising data about a source position and data about a target position; - applying a device sound simulator to the input data to obtain output motion sounds simulating the motion sounds of the virtual device moving between the source location and the target location; - determining the motion sounds detected at the location of the receiver by processing the output motion sounds using differential positioning data based on the mutual positioning between the virtual device and the virtual receiver.

2. The method according to claim 1, wherein: The device sound simulator is configured to output a simulated motion sound by heuristically selecting a motion sound based on stored motion sounds according to received input data.

3. The method according to claim 1, wherein: The device sound simulator is obtained by a function trained via a machine learning algorithm.

4. The method according to claim 1 or 3, wherein: The data of the receiver position is obtained by a virtual reality tool or by selection within the virtual environment.

5. The method according to claim 1 or 3, wherein: Steps b) to e) are performed for a group of mobile virtual devices, and a combined industrial sound at the receiver location is calculated as a combination of the movement sounds of a group of determined corresponding devices.

6. The method according to claim 3, wherein: For the input data, the data of the source S and the target T are selected from the group consisting of: -S / T coordinate positions, S / T motion parameters and configuration of S / T equipment; -S / T coordinate positions and configuration of S / T equipment; -S / T device position and S / T motion parameters; -S / T equipment position.

7. The method according to claim 3, wherein: The sound simulator is device specific, and the device is selected from the group consisting of: -General purpose robots; - Specific types of robots; - a robot without tools; - A robot with a tool; - rotating table; - conveyors; -Joints of the robot; - Moving elements of sports equipment.

8. A data processing system comprising: processor; as well as The data processing system is configured to: - receiving data about the location of the virtual receiver; - receiving data about a virtual device for which motion sounds are to be generated; - receiving input data about a motion task of the device; the motion task data of the device comprising data about a source position and data about a target position; - applying a device sound simulator to the input data to obtain output motion sounds simulating the motion sounds of the virtual device moving between the source location and the target location; - determining the motion sounds detected at the location of the receiver by processing the output motion sounds using differential positioning data based on the mutual positioning between the virtual device and the virtual receiver.

9. The data processing system according to claim 8, wherein: The device sound simulator is obtained by a function trained via a machine learning algorithm.

10. The data processing system according to claim 8 or 9, wherein: For the input data, the data of the source S and the target T are selected from the group consisting of: -S / T coordinate positions, S / T motion parameters and configuration of S / T equipment; -S / T coordinate positions and configuration of S / T equipment; -S / T device position and S / T motion parameters; -S / T equipment position.

11. A non-transitory computer readable medium encoded with executable instructions that, when executed, cause one or more data processing systems to: - receiving data about the location of the virtual receiver; - receiving data about a virtual device for which motion sounds are to be generated; - receiving input data about a motion task of the device; the motion task data of the device comprising data about a source position and data about a target position; - applying a device sound simulator to the input data to obtain output motion sounds simulating the motion sounds of the virtual device moving between the source location and the target location; - determining the motion sounds detected at the location of the receiver by processing the output motion sounds using differential positioning data based on the mutual positioning between the virtual device and the virtual receiver.

12. The non-transitory computer readable medium of claim 11, wherein: The device sound simulator is obtained by a function trained via a machine learning algorithm.

13. The non-transitory computer readable medium according to claim 11 or 12, wherein: For the input data, the data of the source S and the target T are selected from the group consisting of: -S / T coordinate positions, S / T motion parameters and configuration of S / T equipment; -S / T coordinate positions and configuration of S / T equipment; -S / T device position and S / T motion parameters; -S / T equipment position.

14. A method for providing, by a data processing system, a training function for generating motion sounds of a mobile virtual industrial device; The method comprises the following steps: - receiving input training data; wherein the input training data comprises data about a motion task of the device; the motion task data of the device comprises data about a source position and data about a target position; - receiving output training data; wherein the output training data comprises data about output motion sounds of industrial equipment moving between the source location and the target location; wherein the output training data is related to the input training data; - training a function based on the input training data and the output training data by a machine learning algorithm; - providing said training function for modeling a device sound simulator.

15. A data processing system comprising: processor; as well as The data processing system is configured to: - receiving input training data; wherein the input training data comprises data about a motion task of the device; the motion task data of the device comprises data about a source position and data about a target position; - receiving output training data; wherein the output training data comprises data about output motion sounds of industrial equipment moving between the source location and the target location; wherein the output training data is related to the input training data; - training a function based on the input training data and the output training data by a machine learning algorithm; - providing said training function for modeling a device sound simulator.

16. A non-transitory computer readable medium encoded with executable instructions that, when executed, cause one or more data processing systems to: - receiving input training data; wherein, The input training data includes data about the motion task of the device; the motion task data of the device includes data about the source position and data about the target position; - receiving output training data; wherein the output training data comprises data about output motion sounds of industrial equipment moving between the source location and the target location; wherein the output training data is related to the input training data; - training a function based on the input training data and the output training data by a machine learning algorithm; - providing said training function for modeling a device sound simulator.