Method and system for predicting a collision-free pose of a motion system

Abstract

A method and system for predicting collision-free poses of a motion system, the method comprising: receiving a 3D virtual environment; receiving a 3D representation of the motion system, hereinafter referred to as a 3D virtual motion system, and a set of 3D poses defined for the 3D virtual motion system; receiving a target task to be performed by the motion system relative to the surrounding environment; receiving a predetermined position within the 3D virtual environment, the predetermined position defining the localization in which the 3D virtual motion system must be placed within the 3D virtual environment; applying a collision-free detection function, hereinafter referred to as a CFD function, trained by a machine learning algorithm to an input dataset including the 3D virtual environment, the target task, the predetermined position, and the set of poses, the CFD function being configured to generate a set of collision-free poses as output that enable the motion system to perform the target task when located at the predetermined position; selecting at least one collision-free pose from the set; and displaying the 3D virtual motion system represented by the selected collision-free pose in the 3D virtual environment.

Description

Technical Field

[0001] This disclosure generally relates to computer-aided design, visualization, and manufacturing (“CAD”) systems, product lifecycle management (“PLM”) systems, product data management (“PDM”) systems, and similar systems for managing data used for products and other items (collectively, “Product Data Management” systems or PDM systems). More specifically, this disclosure relates to production environment simulation. Background Technology

[0002] Complex motion systems, such as robots, are widely used in industrial applications to perform automated or semi-automated robotic operations along predefined or real-time calculated trajectories. Collisions between complex motion systems and their surrounding environment should be avoided. This is especially true when programming paths for complex motion systems (such as robots) or simulating the task activities of complex motion systems (such as humans) in a three-dimensional (3D) environment. Typical simulation scenarios may include virtual assessments of range, clearance, or performance capabilities associated with interactions with a product or workplace. The goal is to create realistic and collision-free animations of the expected kinematic model interactions within the surrounding environment, enabling accurate evaluation of the proposed design.

[0003] For example, human performance models require accurate representations of posture and movement for accurate injury risk prediction. Therefore, the fidelity of simulated postures is crucial. Specifically, one challenge revolves around defining posture in the presence of obstacles. A person may need to bend over, reach out, or otherwise alter their neutral posture to avoid collisions with their surroundings. This is a challenging problem to address in simulations. Specifically, modern human models have over 70 joints, and representing realistic postures requires considering many factors, including balance, physiological range of motion limitations on joints, strength capabilities, and more. While methods such as Rapid Exploration Random Trees (RRT) can be used to address such problems, current techniques are time-consuming to execute. In addition to the computational time required to search for solutions, known techniques rely on rapidly performing collision detection on the geometry of the scene, which typically requires time-consuming voxelization of the surrounding environment.

[0004] Various methods have been used to provide collision-free poses. These methods include, for example, manually adjusting individual joints in the kinematic chain, using inverse kinematics to mitigate / accelerate manually chained poses, optimization techniques involving constraint surfaces representing collision boundaries, analyzing empirical behavioral models, and using path planning techniques, such as Regression-Regression Theory (RRT), to find collision-free poses. Unfortunately, these known methods incur considerable time costs for manual interaction or computational solutions. For example, performing manual correction for such collisions in a simulation is a time-intensive activity requiring manipulation of many joints and degrees of freedom. This time burden limits the value of human simulation techniques—simulating all tasks of interest is simply too expensive.

[0005] Therefore, improved techniques for collision-free postures are desired. Summary of the Invention

[0006] This invention proposes calculating collision-free poses during the simulation of complex motion systems such as robots or humans. Various disclosed embodiments include methods, systems, and computer-readable media for predicting the poses of motion systems that do not collide with their surrounding environment.

[0007] A method includes receiving a 3D virtual environment as a 3D representation of an surrounding environment. The method also includes receiving a 3D virtual motion system as a 3D representation of a motion system and receiving a predefined set of 3D poses for the 3D virtual motion system. The method includes defining or receiving a target task to be performed by the motion system, and defining or receiving a predetermined position within the 3D virtual environment, the predetermined position defining the location within the 3D virtual motion system that must be placed within the 3D virtual environment to create a 3D system including the 3D virtual environment and the 3D virtual motion system placed at the predetermined position. The method then includes applying a collision-free detection function—hereinafter referred to as a CFD function—trained by a machine learning algorithm to an input dataset including the 3D virtual environment, the predetermined position, the set of poses, and the target task, and the CFD function being configured to generate an output dataset for the 3D virtual motion system, the output dataset being a subset of the set of 3D poses that enable the motion system to perform the target task from the predetermined position without collision with the surrounding environment. The method then includes selecting at least one collision-free pose from the output dataset and displaying the selected collision-free pose of the 3D virtual motion system within a 3D virtual environment.

[0008] A data processing system is also disclosed, including a processor and an accessible memory or database, wherein the data processing system is configured to implement the aforementioned method.

[0009] The present invention also proposes a non-transitory computer-readable medium encoded with executable instructions, which, when executed, cause one or more data processing systems to perform the aforementioned method.

[0010] An example of a computer-implemented method for creating a training dataset for training CFD functions is described. This computer-implemented method includes:

[0011] a) Receive a set of 3D virtual environments and, for each 3D virtual environment, a specified location defined within the 3D virtual environment;

[0012] b) Receive a 3D virtual motion system and a set of poses defined for the 3D virtual motion system, typically a set of 3D poses;

[0013] c) For each 3D virtual environment, receive the target task to be performed when the 3D virtual motion system is located at the specified position;

[0014] d) Automatically select one of the 3D virtual environments and automatically perform the following steps on the selected 3D virtual environment:

[0015] d1) Automatically select a pose from the set of poses;

[0016] d2) Automatically perform the following steps for the selected pose:

[0017] i) Create a dataset that includes identifiers of the selected 3D virtual environment, identifiers of the specified locations, identifiers of the target task, and identifiers of the selected poses;

[0018] ii) Place the 3D virtual motion system, characterized by the selected pose, at a predetermined location defined within the selected 3D virtual environment to create a 3D system comprising the 3D virtual environment and the 3D virtual motion system placed at the predetermined location. Specifically, placement refers to positioning the 3D virtual motion system in the same reference frame as the 3D virtual environment;

[0019] iii) Determine whether the 3D virtual motion system, characterized by the selected pose, can perform the target task when placed at the specified position, and

[0020] -If so, proceed to the next step iv).

[0021] - Otherwise, label the created dataset with the label represented by the first value, store the label associated with the dataset in the database, or assign the label to the dataset, then select another pose in the pose set that has not yet been selected (if any), and if such a pose exists, repeat step d2) for the newly selected pose, otherwise proceed to step e);

[0022] iv) Determine whether a collision occurs between the 3D virtual motion system represented by the selected pose and the 3D virtual environment, and if a collision occurs, label the dataset with the label represented by the first value; otherwise, i.e., if no collision occurs, label the dataset with the label represented by the second value.

[0023] v) Store the labels assigned to the dataset in the database;

[0024] vi) Select another pose (if any) from the pose set that has not yet been selected, and repeat step d2) for the newly selected pose; otherwise, proceed to step e).

[0025] e) Repeat step d) for all 3D virtual environments in the 3D virtual environment set to create a training dataset, which includes:

[0026] - As training input data: a 3D virtual environment and its respective defined position and target task, the set of poses, optionally, a 3D virtual motion system; and

[0027] - As training output data: a set of labels, wherein each label is characterized by the second value or the first value, and wherein each label is associated with a dataset or each label is assigned to the dataset, the dataset including identifiers of 3D virtual environments, identifiers of specified locations, identifiers of target tasks, and identifiers of poses to which the label values ​​have been assigned.

[0028] This invention also provides an example of a computer implementation method for providing a trained CFD function. The computer implementation method includes:

[0029] a) Receive a training dataset through a first interface, wherein the training dataset includes training input data and training output data, wherein the training input data includes:

[0030] - A set of 3D virtual environments, and a specified location and target task for each 3D virtual environment in the set.

[0031] - A set of 3D poses defined for the 3D virtual motion system, and optionally, the 3D virtual motion system itself; and,

[0032] The training output data includes:

[0033] - A set of labels, wherein each label is characterized by a second value or a first value, and wherein each label is associated with a dataset or each label is assigned to the dataset, the dataset including identifiers of 3D virtual environments, identifiers of specified locations, identifiers of target tasks, and identifiers of poses to which the label values ​​have been assigned;

[0034] b) Automatically select one of the 3D virtual environments and automatically perform the following steps for the selected 3D virtual environment:

[0035] b1) Obtain at least one depth image of the 3D virtual environment from the specified location;

[0036] b2) For each of the acquired depth images, the depth image, the target task, and the specified location are used as inputs to a dimensionality reduction algorithm configured to provide a 2D image as output;

[0037] b3) For each of the obtained 2D images, repeat steps (i) and (ii) to train a convolutional neural network for the CFD function, hereinafter referred to as a CNN, until each pose in the set of 3D poses has been selected once according to step i):

[0038] (i) Automatically select a pose that has not yet been selected from the pose set;

[0039] (ii) Training the CNN, wherein 2D images and selected poses (preferably IDs of the selected poses) are used as input training data for the CNN, and label values ​​associated with a dataset including identifiers of the selected 3D virtual environment, identifiers of the specified location, identifiers of the target task, and identifiers of the selected poses are used as output training data. Thus, the CNN receives the input training data and the output training data for its training. Given a set of poses for a motion system, training the CNN will enable the latter to automatically associate a subset of the poses with 2D images encoding the 3D virtual environment, the specified location, and the target task, wherein the subset of poses includes all poses in the given set that enable the motion system to perform the target task from a specified location within the 3D virtual environment;

[0040] c) Repeat step b) until all 3D virtual environments in the 3D virtual environment set have been processed, that is, all 3D virtual environments have been selected once;

[0041] d) Provide a second interface to the trained CFD function. The second interface can be the same as the first interface.

[0042] The features and technical advantages of this disclosure have been outlined quite extensively above, enabling those skilled in the art to better understand the following detailed descriptions. Additional features and advantages of this disclosure that form the subject matter of the claims will be described below. Those skilled in the art will understand that, for the same purposes of performing this disclosure, they can readily use the disclosed concepts and detailed embodiments as a basis for modifying or designing other structures. Those skilled in the art will also recognize that such equivalent constructions do not depart from the spirit and scope of the broadest form of this disclosure.

[0043] Before proceeding with the detailed embodiments below, it may be advantageous to define certain words or phrases used throughout this patent document: the terms “comprising” and “including” and their derivatives mean including but not limited to; the term “or” is inclusive, meaning and / or; the phrases “associated with” and “associated with” and their derivatives may mean including, comprising, interconnected with, containing, contained within, connected to or connected to, coupled to or coupled to, communicable with, cooperating with, interleaved, juxtaposed, proximate, bonded to or bonded to, having, having the characteristics of, etc.; and the term “controller” means any device, system or part thereof that controls at least one operation, whether such device is implemented in hardware, firmware, software or a combination of at least two of these. It should be noted that the functionality associated with any particular controller can be centralized or distributed, local or remote. Definitions of certain words and phrases are provided throughout this patent document, and those skilled in the art will understand that such definitions apply in many (if not most) cases to the prior and future use of such defined words and phrases. While some terms may encompass a wide variety of implementations, the appended claims may expressly limit these terms to a particular implementation. Attached Figure Description

[0044] To gain a more complete understanding of this disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals denote like objects, and in the drawings:

[0045] Figure 1 A block diagram of a data processing system that can be implemented in practice is shown.

[0046] Figure 2 A flowchart for predicting the motion trajectory of a robot according to the disclosed embodiment is shown.

[0047] Figure 3 A configuration table according to the disclosed implementation is shown.

[0048] Figure 4An example of a workstation depth image is presented. Detailed Implementation

[0049] In this patent document, the following are discussed Figures 1 to 4 The various embodiments used to describe the principles of this disclosure are merely illustrative and should not be construed as limiting the scope of this disclosure in any way. Those skilled in the art will understand that the principles of this disclosure can be implemented with any suitably arranged device. Many of the innovative teachings of this application will be described with reference to exemplary, non-limiting embodiments.

[0050] Prior techniques for collision-free pose prediction of motion systems have several drawbacks. The embodiments disclosed herein offer numerous technical benefits, including but not limited to the following examples. Specifically, this invention proposes using artificial intelligence (AI) to determine poses that are highly likely to be collision-free given a given graphics size, graphics task (range) requirements, and geometric terrain. Due to the novel concept according to this invention, rapid collision-free pose prediction can be performed in cluttered environments of complex motion systems such as humans and robots.

[0051] Although the presented implementation focuses on humanoid modeling, the proposed method can also be used to solve any similar problems of any complex motion system, including robots.

[0052] Figure 1 A block diagram of a data processing system 100 is shown, wherein implementations may be carried out as, for example, a PDM system specifically configured by software or otherwise to perform the processing described herein, and particularly as each of a plurality of interconnected and communicating systems as described herein. The data processing system 100 shown may include a processor 102 connected to a secondary cache / bridge 104, which in turn is connected to a local system bus 106. The local system bus 106 may be, for example, a Peripheral Component Interconnect (PCI) architecture bus. In the example shown, main memory 108 and a graphics adapter 110 are also connected to the local system bus. The graphics adapter 110 may be connected to a display 111.

[0053] Other peripheral devices, such as LAN / WAN / wireless (e.g., WiFi) adapter 112, can also be connected to the local system bus 106. An expansion bus interface 114 connects the local system bus 106 to the input / output (I / O) bus 116. The I / O bus 116 connects to a keyboard / mouse adapter 118, a disk controller 120, and an I / O adapter 122. The disk controller 120 can be connected to a storage device 126, which can be any suitable machine-usable or machine-readable storage medium, including but not limited to non-volatile hard-coded media such as read-only memory (ROM) or electrically erasable programmable read-only memory (EEPROM), magnetic tape storage devices, and user-recordable media such as floppy disks, hard disk drives, and compact disc read-only memory (CD-ROM) or digital versatile disc (DVD), as well as other known optical, electrical, or magnetic storage devices.

[0054] In the example shown, an audio adapter 124 is also connected to the I / O bus 116. A speaker (not shown) can be connected to the audio adapter 124 for playing sound. A keyboard / mouse adapter 118 provides connectivity for pointing devices (not shown), such as a mouse, trackball, tracking pointer, touchscreen, etc.

[0055] Those skilled in the art will understand that Figure 1 The hardware shown may vary for a particular implementation. For example, other peripheral devices, such as optical disc drives, may be used in addition to, or instead of, the hardware shown. The examples shown are provided for illustrative purposes only and are not intended to impose architectural limitations on the present disclosure.

[0056] Data processing systems according to embodiments of this disclosure may include an operating system employing a graphical user interface (GUI). The operating system allows multiple display windows to be presented simultaneously in the GUI, each providing an interface to a different application or to different instances of the same application. A user can manipulate a cursor in the GUI using a pointing device. The cursor position can be changed and / or events, such as clicking a mouse button, can be generated to induce a desired response.

[0057] With appropriate modifications, one of various commercial operating systems can be used, such as Microsoft Windows, a product of Microsoft Corporation located in Redmond, Washington. TM Version. As described, modifying or creating an operating system based on this disclosure.

[0058] The LAN / WAN / wireless adapter 112 can connect to network 130 (not part of data processing system 100), which can be any public or private data processing system network or a combination of networks known to those skilled in the art, including the Internet. Data processing system 100 can communicate with server system 140 via network 130, which is also not part of data processing system 100 but can be implemented as, for example, a standalone data processing system 100.

[0059] As used herein, the term "motion system" refers to any real system characterized by motion in its surrounding environment and whose posture changing over time must be determined. It can be a robot, such as an industrial robot, or any other type of kinematic machine, or a living organism, such as a human. The present invention is interested in determining which posture of the motion system at a given time does not collide with its surrounding environment. The surrounding environment at said given time is a known parameter, and therefore a collision-free posture must be determined. The surrounding environment is a real environment modeled in a 3D manner used to create 3D virtual environments, such as a workstation, a cleanroom, etc.

[0060] For this purpose, the present invention proposes to transform the kinematic problem of finding collision-free poses into an image classification problem, applicable to solutions using image classification techniques known in the art. Figure 2 A flowchart 200 is shown for a method for predicting a collision-free posture for a motion system. Such a method can be, for example, as described above. Figure 1 The system 100 is used to execute the process, but the "system" in the following process can be any device configured to execute the process as described. Now, in conjunction with... Figure 2 The method according to the invention will be described in more detail below.

[0061] At step 201, the system 100 according to the invention receives a 3D virtual environment (i.e., a 3D representation of the surrounding environment). Such a surrounding environment can be as follows: Figure 4 The workstation shown is typically used to perform a target task by a motion system, such as an operator or robot, at the workstation, and requires determining a posture for the operator or robot that avoids collisions with any elements of the surrounding environment, such as the workstation.

[0062] At step 202, system 100 receives a 3D virtual motion system (i.e., a 3D representation of the motion system) and a set of poses defined for the 3D virtual motion system. For example, each pose in the pose set represents the actual pose of the motion system, taking into account each degree of freedom of each joint of the motion system. Each pose can be identified by an ID configured to enable system 100 to easily identify or retrieve the pose.

[0063] At step 203, system 100 receives a target task to be performed by the 3D virtual motion system relative to a 3D virtual environment. Therefore, in the virtual world, this target task represents a target task that the motion system must perform relative to its surrounding environment in the real world. The target task is preferably a task defined by the user for the motion system relative to its surrounding environment, and thus also defines the task to be performed in the latter's 3D representation.

[0064] At step 204, system 100 also receives a defined position (also called a defined location) within the 3D virtual environment. The defined position defines the positioning of the motion system within and relative to its surrounding environment, or the positioning of the motion system within and relative to its surrounding environment. In other words, the defined position enables the system to position the 3D virtual motion system within the 3D virtual environment because it defines the positioning of the motion system in its surrounding environment, such as the positioning of a reference point of the motion system in its surrounding environment. Specifically, the defined position defines the position and orientation of the 3D virtual motion system relative to the 3D virtual environment, such that it mimics the real behavior of a real motion system in its surrounding environment. Typically, the defined position is configured to define the position and orientation of the 3D virtual motion system's reference frame relative to the reference frame of the 3D virtual environment. Subsequently, both the 3D virtual motion system and the 3D virtual environment can be modeled in the same reference frame, where the reference point can position the 3D virtual motion system relative to the 3D virtual environment.

[0065] Steps 201 to 204 can occur simultaneously, sequentially, or in any other order. The purpose of steps 201 and 202 is to provide the system 100 according to the invention with information about the surrounding environment in which the target task must be performed, and information about the motion system that must perform the target task. The goal of steps 203 and 204 is to provide the system 100 with information about the target task to be performed and from which position the motion system should perform the target task. Specifically, the target task can be configured to define an action to be performed by the motion system, wherein the action depends on the surrounding environment. The action may require physical interaction with the surrounding environment; the target task defines, for example, the localization of at least one target in the surrounding environment reached by at least a part of the motion system (e.g., by the end effector of a human or robotic arm), or it may require non-physical interaction with the surrounding environment, such as non-contact measurements performed by the motion system on elements of the surrounding environment, or image acquisition of objects in the surrounding environment, and image acquisition performed by the motion system's camera, etc. For example, the target task defines a location, surface, volume, or element in the surrounding environment, and thus defines a location, surface, volume, or element in the 3D virtual environment that will be reached by a part of the motion system, or more generally, when located at the specified location, the motion system must interact with or perform an action relative to that location, surface, volume, or element.

[0066] Considering the degrees of freedom of each moving part or joint of the 3D virtual motion system, the pose set according to the invention can be dynamically generated by system 100, for example, from the geometric configuration of the 3D virtual motion system. System 100 according to the invention can also receive or load pose sets from a database. For example, system 100 can be configured to use a pose library that can be stored in a database or memory 108 of system 100 according to the invention. The library can be configured to include several 3D virtual motion systems, and for each of them, a pose set specifically defined for the 3D virtual motion system under consideration. For example, once a motion system for which a collision-free pose must be found is input to system 100 according to the invention, the latter can be configured to automatically download or retrieve the 3D virtual motion system and the corresponding pose set from the database. Thus, the correct pose set (i.e., the pose set corresponding to the motion system input to system 100 according to the invention) can be received or loaded using the library, but pose sets can also be automatically generated by the system according to the invention. For example, the 3D virtual motion system can be automatically divided into n subsystems, for example, based on the presence of joints in the motion system, using a partitioning criterion. For each of the subsystems, the data processing system 100 is configured to automatically determine a finite set of subsystem postures by modeling, for example, the dynamics of a mechanical joint. The posture set is then obtained by determining all possible combinations of subsystem postures. For this purpose, the system according to the invention can be used or implemented as follows: Figure 3 The configuration table shown.

[0067] Figure 3 A configuration table is shown that can be used or implemented by the system 100 according to the invention, which is used to determine all possible combinations of subsystem poses to create the pose set. The subsystem poses are preferably 3D poses. Figure 3For illustrative purposes, the human musculoskeletal system is used as an example. The musculoskeletal system, i.e., the human body, is divided into four subsystems: torso 301, left arm 302, right arm 303, and legs 304. For each subsystem, a finite set of subsystem postures is defined and stored, for example, in a database, or automatically generated by the system 100 according to the invention. Each row of the configuration table provides different combinations of torso posture, left arm posture, right arm posture, and leg posture, thereby producing a full-body posture defined by the combination of said subsystem postures, and associates ID 305 with said produced full-body posture. In other words, the present invention proposes to decompose a 3D virtual musculoskeletal system into n subsystems, defining a finite set of 3D subsystem postures for each of said n subsystems, wherein each posture in the set of postures defined for the 3D virtual musculoskeletal system is obtained by combining 3D subsystem postures, such that each combination of 3D subsystem postures produces a different 3D virtual musculoskeletal system posture (i.e., the "overall" system posture), and the different 3D virtual musculoskeletal system postures can be identified using said ID. Other IDs can be used to enable the identification of 3D virtual environments, specified locations, and target tasks, respectively. For example, each 3D virtual environment can be associated with one ID, each specified location with another ID, and each target task with yet another ID.

[0068] A 3D virtual environment and a 3D virtual motion system positioned at the predetermined location (i.e., the predetermined location represented in the same reference frame as the 3D virtual environment) together form a 3D system. This invention is capable of predicting the posture of the 3D virtual motion system that may collide with the 3D virtual environment.

[0069] At step 205, and for this purpose, a CFD function is applied to an input dataset including a 3D virtual environment, a target task, a specified position, and a set of poses. More precisely, the CFD function receives the 3D virtual environment, the specified position, the target task, the set of poses, and an optional 3D virtual motion system as input, and provides a set of collision-free poses as output, thereby enabling the 3D virtual motion system to perform the target task when located at the specified position, and thus enabling the "real" motion system to perform the target task when located at the specified position in the "real" surrounding environment. Specifically, the collision-free poses in the collision-free pose set are sorted from most likely to least likely to be collision-free with the surrounding environment. From the obtained collision-free poses, the system can automatically select the most suitable pose, such as the most likely collision-free pose, to control the motion system, for example, to control a real robot that must perform the target task.

[0070] The CFD function according to the present invention is a function trained by machine learning. The CFD function has been specifically trained for a motion system. The CFD function is configured to transform the kinematic problem of collision-free poses associated with a 3D system into an image classification problem. Preferably, the machine learning algorithm used to train the CFD function employs a CNN. Specifically, the CFD function is configured to automatically acquire one or more depth images of a 3D virtual environment. Each depth image provides information about the surrounding terrain. Figure 4 The left side shows the surrounding environment of workstation 401, and the right side provides an example of a depth image 402 of said workstation 401. According to the invention, each depth image is acquired from the predetermined location acting as a viewpoint, i.e., as if the imaging system configured to acquire the depth image were located at the predetermined location. Therefore, each depth image provides a depth map of the 3D virtual environment around the predetermined location, and thus provides information related to the distance separating the predetermined location from the object surface of the 3D virtual environment, i.e., depth data. Depth images can be acquired from the predetermined location along different acquisition directions, such that, for example, depth data of the entire environment surrounding the predetermined location is acquired. Specifically, a CFD function is configured to automatically create a set of depth images for the 3D virtual environment, wherein each depth image in the set has been acquired from the predetermined location but according to a different acquisition direction, i.e., such that each depth image represents a different part of the 3D virtual environment, as seen. Preferably and optionally, a top-view image can be acquired from an image acquisition location located above the predetermined location to capture bin depth. Preferably, the depth image data is normalized to the height or size of the 3D virtual motion system.

[0071] According to the present invention, each depth image represents multiple 3D datasets (u, v, z), where u and v are the coordinates of bins in the depth image, and z is depth information. Furthermore, the pose of each 3D virtual motion system is defined by a 3D joint dataset (e.g., more than 50 joints for the human body), which includes a high-dimensional data space, i.e., a data space exceeding 2 or 3 dimensions. A CFD function is configured to automatically convert the depth image (i.e., depth image data, a specified location, such as 3D localization defined in a 3D virtual environment, and a target task, such as another 3D localization defined in a 3D virtual environment) into a well-organized image form, i.e., a matrix M of size m×n, i.e., a 2D image encoding the depth image, target task, and specified location. For this purpose, the CFD function uses a dimensionality reduction algorithm. The dimensionality reduction algorithm is configured to organize the multidimensional data from the depth image, specified location, and target task into a 2D matrix (image) suitable for CNN techniques. Preferably, the DeepInsight method described by Alok Sharma et al. (“DeepInsight: A Method for Converting Non-Image Data into Images of Convolutional Neural Network Architectures”, Scientific Reports 9, 11399 (2019)) is used by a CFD function to create the 2D image. For each depth image, specified location, and target task received as input, the DeepInsight method is able to output 2D image encoding information contained in the received input (depth image, specified location, and target task). Of course, the CFD function can use any other dimensionality reduction algorithm capable of converting 3D image data (i.e., multiple sets of multidimensional data) into a 2D plane or matrix (i.e., an organized 2D image form) suitable for machine learning algorithms using CNNs. Preferably, the dimensionality reduction algorithm can be configured to additionally utilize the bin depth information to create the 2D image.

[0072] For each of the acquired depth images, the dimensionality reduction algorithm outputs a 2D image. The CFD function then uses the acquired 2D image as input to a CNN. The CNN is configured to automatically determine a collision-free pose set from the received 2D images, the collision-free pose set comprising each pose that enables the motion system to perform a target task from a specified position. Preferably, after receiving the 2D image as input, the CNN automatically outputs a list containing the ID, i.e., an identifier, of each pose in the received pose set, which is the collision-free pose that enables the motion system to perform the target task from a specified position. Therefore, this collision-free pose set is automatically created by the CFD function. Preferably, if there are several depth images, and therefore multiple 2D images, the CFD function will output several sets of output data, which are collision-free pose sets (basically, one collision-free pose set is created for each 2D image). In this context, to determine which poses ultimately qualify as collision-free poses, the CFD function is configured to determine the "final" set of output data, which is the intersection of the obtained sets of output data. This final set includes all poses that, when considering several 2D images, belong to all output datasets generated by the CFD function. Therefore, if a pose is included in every set of collision-free poses in the output created from the 2D images received as input, then that pose is considered collision-free by the system. If a pose belongs to the set of collision-free poses when using a first 2D image as input, but does not belong to another set of collision-free poses when using a second 2D image as input, then that pose is automatically rejected from the "final" set of collision-free poses by the system.

[0073] At step 206, the data processing system 100 automatically selects at least one collision-free posture from the obtained set of collision-free postures. For example, the system can be configured to automatically classify the set of collision-free postures according to at least one predefined criterion, such as ergonomic criteria, performance criteria, speed criteria, energy consumption criteria, or combinations of different criteria.

[0074] At step 207, system 100 is configured to display a 3D virtual motion system within a 3D virtual environment, wherein the posture of the displayed 3D virtual motion system is a selected collision-free posture. The system's display 111 can be used for this purpose. System 100 can then automatically determine the collision-free motion of the motion system based on the selected collision-free posture, and can then control the motion system based on the determined collision-free motion, thus ensuring the safe movement of the motion system in its surrounding environment.

[0075] To train a CFD function, this invention proposes the automatic creation of a training dataset. For this purpose, the data processing system 100 according to the invention can also be configured to receive a set of 3D virtual environments and a set of 3D poses defined for a 3D virtual motion system, optionally, the 3D virtual motion system itself. For each set of 3D virtual environments, one or more predefined positions can be defined, and for each predefined position, one or more target tasks to be performed from that position are defined. All this information is used as input to create the training dataset.

[0076] Then system 100 performs the following steps to create the training dataset:

[0077] A) Automatically select one of the 3D virtual environments, and automatically execute the following steps for the selected 3D virtual environment, a specified location within the 3D virtual environment, and the target task to be performed from that specified location:

[0078] A1) Automatically select a pose from the pose set defined for the 3D motion system, and then...

[0079] A2) Automatically perform the following steps for the selected posture:

[0080] A21) Create a dataset that includes identifiers for the selected 3D virtual environment, a specified location, a target task, and a selected pose. The dataset may include, for example, distinct IDs associated with the 3D virtual environment, the specified location, the target task, and the pose, respectively.

[0081] A22) A 3D virtual motion system, characterized by a selected pose, is placed at a defined location within a selected 3D virtual environment to create a 3D system comprising the 3D virtual environment and the 3D virtual motion system placed at the defined location. "Placement" refers to representing the 3D virtual motion system and the 3D virtual environment in the same reference frame, wherein the position and orientation of the 3D virtual motion system relative to the 3D virtual environment are determined with reference to information included in the defined location. Optionally, the 3D virtual motion system or the 3D virtual environment may be normalized to maintain a predefined scale between the motion system and its surroundings.

[0082] A23) Determine whether the 3D virtual motion system characterized by the selected pose can perform the target task when placed at the specified position, and if so, proceed to the next step A24); otherwise, automatically label the created dataset with a label characterized by a first value and store the label associated with the dataset in a database, the label being configured to associate the first value with the dataset. Then, the system selects another pose in the pose set that has not yet been selected (if any), and if such a pose exists, repeat step A2) for the newly selected pose; otherwise, if it does not exist, i.e., if all poses have been selected once, proceed automatically to step B).

[0083] A24) Determine whether a collision has occurred between the 3D virtual motion system, characterized by the selected pose, and the 3D virtual environment. For this purpose, system 100 may use a collision detection engine known in the art. Such a collision engine typically uses a collision detection algorithm that runs on 3D objects;

[0084] (A25) The created dataset is labeled with the labels, wherein if a collision occurs, the label is characterized by a first value, and if no collision occurs, the label is characterized by a second value. In other words, the label values ​​are associated with each dataset including the identifier of the selected 3D virtual environment, the identifier of the specified location, the identifier of the target task, and the identifier of the pose. Therefore, the label values ​​are assigned to specific combinations of elements, which are the 3D virtual environment, the pose of the 3D virtual motion system in the 3D virtual environment, the specified location of the 3D virtual motion system, and the target task to be performed from the specified location in the 3D virtual environment;

[0085] (A26) Labels associated with the created dataset are stored in a database. Specifically, the label values ​​associated with the dataset include an identifier of the selected 3D virtual environment, an identifier of the specified location, an identifier of the target task within the 3D virtual environment, and an identifier of the pose selected to perform the target task from the specified location. In other words, each pose or pose ID can be associated with multiple label values, depending on which dataset it belongs to. In effect, each label value is defined for a specific target task to be performed from a specific specified location within a specific 3D virtual environment. For example, if several target tasks and several specified locations are defined for the same 3D virtual environment, different label values ​​can be assigned to poses based on the selected specified location and target task.

[0086] A27) Select another pose from the pose set that has not yet been selected (if any), and repeat step A2) for the newly selected pose; otherwise, if all poses have already been selected once, proceed automatically to step B). In other words, for the 3D virtual environment, specified position, and target task selected according to step A), poses are continuously selected, and their potential collisions with the surrounding environment are determined or tested until all poses in the set have been tested relative to the selected 3D virtual environment, specified position, and target task.

[0087] B) Repeat steps A1 and A2 for all other target tasks (if any) to be performed from the specified location, and once completed, repeat steps A1 and A2 for all other specified locations and (one or more) target tasks to be performed from the considered specified location in the selected 3D virtual environment, and once completed, repeat step A for another 3D virtual environment, until all 3D virtual environments in the set have been processed, thereby enabling the creation of a training dataset comprising:

[0088] - As training input data: a 3D virtual environment and its(one or more) respective prescribed positions and target tasks, a set of 3D poses, and an optional 3D virtual motion system; and

[0089] - As training output data: a set of labels, wherein each label is characterized by either the first value or the second value, and wherein each label is associated with or assigned to a dataset, the dataset including identifiers of a 3D virtual environment, identifiers of a specified location, identifiers of a target task, and identifiers of poses for which the label values ​​have been assigned. In other words, the training output is associated with each combination of the 3D virtual environment, the specified location within the 3D virtual environment, the target task to be performed from the specified location, and the pose of the 3D virtual motion system used to perform the target task from the specified location; that is, the label value is the first label value if a collision exists, and the label value is the second label value if no collision exists.

[0090] The obtained training dataset can then be used to train a CFD function such that the trained CFD function can be used for a motion system relative to any surrounding environment and for a motion system relative to any specified position and a target task to be performed from said specified position. For example, the training dataset can be received using a first interface of the system according to the invention. The CFD function is then trained by using the training input data as input and the training output data as output. Based on the training input data, for each 3D virtual environment, the system 100 first acquires one or more depth images, wherein the depth images are acquired from each specified position in the 3D virtual environment. Preferably, the depth image data of each of the acquired depth images is normalized to the height or size of the 3D virtual motion system to create a normalized depth image, thereby making the learning insensitive to the size of a particular motion system and thus enabling the CFD function to work with any motion system characterized by similar or identical shapes, although the sizes may differ. The normalized depth image can then be used in subsequent steps instead of the "initially acquired" depth image. Then, for each of the acquired depth images, system 100 uses the acquired depth image, the target task, and the specified location as input to a dimensionality reduction algorithm (e.g., the DeepInsight algorithm), which is configured to provide a 2D image encoding the received input as output. System 100 then uses all created 2D images and all poses as input to a CNN for the CFD function, wherein, for each 2D image, the system iteratively selects a pose from the pose set until all poses have been selected once with the 2D image, and uses the identifier of the selected pose or the latter, such as its ID, and the 2D image as training input data for the CNN, wherein label values ​​associated with a dataset including the identifier of the 3D virtual environment, the identifier of the specified location, and the identifier of the target task encoded in the 2D image, as well as the identifier of the selected pose, are used as output training data for CNN training. At the end of training, the trained CFD function can provide a second interface and is preferably stored in the memory or database of system 100 according to the invention.

[0091] Following the training, the end user can simply place the motion system, such as a robot or virtual human, in a new surrounding environment, such as a designated location in front of a workstation, define a target task for the robot or virtual human, and then activate the trained CFD function to identify which postures are most likely to be collision-free. Advantageously, the trained CFD function will work with any morphologically similar motion system, regardless of the size of the motion system, as long as the depth image is normalized to the size of the motion system. Then, among the most likely collision-free postures, system 100 can automatically determine which is most suitable, for example, based on minimum effort and / or maximum comfort and / or other performance metrics. The motion system with the most suitable posture is then applied.

[0092] This invention transforms the multidimensional pose prediction problem for 3D virtual motion systems into a classification problem, enabling techniques such as CNNs to identify collision-free poses. Using depth images to acquire 3D geometric terrain information of the 3D virtual environment avoids the need for voxelization of the environment surrounding the motion system. The CFD function advantageously provides a short list of collision-free poses, allowing the application of motion system performance models or suitability criteria to select the optimal pose for performing a target task from a given location.

[0093] Of course, those skilled in the art will recognize that, unless explicitly indicated or required by the sequence of operations, some steps in the process described above may be omitted, performed simultaneously or sequentially, or performed in a different order.

[0094] Those skilled in the art will recognize that, for simplicity and clarity, the complete structure and operation of all data processing systems applicable to this disclosure are not shown or described herein. Instead, only data processing systems that are unique to this disclosure or necessary for understanding this disclosure are shown and described. The remaining construction and operation of data processing system 100 may conform to any of the various current implementations and practices known in the art.

[0095] It is important to note that although this disclosure is described in the context of a fully functional system, those skilled in the art will understand that at least a portion of this disclosure can be distributed in the form of instructions contained in any form of machine-usable, computer-usable, or computer-readable medium, and this disclosure applies equally regardless of the specific type of instruction or signal-bearing medium or storage medium used to actually perform the distribution. Examples of machine-usable / computer-usable / readable media include: non-volatile, hard-coded media, such as read-only memory (ROM), or erasable electrically programmable read-only memory (EEPROM), and user-recordable media, such as floppy disks, hard disk drives, and compact disc read-only memory (CD-ROM) or digital versatile discs (DVDs).

[0096] Although exemplary embodiments of this disclosure have been described in detail, those skilled in the art will understand that various changes, substitutions, variations, and modifications disclosed herein may be made without departing from the spirit and scope of the broadest form of this disclosure.

[0097] The descriptions in this application should not be construed as implying that any particular element, step, or function is a necessary element that must be included within the scope of the claims: the scope of the subject matter of the patent is defined only by the granted claims.

Claims

1. A method for predicting the collision-free posture of a motion system surrounded by an environment by a data processing system (100), the method comprising the steps of: a) Receive (201) a 3D virtual environment as a 3D representation of the surrounding environment; b) Receive (202) a 3D virtual motion system as a 3D representation of the motion system and a set of 3D poses defined for the 3D virtual motion system; c) Receive (203) the target task to be performed by the 3D virtual motion system relative to the 3D virtual environment; d) Receive (204) a specified position within the 3D virtual environment, the specified position defining the positioning within the 3D virtual motion system that must be placed in the 3D virtual environment; e) A collision-free detection function, or CFD function, trained using a machine learning algorithm employing a convolutional neural network (CNN) is applied (205) to the input dataset, wherein the input dataset includes the 3D virtual environment, the target task, the specified location, and the set of 3D poses, wherein the CFD function is configured to generate an output dataset, which is a set of collision-free poses comprising all 3D poses from the received set of 3D poses that enable the 3D virtual motion system to perform the target task when located at the specified location, wherein the CFD function is configured to automatically acquire one or more depth images of the 3D virtual environment to target the 3D virtual environment. The kinematic problem of finding collision-free poses for a virtual motion system is transformed into an image classification problem for recognizing collision-free poses. The CFD function uses a dimensionality reduction algorithm to automatically convert the depth image, the specified location, and the target task into a 2D image encoding the depth image, the target task, and the specified location. The CNN is applied to the 2D image representing the 3D virtual environment, the specified location, and the target task. The CNN automatically outputs from the received 2D image a set of collision-free poses, including all poses from the 3D pose set that enable the motion system to perform the target task when located at the specified location. f) Select (206) at least one collision-free pose from the set of collision-free poses; g) Display (207) the 3D virtual motion system characterized by the selected collision-free pose in the 3D virtual environment, and use the selected collision-free pose to control the motion of the motion system.

2. The method according to claim 1, wherein, The target task is configured to define an action to be performed by the motion system relative to the surrounding environment, and the action depends on the surrounding environment.

3. The method according to claim 1 or 2, wherein, Selecting (206) at least one collision-free pose includes: automatically classifying the collision-free poses in the set of collision-free poses according to at least one predefined criterion.

4. The method according to claim 1 or 2, wherein, To provide a trained CFD function, the method includes: a) Receive a training dataset using a first interface, wherein the training dataset includes training input data and training output data, wherein the training input data includes: - A set of 3D virtual environments, and for each 3D virtual environment in the set, a specified location and target task. - A set of poses defined for 3D virtual motion systems; and, The training output data includes: - A set of labels, wherein each label is characterized by a second value or a first value, and wherein each label is associated with a dataset, or each label is assigned to the dataset, the dataset including identifiers of the 3D virtual environment, identifiers of the specified location, identifiers of the target task, and identifiers of poses for which label values ​​have been assigned; b) Automatically select one of the 3D virtual environments and automatically perform the following steps for the selected 3D virtual environment: b1) Obtain a depth image of the 3D virtual environment from the specified location; b2) Using the depth image, the target task, and the specified location as input to a dimensionality reduction algorithm configured to provide a 2D image as output; b3) Repeat steps (i) and (ii) to train the CNN of the CFD function until each pose in the pose set has been selected once: (i) Automatically select a pose from the set of poses that has not yet been selected; (ii) Training the CNN, wherein the 2D image and the selected pose are used as input training data for the CNN, and label values ​​associated with a dataset including the identifiers of the 3D virtual environment, the identifiers of the specified location, the identifiers of the target task, and the identifiers of the selected pose are used as output training data; c) Repeat step b) until all 3D virtual environments in the set of 3D virtual environments have been processed; d) Provide a second interface to the trained CFD function.

5. The method according to claim 4, wherein, To create the training dataset for training the CFD function, the method includes: a) Receive a set of 3D virtual environments and, for each 3D virtual environment, a specified location defined in the 3D virtual environment; b) Receive a 3D virtual motion system and a set of 3D poses defined for the 3D virtual motion system; c) For each 3D virtual environment, receive the target task to be performed by the 3D virtual motion system relative to the 3D virtual environment; d) Automatically select one of the 3D virtual environments and automatically perform the following steps on the selected 3D virtual environment: d1) Automatically select a pose from the set of poses; d2) Automatically perform the following steps for the selected pose: i) Create a dataset that includes identifiers of the selected 3D virtual environment, identifiers of the specified location, identifiers of the target task, and identifiers of the selected pose; ii) Place the 3D virtual motion system characterized by the selected pose at a defined location within the selected 3D virtual environment to create a 3D system including the 3D virtual environment and the 3D virtual motion system placed at the defined location; iii) Determine whether the 3D virtual motion system represented by the selected pose can perform the target task when placed at the specified location, and if so, proceed to the next step iv); otherwise, label the created dataset with a label represented by the first value, store the label associated with the dataset in the database, and then select another pose from the pose set that has not yet been selected, if any, and if such a pose exists, repeat step d2); otherwise, proceed to step e). iv) Determine whether a collision occurs between the 3D virtual motion system, characterized by the selected pose, and the 3D virtual environment; v) Label the created dataset with the label, wherein if a collision occurs, the label is characterized by the first value, otherwise, if no collision occurs, the label is characterized by the second value. vi) Store the labels of the created dataset in the database; vii) Select another pose from the pose set that has not yet been selected, and repeat step d2 for the newly selected pose; otherwise, if all poses have been selected, proceed to step e). e) Repeat step d) until all 3D virtual environments in the set of 3D virtual environments are selected to create a training dataset, which includes: - As training input data: the 3D virtual environment and its respective defined position and target task, and the set of 3D poses; and - As training output data: a set of labels, wherein each label is characterized by the second value or the first value, and wherein each label is associated with a dataset, or each label is assigned to the dataset, which includes an identifier of the 3D virtual environment, an identifier of the specified location, an identifier of the target task, and an identifier of the pose to which the label value has been assigned.

6. A data processing system (100), comprising: Processor (102); as well as Accessible memory (108), the data processing system is specifically configured to: a) Receive (201) a 3D representation of the surrounding environment, i.e., a 3D virtual environment; b) Receive (202) a 3D representation of the motion system, i.e., a 3D virtual motion system and a set of 3D poses defined for the 3D virtual motion system; c) Receive (203) the target task to be performed by the motion system relative to the surrounding environment; d) Receive (204) a specified position within the 3D virtual environment, the specified position defining the positioning within the 3D virtual motion system that must be placed in the 3D virtual environment; e) A collision-free detection function, or CFD function, trained using a machine learning algorithm employing a convolutional neural network (CNN) is applied (205) to the input dataset, wherein the input dataset includes the 3D virtual environment, the target task, the specified location, and the set of poses, wherein the CFD function is configured to generate an output dataset, which is a subset of the set of 3D poses that enable the motion system to perform the target task when located at the specified location, wherein the CFD function is configured to automatically acquire one or more depth images of the 3D virtual environment to perform a search for the 3D virtual motion system. The kinematic problem of finding collision-free poses is transformed into an image classification problem for recognizing collision-free poses. The CFD function uses a dimensionality reduction algorithm to automatically convert the depth image, the specified location, and the target task into a 2D image encoding the depth image, the target task, and the specified location. The CNN is applied to the 2D image representing the 3D virtual environment, the specified location, and the target task. The CNN automatically outputs from the received 2D image a set of collision-free poses, including all poses from the 3D pose set that enable the motion system to perform the target task when located at the specified location. f) Select at least one collision-free pose from the output dataset (206); g) Display (207) the collision-free pose of the selected 3D virtual motion system in the 3D virtual environment, and use the selected collision-free pose to control the motion of the motion system.

7. The data processing system (100) according to claim 6 is configured to automatically classify the collision-free poses of the output dataset according to at least one predefined criterion.

8. A non-transitory computer-readable medium encoded with executable instructions, which, when executed, cause one or more data processing systems (100): a) Receive (201) a 3D representation of the surrounding environment, i.e., a 3D virtual environment; b) Receive (202) a 3D representation of the motion system, i.e., a 3D virtual motion system and a set of 3D poses defined for the 3D virtual motion system; c) Receive (203) the target task to be performed by the motion system relative to the surrounding environment; d) Receive (204) a specified position within the 3D virtual environment, the specified position defining the positioning within the 3D virtual motion system that must be placed in the 3D virtual environment; e) The collision-free detection function, or CFD function, trained using a machine learning algorithm that employs a convolutional neural network (CNN) is applied to the input dataset, where, The input dataset includes the 3D virtual environment, the target task, the specified location, and the set of poses. The CFD function is configured to generate an output dataset, which is a set of collision-free poses that enable the motion system to perform the target task when located at the specified location. The CFD function is configured to automatically acquire one or more depth images of the 3D virtual environment to transform the kinematic problem of finding collision-free poses for the 3D virtual motion system into an image classification problem for identifying collision-free poses. The CFD function uses a dimensionality reduction algorithm to automatically convert the depth images, the specified location, and the target task into 2D images that encode the depth images, the target task, and the specified location. The CNN is applied to the 2D images representing the 3D virtual environment, the specified location, and the target task. The CNN automatically outputs from the received 2D images the set of collision-free poses, including all poses from the 3D pose set that enable the motion system to perform the target task when located at the specified location. f) Select at least one collision-free pose from the output dataset (206); g) Display (207) the selected collision-free pose of the 3D virtual motion system in the 3D virtual environment, and use the selected collision-free pose to control the motion of the motion system.

Images