Robot arm guiding method, guiding system

Abstract

In a method for guiding a robot arm (6) in real time for processing body surface data, A step of generating a 3D model (MOD_P) of the entire or part of a human body surface including meshing (M1, M2) of points; Calculator; a step of planning a treatment trajectory (TRAJ1) on the surface of the body model (MOD_S) using the parameters of the sensor; A step of activating at least one transmission of a transmitter of an actuating device (8) and / or the collection of at least one sensor (the actuating device (8) is positioned at the end portion of the robot arm (6), and the activation is performed when the axial direction of the sensor or the transmitter coincides with a predetermined straight line passing through the target point (PTARGET), and the target point is referenced on the generated body model (MOD_P); A guiding method including

Description

Technology Field

[0001] The field of the present invention relates to a method for constructing a three-dimensional image of a human body in order to dynamically and automatically control a device located near the human body. More specifically, the field of the present invention relates to a method for enabling real-time dynamic control that requires rapid calculation of a three-dimensional surface. Accordingly, the present invention relates to a method for generating a surface using an algorithm that combines a parameterized model with a learning technique based on artificial intelligence. Background Technology

[0002] Currently, there are various techniques for reconstructing three-dimensional surfaces in space. However, these techniques are not suitable for the purpose of guiding medical devices performing treatment close to the patient's skin surface in real time. In practice, reconstructing body surfaces requires image processing, which demands significant computational resources and often a large number of acquisition systems. Furthermore, due to this complexity, partial processing or approximation calculations requiring human assistance are required.

[0003] Due to these limitations, performing precise guidance of the device within space is currently not feasible. Generally, guiding a robotic arm within space that is automatically controlled by calculations of deformable or movable surfaces remains difficult to implement. Nevertheless, there is a need for robotic arm control that is automatically guided and responds based on specific analyses performed in real time. Conventional solutions require additional user control or long processing times.

[0004] Therefore, it is necessary to define a means to resolve such problems. The problem to be solved

[0005] The present invention aims to resolve the aforementioned problems. means of solving the problem

[0006] According to one embodiment, the present invention relates to a method for guiding a robot arm in real time for processing body surface data, and the guiding method comprises: generating a model of a body including meshing of points; a calculator; planning a treatment trajectory on the surface of the body model using parameters of a sensor; calculating a guiding trajectory for a distal end of a robot arm; generating a kinematic of a robot arm such that a point of a head portion placed at the distal end of the robot arm is automatically controlled by the guiding trajectory; activating the kinematic of the robot arm to move along the guiding trajectory (said that the trajectory is calculated in real time from the generated body model); and receiving the position of at least one target point in space. The method includes the step of activating at least one transmission of a transmitter of an actuating device and / or the collection of at least one sensor (the actuating device is positioned at the end portion of a robot arm, and activation occurs when the axial direction of the sensor or transmitter coincides with a predetermined straight line passing through a target point, and the target point is referenced on a generated body model);

[0007] One advantage is the ability to utilize guiding methods that consider a body model replicating the patient's body. This reduces computational load and enables real-time automatic control of medical equipment. By using a meshed body model calculated from the patient's body, the reconstruction of the real world is optimized. Another advantage is that using a simple body model allows for the consideration of the patient's potential movements or the prediction of collisions. The body model used, which includes a set of points or meshing, can be utilized more easily for real-time guiding compared to systems based on direct image collection of the patient for robot movement prediction.

[0008] According to one embodiment, the present invention relates to a method for guiding a robot arm in real time for processing body surface data, and the guiding method comprises: generating a 3D model of the whole or part of a human body surface including meshing of points; planning a treatment trajectory on the surface of the body model using a computing unit; calculating a guiding trajectory for the original part of the robot arm; generating kinematics of the robot arm so that a point of a head part placed on the original part of the robot arm is automatically controlled by the guiding trajectory; activating the kinematics so that the robot arm moves along the guiding trajectory (the guiding trajectory is calculated in real time from the 3D model of the human body surface); generating a new 3D model of the human body surface at regular time intervals to automatically manage the displacement of the robot arm in real time, and calculating a new kinematic of the robot arm accordingly (a new treatment trajectory calculation and a new guiding trajectory calculation are performed as the surface of the 3D model is newly calculated); and receiving the position of at least one target point in space. The step of activating at least one transmission of a transmitter of an actuating device and / or the collection of at least one sensor (the actuating device is positioned at the base of a robot arm, and activation occurs when the axial direction of the sensor or transmitter coincides with a predetermined straight line passing through a target point, and the target point is referenced on a generated body model); is included.

[0009] According to one embodiment, a body model is generated from a system for collecting images of a patient's body, and said system is positioned at a point in space. One advantage is that by collecting a partial view of the human body, it is possible to generate the entire human body from an algorithm based on the utilization of a neural network.

[0010] According to one embodiment, the treatment trajectory is generated within a treatment area on the surface of a body model.

[0011] One advantage is that the generated surface of the body model is used as an input to define the generated guiding trajectory. Therefore, guiding is optimized due to the simplicity of utilizing the meshed body model.

[0012] According to one embodiment, the treatment trajectory is automatically generated by defining a treatment area on the surface of a body model.

[0013] One advantage is that the task is automated by allowing the user to simply select a region. The region is defined on the generated body model. Therefore, the defining process is simplified, and trajectories can be easily generated.

[0014] According to one embodiment, the guiding trajectory is calculated from predetermined distance and direction instructions to generate the relative position of the original part of the robot arm to the treatment trajectory.

[0015] One advantage is obtaining a simplified guiding method by an automatic control system that considers only minimal parameters.

[0016] According to one embodiment, the robot arm comprises at least three sets of branches connected by two joints, wherein each set of branches is connected to one another by a pivot link, and the set of pivot links comprises a kinematic planned for a given guiding trajectory and a generated body model.

[0017] One advantage is that the kinematic generation of the robot arm can be separated within the robot arm's computational unit based on the position of the guiding point of the trajectory generated in the operating console.

[0018] According to one embodiment, the guiding method includes the step of generating a new body model at regular time intervals to enable real-time control of the robot arm as a new trajectory of the robot arm is generated (modification of the treatment trajectory is performed as the surface of the body model is newly calculated), and further includes the step of calculating a new kinematic of the robot arm that is automatically controlled by the defined new trajectory. One advantage is that excellent guiding performance is obtained in real time. Guiding calculations are simplified by considering the body model. Automatic control of the robot arm can be performed quickly, which can reduce collisions or sudden movements of the patient.

[0019] According to one embodiment, the guiding method includes the step of calibrating the position of a robot arm from a fixed position where an image is collected so as to generate a body model. The advantage of the calibration is that the position of the body model relative to the position of the end of the robot arm can be known.

[0020] According to one embodiment, a laser beam shot strategy of a laser transmitter fixed to an operating device is established using a treatment trajectory.

[0021] According to one embodiment, the body model includes a uniform mesh for generating a new mesh defined by an array of circles, at least one of the circles defines the location of a target point in space. One advantage is to simplify the activation of the actuation device of the robot arm based on a point that can be easily found on the mesh.

[0022] According to one embodiment, the new meshing is generated according to a shot power instruction and / or an instruction regarding the degree of overlap between two shots. One advantage is that meshing suitable for a robot arm application can be obtained.

[0023] According to one embodiment, sampling for the number of shots performed is characterized by being based on new meshing, a set shot output, and a defined degree of overlap between two adjacent (colocalized) circles. One advantage is to provide the user with an easy-to-use operating console.

[0024] According to one embodiment, an ultrasonic signal shot strategy of an ultrasonic transmitter fixed to an operating device is established using a treatment trajectory.

[0025] According to one embodiment, a set of images is obtained from an optical device placed in an operating device, and each image is indexed into at least one node or mesh of the meshing of the body model according to the location of a target point on the body model and the viewing angle of the optical device. One advantage is to reconstruct an outer layer, such as skin, on a parametric body model corresponding to the patient's body.

[0026] According to one embodiment, at least one pair of images are simultaneously collected from an optical device comprising two sensors embedded in an operating device and positioned on different axes of the operating device, each image is indexed into at least one node or mesh of the meshing of the body model according to the position of a target point on the body model and the field of view of the optical device, and the image is processed to reconstruct a three-dimensional local image.

[0027] According to one embodiment, a three-dimensional local image is superimposed as a 3D representation on a generated body model.

[0028] According to one embodiment, a first image collected by a first optical device of an operating device is compared with an image set in real time, and the comparison operation includes, where applicable, generating an indicator of interest, and can generate a digital control command for operating a robot arm based on the indicator of interest.

[0029] According to one embodiment, by means of a guiding instruction, (i) a first operation of a robot arm at a target location from which a first image is extracted and (ii) a second operation of an actuator for activating a second optical device are performed, and due to the activation of the second optical device, a second image within the near field of an area including a target point can be obtained. One advantage is that an image bank can be created that can be used, in particular for diagnostic purposes, for qualitative research purposes regarding patients and diseases, or an image history containing information about the skin at multiple scales for a single patient can be created.

[0030] According to one embodiment, the comparison step is a classification step of collected images by a neural network trained on an image set. One advantage is to optimize the step of detecting specific parts on the patient's skin.

[0031] According to another aspect, the present invention relates to a system for guiding a robot arm, and the guiding system comprises: a calculator for generating a plurality of body models at multiple time points; a robot arm automatically controlled by the calculator of a control unit (the calculator calculates the kinematics of the robot arm from a trajectory calculated from a point defined on the meshing of the collected body models); and an actuation device disposed at the end of the robot arm and comprising at least one transmitter or sensor (activation of the transmitter or sensor occurs when the axis direction of the transmitter or sensor is positioned along a straight line passing through a target point, the target point is referenced on the generated body model).

[0032] According to one embodiment, the present invention relates to a system for guiding a robot arm, and the guiding system comprises: a device for collecting images of a patient's body to obtain a parameterized body model; and a robot arm having an actuation device for carrying out the method of the present invention.

[0033] According to another alternative embodiment, the guiding system of the present invention comprises various means described in various embodiments of the present invention. Additionally, according to another alternative, the guiding system is configured according to various embodiments of the method of the present invention.

[0034] According to one embodiment, the present invention relates to a system for guiding a robot arm, and the guiding system comprises: a robot arm automatically controlled by a computational unit of a control unit (the computational unit calculates the kinematics of the robot arm from a trajectory calculated from points defined on the meshing of a 3D model collected on the surface of a human body); a computational unit for generating a plurality of body models at multiple time points (as a new 3D model of the surface of the human body is generated at regular time intervals, a new kinematic of the robot arm is calculated to automatically operate the displacement of the robot arm in real time, and as the surface of the 3D model is newly calculated, a new trajectory is calculated); and an actuation device disposed at the end of the robot arm and comprising at least one transmitter or sensor (activation of the transmitter or sensor occurs when the axial direction of the transmitter or sensor is positioned along a straight line passing through a target point, said target point is referenced on a 3D model generated on the surface of the human body).

[0035] According to another aspect, the present invention relates to a method for generating a three-dimensional working surface of a human body, the generating method comprising: receiving input data corresponding to geometric data; generating a first point cloud from the input data (each point is associated with a three-dimensional spatial coordinate); determining a plurality of attributes for each point of the first point cloud (the attributes include at least one shape descriptor, and at least one attribute of a point includes data representing a normal vector for the corresponding point of the first point cloud); associating the attributes with each point to generate an enriched second point cloud; using regression, (i) at least one rotation at at least one point of the second point cloud or at least one joint of a human body model; and (ii) a step of calculating a set of geometric parameters including at least one shape coefficient (the regression is performed through a series of matrix operations performed according to multiple layers of a neural network trained through multiple partial views of a parametric model parameterized with multiple parameterization configurations); a step of determining a parametric model to generate a body model including a first meshing; comprising.

[0036] According to one example, the input data includes data describing an image. Geometric data can be derived, for example, from data describing a 2D image. Thus, according to one case, the input data is an image.

[0037] One advantage of this method is that it enables real-time automatic control of medical device guiding located near the patient's body using a minimal data collection system (e.g., a pair of cameras). In particular, guiding performance is achieved based on the learning method of a high-performance neural network.

[0038] According to one embodiment, the method comprises the step of processing a first meshing to generate a homogeneous second meshing. One advantage of using homogeneous meshing is that it facilitates automatic control commands for a sensor or transmitter of the head of a robot arm. In practice, processing points of homogeneous meshing facilitates the use of devices in the head of the robot arm.

[0039] According to one embodiment, a shape descriptor for at least one given point of the first point cloud is calculated from the adjacency matrix of the first graph using a spectral partitioning method of the first point cloud. One advantage is that, in the vicinity of a point, a shape descriptor weighted according to a distance criterion for the applied point is considered. Another advantage is that, in the vicinity of a point, the topology of the local surface can be considered.

[0040] According to one embodiment, at least one local shape descriptor and / or at least one global shape descriptor is of any one of a "Wave Kernel Signature," a "Heat Kernel Signature," or a "Gaussian Curvature." One advantage is defining the shape descriptor based on spectral analysis of a region near a single point. Another advantage is the simplified use of formulas and normalized operators, which allows for the optimization of digital processing.

[0041] According to one embodiment, the normal vector for one given point in the first point cloud is calculated by considering the locally generated surface around each point located near the point. One advantage is to parameterize the resolution or granularity required to analyze the topology of the local surface on the body surface to be treated.

[0042] According to one embodiment, a surface locally generated at a point under consideration in a first point cloud is calculated based on (i) an underlying surface defined by a point in the first point cloud, or (ii) an approximation of a local surface directly obtained from the k nearest neighbor points to the point under consideration.

[0043] According to one embodiment, each point of the first point cloud is associated with at least two shape descriptors. One advantage is to enhance the description of the model. One interest is to obtain an improved learning effect while training the neural network. Additionally, one advantage is to improve the performance of the regression step of the neural network.

[0044] According to one embodiment, the first point cloud is generated from data obtained through (i) collecting a plurality of images from at least two optical devices (the position of each point of the first point cloud is derived through stereoscopic measurement of at least two collected images), and / or (ii) collecting images including a modified pattern (the pattern is projected by a projector and includes a known geometric structure), and / or (iii) collecting ultrasound signals transmitted to and reflected from the surface of the patient's skin, and / or (iv) collecting laser signals transmitted to and reflected from the surface of the patient's skin.

[0045] One advantage is to implement the method of the present invention using various collection techniques. Another advantage is to obtain a point cloud that can be utilized by the method of the present invention, particularly for guiding medical devices. Different collection techniques may be combined or substitute for one another. Nevertheless, an image collection system, such as a camera, corresponds to a collection system that is easy to implement and set up and can be easily installed.

[0046] According to one embodiment, a parameterized body model is obtained from parametric modeling of a human body in which a set of joints is defined, and the parameters model (i) at least one rotation angle for at least one joint; and (ii) at least one shape coefficient.

[0047] One advantage is defining a simple parametrization of the human body that takes into account the different degrees of freedom of each limb. Another advantage is performing real-time automatic control of the guided device according to the patient's possible movements.

[0048] According to one embodiment, the parameter also models (i) at least one global translation of the model in space; and / or (ii) at least one global rotation of the model in space.

[0049] One advantage is that it can account for patient displacement and adjust the patient's body collection system according to the placement of various devices.

[0050] According to one embodiment, a parameterized body model is obtained by defining a specific parameterization, wherein the specific parameterization includes a definition for a set of modeling parameters, thereby enabling the definition of a specific representation of the parameterized body model. The parameterized body model may then be represented by a third point cloud, graph, or meshing. One advantage is to generate a point cloud that can be directly utilized from the obtained parameterized body. Another advantage is to obtain a large number of bodies according to various poses in order to obtain excellent generalization performance of the neural network.

[0051] According to one embodiment, a plurality of partial views are generated. Each partial view is obtained, for example, from a subset of points of a third point cloud. One advantage is that a large number of views can be generated that correspond to the captured actual image. Accordingly, higher quality learning of the neural network is achieved, and the performance of the regression step is improved as learning is performed using a large number of partial views.

[0052] According to one embodiment, each subset of points of the third point cloud is acquired by selecting a point captured through a specific viewpoint in space, and a set of partial views is generated for a plurality of viewpoints in space.

[0053] According to one embodiment, the regression comprises (i) a neural network convolution or layer comprising a plurality of multiplication operations on a matrix including weighting coefficients obtained from a learning method; and (ii) a non-linear operation.

[0054] According to one embodiment, the input of the regression includes (i) points of the point cloud and attribute values ​​of the points; and / or (ii) an adjacency matrix of the graph corresponding to the point cloud.

[0055] According to one embodiment, the present method comprises a preliminary step of neural network learning, wherein the preliminary step is performed by a deep learning method comprising: (i) a learning step performed through a plurality of partial views of a parametric model parameterized by multiple parameterization settings; and (ii) a step of generating a matrix including weighting coefficients for each layer of the learned network.

[0056] According to another aspect, the present invention relates to a system comprising: an interface for collecting input data; a processing unit; and a memory, for processing input data collected by the method of the present invention to generate a parameterized body model.

[0057] According to one embodiment, the present invention relates to a system comprising: means for collecting distance data for a set of points on a body surface located in space; and a guiding robot arm (the robot arm has at least one degree of freedom, and further comprises a sensor or a transmitter, the position of the sensor or transmitter is automatically controlled by a guiding trajectory obtained from a treatment trajectory defined on a surface of partial field of view for a body model, and the body model is generated by the method of the present invention).

[0058] The system of the present invention may be configured according to various parameters mentioned to describe the method of the present invention. Additionally, the system of the present invention may include various means described in various embodiments of the method of the present invention, depending on various embodiments. Effects of the invention

[0059] The present invention has the effect of providing a method for collecting images of a body surface and reconstructing a three-dimensional surface of said body. The present invention also has the effect of providing an image collection and processing system. The present invention also has the effect of providing a method and system for activating a medical device through a modeled body model. The present invention has the effect of providing a guiding method, for example, for a robotic object for diagnosis or treatment of a patient's skin. Brief explanation of the drawing

[0060] Other features and advantages of the present invention will become apparent by referring to the detailed description of the invention below and the accompanying drawings. FIG. 1 illustrates the main steps of one embodiment of the method of the present invention. FIG. 2 illustrates an embodiment of a system for controlling the movement within space of a robotic arm positioned proximal to the surface of a body to be treated. FIG. 3 illustrates an embodiment of a system of the present invention comprising: an operating console; a positioning system; and a robot arm that can be operated by the method of the present invention. FIG. 4 illustrates an example of a robot arm according to the present invention having six axes and guided to move along a planned trajectory. FIG. 5 illustrates an example of a robot arm according to the present invention having six axes, with reference points of the head portion of the robot arm shown. FIG. 6a illustrates a first example of an operating device disposed in the head portion of a robot arm according to the present invention for use in dermatology or the like. FIG. 6b illustrates an example of a trajectory generated by the system of the present invention. FIG. 7 illustrates an example of a first meshing of a point on a surface to be treated according to the method of the present invention. FIG. 8 illustrates a first example of a second meshing obtained according to one embodiment of the present invention. FIG. 9 illustrates a second example of a second meshing obtained according to one embodiment of the present invention. Specific details for implementing the invention

[0061] The present invention relates to a method for collecting images of a body surface and reconstructing a three-dimensional surface of said body. The present invention also relates to an image collection and processing system. The present invention also relates to a method and system for activating a medical device through a modeled body model. The present invention relates to a guiding method for a robotic object for diagnosis or treatment of, for example, a patient's skin.

[0062] In the following description, a system for collecting points (5) of the patient (1) body is preferably placed at a fixed point within the space. As an example, such a point collection system is fixed to a ceiling, a wall, or any point within the space intended to remain fixed while carrying out the method of the present invention. Such a collection system (5) is referenced at a reference point denoted as R0 (O, x0, y0, z0). It may also be referenced at a reference point R1 (O1, x1, y1, z1) associated with the collection system (5). Depending on the position calibration between the "real world" reference frame and the collection system (5), the collection system (5) may be referenced at the reference point R0 by the R1→R0 transformation.

[0063] In the following description, the robot arm (6) is placed in a space that is included within the range of the acquisition field of the acquisition device (5). This may be a treatment room in a hospital or a patient room where a doctor's procedure is performed. The robot arm (6) may be placed on a pedestal, such as a movable piece of furniture (25), which can be fixed in a given position while treatment is performed according to the method of the present invention. This robot arm (6) may be referenced at a reference point denoted as R0 (O, x0, y0, z0). The position of the robot arm (6) in the space can be found within the reference point R2 (O2, x2, y2, z2), so that the movement of the robot arm (6) can be easily expressed. The movement of the robot arm (6) can be expressed at the reference point R0 of the acquisition system (5) according to the transition from reference point R2 to R0. This change of reference point may be particularly efficient when implementing a collision detection function between the robot arm (6) and the patient's body (1).

[0064] According to one embodiment, the collection system (5) may be integrated with a reference point R2. In this case, the collection system may be fixed to a support that supports the robot arm (6). This configuration can be achieved, for example, by using a support (not shown) that can raise the position of the collection system (5). One advantage of such a configuration is that it facilitates the installation of the system of the present invention. Another advantage is that pre-calibration between the positions of the collection system (5) and the robot arm (6) is possible.

[0065] According to another embodiment, the collection system (5) may be fixed to a base on which an operating console (20) is installed. In this case, an autonomous collection system including a hardware part and a software part may be used. Additionally, the collection system (5) may be fixed using a support or other mechanical support device.

[0066] In the following description, the head portion (8) of the robot arm (6), also referred to as the actuation device (8), is positioned at the base portion of the robot arm (6). The head portion (8) can be connected to a reference point denoted as R3 (O3, x3, y3, z3) as shown in FIG. 5. The position of the head portion (8) of the robot arm (6) can be easily identified at reference point R2 (O2, x2, y2, z2) based on information regarding the kinematic that moves the branch of the robot arm (6). One interest is to reference the image collected by the actuation device (8) at, for example, reference point R0 (O, x0, y0, z0).

[0067] The description of the present invention applies to all types of robotized objects, in particular to robotized arms. The robot arm of the present invention includes a plurality of axes of freedom and may have degrees of freedom between 2 and 9 in particular. In the following description, an application example of a robot arm having 6 axes will be considered to explain one embodiment. However, the present invention is not limited by such example.

[0068] According to another alternative, the robot arm may be fixed to a horizontal support and the first arm may be extended vertically, or the robot arm may be fixed to a vertical support and the first arm may be extended horizontally. Various configurations are suitable for the present invention.

[0069] In the following description, "body model" refers to a three-dimensional model of the body. This body model can be represented through a point cloud. The point cloud can be generated through specific modeling of a parametric body model obtained through specific parameterization.

[0070] Additionally, such a body model can be used to generate views of various parts of the body corresponding to a subset of a point cloud modeling the entire body. For this purpose, a viewpoint for extracting the partial views of the body model can be defined. Such a body model can be obtained from specific parameterization settings of a parametric model of the human body.

[0071] In the following description, the partial field of view of the "body model" refers to a three-dimensional model of the body according to a specific point in time when the surface of the body model is observed.

[0072] In the following description, "parametric body model (MOD_S)" refers to a model that can be parameterized to generate transformations for specific parts of the body or fields of view according to the settings of given parameters. The body model defined above can therefore be generated by specific parameterization of the parametric human body model, and this is referred to as the parameterized body model (MOD_P).

[0073] A parametric body model may be used to perform a regression step for the purpose of determining a set of parameters for generating a body model to be utilized for guiding a medical device, for example, according to a first use, when carrying out the method of the present invention. Additionally, according to a second use, it may be used to generate a weight matrix to be used in the regression step during a deep learning process.

[0074] According to one example, a parametric body model (MOD_S) can be obtained from a parametric model by inputting the following parameters.

[0075] (i) Global movement of the model in space: T x , T y , T z , that is, 3 parameters;

[0076] (ii) Global rotation of the model in space: R x , R y , R z , that is, 3 parameters;

[0077] (iii) Rotation angle of each joint (e.g., each of the three rotation parameters R xi , R yi , R zi A total of 72 parameters were defined using a parametric model including 24 joints with );

[0078] (iv) Shape coefficients (e.g., using a parametric model containing 20 shape coefficients, where the shape coefficients influence human body shape).

[0079] In this example, a human body model is created as 88 parameters are determined.

[0080] Other embodiments may be implemented according to the method and system of the present invention.

[0081] Collection of points

[0082] According to one embodiment, the method includes a point collection step (ACQ). Collection may be performed, for example, using an image collection device. In one embodiment, the optical system (5) of FIG. 3 includes two sensors (51, 52) for collecting images (IM1, IM2). These may include at least two color cameras or infrared cameras, such as cameras like 3D cameras. According to another embodiment, color or infrared cameras may be used. Preferably, a pair of identical cameras are used. A pre-calibration step may be performed to set the color of each camera and correspond to the same point in space.

[0083] As another inter-camera calibration technique, "stereoscopic calibration" is performed between two cameras to generate a 3D reconstruction of an object. The calibration aims to calculate the transformation between two images collected by each camera.

[0084] According to one example, this transformation involves the detection of key points and the calculation of descriptors obtained by the scale-invariant visual feature transformation, better known by the acronym "SIF". A correlation step to map the key points can be performed using an estimation technique, such as the "RANSAC" method, referred to as "RANdom SAmple Consensus," for example. Next, an isometric transformation calculation is performed between the detected key points. Finally, based on the transformation, points in the image can be triangulated to construct a depth map.

[0085] According to the first method, 2D images collected for the same object at two points in time at known locations can be combined according to known techniques to obtain a depth map or a distance map.

[0086] According to the method of the present invention, a point cloud (G1) can be obtained by processing and combining 2D images. The points constitute a first graph (G1). According to one example, the points of the obtained first graph (G1) are non-oriented and non-connected. In its simplest form, the graph (G1) is a list of points that are not aligned and whose coordinates are referenced in space (e.g., referenced at R0, R1, R2, or R3). Accordingly, each point is defined independently of other points. Therefore, in its simplest form, the graph (G1) is a point cloud.

[0087] Images (IM1, IM2) can be collected simultaneously. Advantageously, the optical devices are positioned along parallel axes and are colocalized. One interest is to generate a depth map by processing the images using a stereoscopic technique. The depth map enables referencing spatial coordinates x, y, and z at multiple points.

[0088] According to one embodiment, the color of an image is used to create a tone difference, that is, a distance variation.

[0089] A depth map is obtained by encoding the distance between different collected points, and its resolution can be set. The grayscale can vary depending on a first value range. For example, by selecting a 16-bit encoding, the grayscale encoding of the depth map makes it possible to obtain the distance between different points with high precision.

[0090] According to the method of the present invention, it is possible to continuously collect an image stream by an optical device (51, 52) and process it in real time. Symbols IM1 and IM2 refer not only to the collected images but also to each camera (51, It also refers to the image stream collected by 52).

[0091] The image can be processed to extract a collection perimeter that defines the area for treatment. To this end, a template is used to avoid considering points outside the region of interest defined by the body surface.

[0092] By collecting two images (IM1, IM2) and including the entire surface to be treated, it is possible to generate a depth map of the entire body. The depth map includes a set of points and the relative positions between each point.

[0093] According to another example, no surface is treated during this stage of the method. The distance between points is generated independently of form recognition, such as the outer layer of the body.

[0094] According to one example, the number of points generated per unit surface can be set according to the required resolution of the point cloud. To this end, the resolution can be increased by setting the system to collect more or fewer images during the collection process.

[0095] According to one embodiment, a positioning system (POS_SYS) can define a frame of reference and a coordinate system, thereby enabling the position of each point within a single frame of reference. Advantageously, such a positioning system may be a collection device (5).

[0096] A transformation between R0 and R1 can be obtained through stereoscopic correction, where R1 is a reference point associated with the collection system (5). Next, the method of the present invention enables the robotized object (6) to be corrected with respect to the collection system (5), thereby R 2 → R 1 → It enables the transformation of images within the reference frame of R0. When the reference frame R3 is associated with the head part (8) of the robot arm (6), R is known to the computational part and memory of the system operating the robot arm (6) as the kinematics for each arm's joint are known. 3 → You can see the conversion to R2.

[0097] Other collection techniques

[0098] According to one embodiment, a projector device is used to project a pattern onto a body surface. The collected image is subsequently processed to analyze the deformation of the projected pattern. By using this technique, additional data can be acquired to improve 3D reconstruction. In particular, by detecting the deformation of the projected pattern, the precision of the 3D reconstruction can be improved. In particular, by detecting the deformation of the projected pattern, the precision of the distance measurement calculated to generate a depth map can be improved. Additionally, the use of a projector can reduce noise when constructing a point cloud for generating a first graph (G1).

[0099] Consequently, the reconstruction of the depth map by stereoscopy can be obtained by a pair of cameras and optionally at least one pattern projector. The projected light may be in the visible range or outside the visible range, such as infrared. According to one example, the acquisition system (5) may include a pair of color 3D cameras and at least one infrared projector.

[0100] According to another embodiment, the camera used may be an infrared camera or have a frequency range close to infrared.

[0101] Other techniques for constructing depth maps, such as laser transceivers, can be used. Due to the reflection of the laser beam, data regarding the geometry of the reflective surface can be generated. Another technique that can be used is an ultrasonic transceiver. An example of a system that can be implemented is a LIDAR-type system.

[0102] According to one embodiment, through the method of the present invention, a point cloud derived from a collected image can be recovered and transmitted to a graphics processor such as a GPU (graphics processing unit).

[0103] Graph 1 (G 1 Creation of )

[0104] The method of the present invention includes a step of generating a set of points associated with spatial coordinates referenced, for example, at a reference point R0 in space. Such coordinates can be obtained from a prior correction of a positioning system (SYS_POS). The correction may be performed using a specific pattern collected by the optical unit of the positioning system (SYS_POS) at the start of the method according to the present invention. According to one embodiment, the positioning system (SYS_POS) and the optical system (SYS_OPT) may be the same system, and according to another embodiment, they may be two separate systems.

[0105] Therefore, by utilizing the depth map obtained by the collection system (SYS_OPT), a set of points in space associated with coordinates referenced from the reference point R0 in space can be obtained.

[0106] The method of the present invention includes a data reception step (REC_DATA1), wherein the data (DATA1) corresponds to a point in space referenced at a reference point R0 in space.

[0107] According to one example, the method of the present invention may include a step of segmenting data associated with a predefined surface. The surface may be, for example, the surface of a human body. Segmentation may be performed automatically by analyzing data (e.g., data derived from collected 2D images and / or acquired depth maps). For example, the method of the present invention may include a step of analyzing distance differences between specific points or performing colorimetric processing. By filtering the point cloud using a depth map, it may be possible to include only points of interest, i.e., points corresponding to the human body.

[0108] According to another example, a surface contour can be collected using an operating console such as the PC (20) shown in FIG. 3. According to such an example, the contour of the body surface can be obtained using a selection tool such as a stylet or a mouse and an image displayed on the screen of the console (20) by the user. According to another example, the selection can be made by a tactile command on the screen or by a voice command. One concern is, for example, to ignore points included on the table (22) on which the patient (1) is lying.

[0109] The method according to the present invention includes a step of generating a first graph (GEN_G1). The first graph (G1) can be obtained from a matrix representation of a set of points generated on the body surface of a patient (1). Each point is v i It is denoted as such, and can be represented as a vector including coordinates within reference point R0 in particular. According to one embodiment, the first graph (G1) at this stage is non-ordered within the collection reference frame. Here, the expression "graph" is also used to refer to the simplest embodiment, or a set of points corresponding to a list of points with coordinates (i.e., a simple point cloud). In other embodiments, the graphs may or may not be connected to each other, and may or may not be directional.

[0110] Filtering

[0111] According to one embodiment, the method of the present invention includes a step of filtering a obtained graph, wherein the filtering may be temporal filtering and / or spatial filtering. Through the filtering step, it is possible to reduce noise that occurs while generating a first graph (G1) following the detection step.

[0112] Attribute determination

[0113] The method according to the present invention comprises each point (v i A step of determining at least one attribute for ) (DET_AT i Includes ). The attribute is a point(v i It corresponds to data associated with ), and such attributes can have various characteristics such as Cartesian data, shape descriptors, identifiers, and vectors.

[0114] According to one embodiment, the attribute is a point (v) of the first graph (G1). i Normal vector (n) for ) i Includes data representing ). normal vector(n i ) can be determined by several different methods. One interest in calculating the normal vector as an attribute of a point in a point cloud is to obtain information about the convexity of the local surface.

[0115] According to one embodiment, a specific point (v) of the first graph (G1) i Normal vector (n) at ) i ) is a point (v) located near each point (vi) under consideration. iLk It is calculated by considering the locally generated surface from ). To do this, the spatial coordinates of each point are compared with the coordinates of the point being considered. According to one example, collected distance and depth data are used to identify points near the point being considered.

[0116] According to the first example, the surface considered for calculating the normal vector is calculated from the underlying surface defined by the points of the first graph (G1).

[0117] According to the second example, the surface considered for calculating the normal vector is calculated from a method executed on a GPU. According to another example, the normal vector is calculated through a method that uses an approximation of the local surface obtained directly from the k nearest neighbor points to the point considered. However, this triangulation method of linking a list of 3D points can be more expensive than the method executed on a GPU.

[0118] One application of calculating normal vectors for points in a graph is to increase the description of the points to enhance the point cloud.

[0119] Form descriptor

[0120] According to one embodiment, the attribute is each point (v) of the first graph (G1). i At least one form descriptor (Df) for ) i Includes (notated as ). A form descriptor is a local descriptive signature for a space at a single point.

[0121] According to one example, the morphological descriptor (Df i ) is the point being considered (v i It is calculated from a subset of points located in the vicinity of ). Accordingly, the present method includes the step of generating a graph of points reconstructed or extracted from a body model (the body model is obtained from / for each collection device that can utilize such data). For example, if the collection device is positioned at the end of a robot arm, a partial view of the body model is generated from the position of the collection device so that a specific treatment can be performed.

[0122] Additionally, shape descriptors can be derived from each point of a 3D body model obtained from a point cloud. As long as they possess symmetry and isotropy, shape descriptors are invariant under isometric transformations, such as rotation of the body model. This is advantageous for transposing shape descriptors regardless of the viewing angle being considered.

[0123] One advantage is that shape descriptor calculations are performed directly on points of the first graph (G1). The present invention, therefore, allows for the optimization of shape descriptor calculations by applying shape descriptors directly to predefined points of the graph rather than to surface parts. The descriptor calculations can be performed on the adjacency matrix of the connected graph obtained from the graph (G1). A connected graph means that each point is connected to all other points. The adjacency matrix allows for obtaining points directly connected to a given point. Additionally, the degree of proximity between two points can be derived from the adjacency matrix. According to the shape descriptor, the neighborhood of a point can be derived from the adjacency matrix.

[0124] Alternatively, according to the method of the present invention, it is possible to apply a distance threshold such that a point is connected only to the nearest N neighbor points.

[0125] According to one embodiment, the form descriptor is selected to be isometric and invariant. One interest is to obtain a form descriptor that is invariant when a body movement resulting from an isometric transformation occurs.

[0126] According to the first example, the shape descriptor corresponds to the "Wave Kernel Signature" type denoted as WKS. According to the second example, the shape descriptor corresponds to the "Heat Kernel Signature" type denoted as HKS. According to the third example, the shape descriptor corresponds to the "Gaussian Curvature" type.

[0127] According to various other embodiments, for each point considered of the first graph (G1), a plurality of shape descriptors (Df i ) is calculated and is associated with each of the above points.

[0128] Certain descriptors, such as HKS or WKS descriptors, belong to a group of spectral shape analysis methods. In particular, HKS descriptors are based on the solution to heat equations and the Laplace-Beltrami operator.

[0129] One advantage of such a shape descriptor is that it is invariant under certain transformations, particularly equidistant transformations. Additionally, such a shape descriptor can be calculated from the adjacency matrix of the graph (G1). One advantage is that it can consider the relative distance between a point under consideration and a set of nearby points.

[0130] According to one example, the method of the present invention is used by considering a threshold value applied to an adjacency matrix, so that only a subset of points whose distance is greater than a specific threshold value can be considered.

[0131] The WKS descriptor is based on the solution to the Schrödinger equation. One advantage of the WKS descriptor is that it enables spectral separation of the modeling obtained by the operator.

[0132] A shape descriptor can be selected to model a local or global 3D shape. When such a descriptor encodes local modeling, a validity radius can be introduced. However, in the case of global modeling, the operator used has the advantage of having complete geometric invariance.

[0133] Descriptors in the form of "Global Point Signatures," denoted by GPS, can also be used. These descriptors are based on Laplace-Beltramy type operators. The primary concern of these operators is to obtain global modeling.

[0134] A descriptor in the form of "Gaussian Curvature" denoted by GC can also be used. This descriptor allows for the consideration of a parameterized surface, and thus, conversely, allows for the consideration of points that locally define the surface.

[0135] The "Spectral Graph Wavelet Signature" descriptor, denoted as SGWS, is also used to define multi-resolution local descriptors.

[0136] One interest of using shape descriptors based on spectral shape analysis is that such shape descriptors can be mathematically processed to assemble connected or adjacent surfaces, partition regions, and execute shape recognition algorithms.

[0137] One advantage is that it enhances the regression performance of the neural network by strengthening the description of the points.

[0138] Another advantage of the shape descriptor is to generate an identifier for a point on the first graph (G1). When a patient's body moves or changes over time, the surface of the body is considered a non-isometric deformable object. The shape descriptor generates specific features for each point so that it can be re-identified according to the method of the present invention after deformation has occurred.

[0139] Association of attributes

[0140] Therefore, each point (v) of the first graph (G1) i ) is, coordinate {x i , y i , z i}; normal vector n i ; and a set of form operators {Df i1 , Df i2 , DF i3 , … , Df iN This step can be associated with}. This step is ASSO_AT in Fig. 1. i It is written as.

[0141] Afterwards, points (v i ) can be composed of a finite-dimensional vector, and each dimension of the vector corresponds to a specific attribute. Accordingly, a second graph (G2) can be generated. The second graph (G2) is an attribute (AT i It corresponds to a list of vectors reinforced by ).

[0142] return

[0143] This is a computational step that enables the three-dimensional reconstruction of the human body collected and enhanced through approximation calculations based on a parametric human body model. This step is denoted as REG in Fig. 1. The purpose of the regression step is to estimate specific parameters and set the parameterizable body model using those parameters. Thus, a complete parameterized body model can be estimated from a partial field of view collected by a camera.

[0144] For this purpose, the method of the present invention comprises: shape and rotation (ROT i It includes the step of calculating a set of geometric parameters including ) parameters.

[0145] Rotation can be calculated for multiple points of the second graph (G2). According to one embodiment, rotation is calculated for each point of the second graph. According to one embodiment, each rotation can be associated with one joint. One joint can be associated with a limited area of ​​the parametric body model (MOD_S), and thus a given joint can be associated with the parameterized body model (MOD_P). Thus, as a result of the regression operation, a given joint can be associated with a set of points of the second graph (G2).

[0146] Shape parameters can be defined by the point locations of the parameterized body model within a specific reference point.

[0147] According to one embodiment, the input parameter is a set of points of point cloud G1 or G2 provided with attributes calculated in Euclidean space R3.

[0148] According to one alternative, the input parameter may be an adjacency matrix corresponding to the graph (G1) of each point in the point cloud. This input allows for the examination of the spectral domain of the input and does not necessarily require consideration of properties.

[0149] Shape parameters and rotations for a set of joints correspond to the output parameters of the neural network. Due to these parameters, it is possible to generate a parameterized body model (MOD_P) through the configuration of the parametric body model (MOD_S).

[0150] According to one embodiment, the regression step includes a series of mathematical operations, such as matrix operations between an input matrix and a weight matrix learned in advance through a deep learning method.

[0151] A non-linear function follows each operation. This is an activation function or transfer function that enables the generation of outputs from network nodes. For example, non-linear functions such as ReLU, Sigmoid, or Tanh can be executed.

[0152] According to one example, the present step includes regression performed from a series of layer convolutions of a neural network learned through a plurality of predefined parameterized body models (MPPi}i∈[1;N]).

[0153] This can be a convolution or a layer of the neural network. In practice, the network may include a convolution in the first layer, or a layer of fully connected neurons at the end of the model. In the latter case, this is a neuron connected to every neuron in the preceding layer and every neuron in the next layer.

[0154] A convolution layer may include a scanning step of the input matrix that generates a series of matrix operations. Other layers of a neural network generally include matrix operations on the size of the input matrix. In all cases, the corresponding step includes matrix operations.

[0155] According to one example, each convolution involves a matrix product between the input matrix and the weight matrix, and additional biases are taken into account.

[0156] The application of processing by successive layers within a neural network involves the process of generating the layer's output by applying a series of matrix multiplications followed by non-linear functions. This sequence of operations defines the depth of the neural network.

[0157] According to one embodiment, the neural network is a multilayer perceptron known by the abbreviation MLP. According to one example, the neural network may be a network corresponding to an MLP.

[0158] According to one example, a neural network is configured so that the output remains invariant with respect to permutations of points in an input list. This configuration is made possible by using a pooling function, such as the function known as "max pooling" or "average pooling."

[0159] According to one embodiment, the structure of the neural network enables the extraction of input attributes at various scales. The vicinity of each point of the point cloud at various scales is continuously considered to extract the encoding within the space of the reduced input dimension.

[0160] This encoding is then passed to a series of fully connected layers, where regression is performed on the parameter set of the parametric human body model.

[0161] The vicinity of the sampled points is sampled and / or grouped, and a successive step of grouping these points is repeated and encoded by a convolution layer to generate a map for the acquired parameters.

[0162] The method according to the present invention advantageously allows the symmetry of a set of points at the input of the network to be preserved at the output of the network. Accordingly, by using such a symmetry-preserving function, it is possible to apply a neural network to an unconnected and undirected graph. The function ensures that this step is independent of the alignment of the input points.

[0163] The function also enables the execution of a self-encoder within the regression processing.

[0164] A set of parameterized models can be obtained from generic models. One interest of such models is to enable the generation of multiple poses for the human body through the parameterization of shape parameters and joints. One feature of such a body model is that it can be modeled by a set of joints, each comprising at least one rotation and position. The rotation of a joint can be defined by an angle. The position of a joint can be associated with at least one point in a graph or at least one position for multiple points. According to another example, the position of a joint can be generated from an operation that combines different positions in a graph, and such operation is performed, for example, to generate a new position for a new point. The new point can be, for example, the barycenter of a subset of points in the graph. Shape parameters can be parameterized to define global shape parameters for the body.

[0165] A specific pose of a body model can be defined through a configuration data set for joint sets with position and rotation. Therefore, it is possible to generate multiple poses for the body model.

[0166] According to one embodiment, the parametric modeling for a body model is a meshed point network. Each parameterized joint generates a configuration that associates one subset of points in the meshed point network.

[0167] According to this example, 24 joints can be defined from 72 rotational parameters, and each joint is associated with 3 rotations. Additionally, this parametric model may include 3 global rotations. To implement an embodiment of the present invention, a subset of joints, such as the wrist, elbow, etc., may be extracted.

[0168] Each joint may also include a set of shape parameters, such as wrist width, wrist length, etc.

[0169] According to one embodiment, a general parameter can be extracted using a Principal Component Analysis (PCA) function, and a set of parameters for multiple joints can be adjusted by setting the general parameter. For example, a set of local parameters connected to multiple joints can be pre-adjusted by setting an individual's body size, abdominal thickness, obesity level, pelvic size, etc.

[0170] The parameterization of the body may include multiple vectors that define shape and angles. For example, these could be ankles, waists, arms, etc. Using general settings, a body silhouette or specific template that can be parameterized can be obtained.

[0171] Other parameterizable body models may be used in conjunction with the method of the present invention.

[0172] Neural network learning

[0173] Neural networks are trained by an automated learning method better known as "machine learning."

[0174] Such a machine learning algorithm includes the step of retrieving a database of high-resolution body scans. Next, a step is performed to transform the body template of a parametric model by applying an isometric transformation. The transformation may include translations, rotations, and combinations thereof applied to different facets of the template to match the scans derived from the retrieved data. This step may include, for example, the execution of a cost function to be optimized by gradient descent.

[0175] The present method includes a step of pooling transformations performed for each scan. Subsequently, the principal components of the transformations are derived by an ACP-type method.

[0176] Through the application of this algorithm, the major shape and position parameters of the human body can be extracted.

[0177] A ray tracing algorithm may be applied to generate a set of partial views (VPs) for a set of parameterized body models obtained through multiple parameterizations of the parametric body model. The algorithm generates multiple partial views from a specific parameterized body model by considering multiple different viewpoints. One advantage is that when executing the method according to the present invention, a correction step can be avoided, and images of the patient's body collected can be made closer to the real world. The proliferation of learning through multiple partial views allows for avoiding body processing steps, such as translation or rotation of collected data representing the patient's body.

[0178] One interest of the present solution is to generate multiple body models using various parameters with multiple partial views of the body models. Such partial views (VPs) correspond to a subset of a point cloud defining the entire body. One advantage is that the neural network is trained with multiple settings that can correspond to actual body collection instances when using the method according to the present invention. Viewpoints correspond, for example, to camera capture points. Multiple viewpoints are applied along the surface of each body generated from the parametric model to construct the most complete data library representing the body models. Thus, multiple body surfaces represented according to different viewpoints are stored in memory. The dataset generated to train the neural network to perform a regression step is a set of partial views (VPs) for multiple parametric body models (MOD_P).

[0179] One advantage is that the regression weights are learned through shape descriptors associated with each point of the body model as it can be perceived in the real world. Shape descriptors can be obtained for each generated partial body model viewpoint (VP) and for each point of the given body model. The shape descriptors obtained through learning enable the definition of a weight matrix for each layer of the neural network, and the weight matrix is ​​a matrix containing weighting coefficients. In the regression step, which is performed in real time while computing the patient's body model, the regression step is preferably performed using the weight matrix defined in the learning step.

[0180] Therefore, the generated parameterized body model (MOD_P) can train a neural network to perform the regression step (REG). Subsequently, parameters representing the characteristics of the body under consideration can be extracted through real-time regression.

[0181] By following this step, a body model from which surfaces can be easily extracted can be obtained, and the entire parameterized body model, rather than just the parts captured by the camera, can be considered.

[0182] One advantage according to the present invention is to reconstruct a three-dimensional surface in real time through inference. This solution is based on using a model trained by deep learning technology. One interest is to avoid the need to use optimization methods, such as iterative methods (e.g., gradient algorithms), which entail long computation times and integration limits, in the real-time automatic control of a robot arm. One interest of the method according to the present invention is to implement optimization during learning, rather than while data collected during the guiding of the robot arm is being processed.

[0183] Deep learning inference involves matrix multiplication operations using pre-learned weights.

[0184] According to one embodiment, the layers of a neural network include a resolution depending on the layer being considered. As one goes down to deeper layers of the network, the scale can increase progressively. For example, a neural network such as an MLP network may be used.

[0185] First meshing, M 1

[0186] Accordingly, the present method includes a step of extracting a meshed surface. The meshing of the acquired surface is a first meshing denoted as M1. According to one embodiment, the first meshing (M1) is a meshing in which a triangle is defined by a point on the surface and a nearby point considered together. FIG. 7 shows three adjacent points (n) within the surface being considered. i An example of meshing (M1) including a plurality of triangles formed by each of the following is shown.

[0187] According to another embodiment, in another embodiment of the invention, other meshing patterns may be used, which generally apply to squares, squares, pentagons, and any type of polygon where the points of the meshing form vertices.

[0188] Meshing (M1) is generated after the regression operation. Before the regression step, the point cloud is considered such that each point is independent of the others, and the points are not connected to each other.

[0189] Second meshing, M 2

[0190] According to one embodiment, the method of the present invention comprises the step of defining a second meshing (M2) from a first meshing (M1). The second meshing (M2) may preferably be defined to generate a semi-homogeneous triangle, such as an isosceles triangle. In this example, the triangle may be transformed according to a specific algorithm, for example, by utilizing the symmetry properties of the generated meshing.

[0191] FIG. 8 illustrates an example of a second meshing (M2) obtained according to a transformation. The point set of the meshing (M2) is an unchanged first point set (n i ) and, for each triangle considered, a second set of points (n) generated by bisectors for generating two new isosceles triangles. i Includes ')

[0192] Other uniformization techniques for meshing to regularize meshing may be adopted. According to other techniques, isosceles triangles such as those shown in FIG. 9 may be generated. One advantage is that a completely uniform meshing is generated, which facilitates guiding of the operating device (8) in particular.

[0193] The method according to the present invention applies a transformation to the points of meshing so that the attributes of the points can also be transformed according to the Cartesian coordinate positions of each newly created point.

[0194] According to one example, the required resolution of meshing M1 or M2 is determined by parameterization of the sensor or transmitter at the end of the robot arm. The resolution of the graph (G1) can be configured in various sizes and can be independent of the resolution of meshing M1 or M2. For example, circles 36, 37, and 38 represent the desired aiming points of the output to be delivered from the laser onto the surface of the patient's (1) skin. One concern of the solution is to generate a dermatological resolution suitable for the treatment to be applied.

[0195] Accordingly, FIG. 8 illustrates circles (36) of different sizes obtained by meshing with isosceles triangles of different dimensions. Accordingly, a coefficient must be applied when a shot is emitted in order to adapt the output applied to the signal. According to another example, the second meshing (M2) can be homogenized by a series of geometric processing to obtain triangles of the same dimensions.

[0196] FIG. 9 shows the aiming point 37 or 38 of the output to be delivered from the laser onto the skin surface. One interest of this meshing setup is to ensure that the same shot setup can be obtained between one triangle and another triangle. The triangles (38) are in contact with each other, allowing the parameter of the degree of overlap to be controlled.

[0197] According to one embodiment, a new meshing (M3) is defined by the circles (36, 37, 38) generated from the second meshing (M2).

[0198] Real-time automatic control

[0199] According to one embodiment, the collection device (5) performs real-time collection (ACQ) of points on all or part of the patient (1) body to generate a body model (MOD_P) parameterized by the method of the present invention adapted to automatically control the movement of the robot arm (6). Guiding of the robot arm (6) is performed through a treatment trajectory (TRAJ1) indicated as 30 in FIG. 4, established within a defined area (9) on a modeled and recalculated surface.

[0200] The trajectory (TRAJ2) of the robot arm (6) can be automatically generated from the treatment trajectory (TRAJ1) according to the relative distance and direction between the skin surface and the head portion (8) and the robot arm (6), defined by means such as a user interface (UI), for example. The direction is determined at 90° by default. That is, the principal axis of the sensor or transmitter of the head portion (8) of the robot arm (6) coincides with the normal N for the target point of the body surface under consideration, in particular the previously generated parameterized body model (MOD_P). The distance is the target point (P) on the modeled body surface generated from the sensor or transmitter point of the robot arm (6) head portion (8) and the patient's position. TARGET Corresponds to the distance between (indicated as ).

[0201] One interest in calculating the modeled surface of the patient (1) body in real time is, for example, to anticipate the movement of the patient (1) and to consider movements such as those arising from the patient's (1) breathing.

[0202] One advantage of the method according to the present invention is that a new modeled surface can be generated within 30 ms. This response time, in particular, allows the movement of the robot arm (6) to be automatically controlled at a sufficient speed to anticipate and avoid collisions or sudden movements as quickly as possible. The collection setting by the camera may correspond, for example, within the range of 10 to 90 fps.

[0203] Real-time automatic control of the robot arm (6) enables at least one transmission of the transmitter of the actuating device (8) and / or at least one collection of the sensor. Subsequently, the axial direction of the sensor or transmitter is set to a target point (P TARGET Activation occurs when perpendicular to the straight line passing through ). Target point (P TARGET ) is referenced on the generated parameterized body model. The latter is generated according to the position and orientation of the actuation device (8). Target point (P TARGET ) can be obtained before guiding of the robot arm (6), for example, at the stage of planning a laser shot or ultrasonic shot strategy. According to another example, a target point (P TARGET ) can be acquired while guiding is being performed, for example, when a change in guiding is enabled to activate another detector by contrasting the implemented image.

[0204] In all cases, the target point (P TARGET ) is defined according to the guiding strategy, and its position may be modified when the body surface under consideration is transformed, leading to adaptation of the guiding strategy.

[0205] According to one embodiment, the collection (ACQ) step by the device (5) may be coupled with image collection by an optical unit positioned at the end of the robot arm (6). This applies when the movement of the robot arm (6) obscures the area of ​​interest on the body of the patient (1) to be collected at the point of the collection device (5), or when a small degree of movement is controlled by the optical unit at the end of the robot arm (6).

[0206] Therefore, when performing treatment, two modes of automatic control are performed and combined to create a new modeled surface.

[0207] robotic arm

[0208] The robot arm (6) is guided by a motion control unit (GEN_TRAJ) that considers various constraints for automatically controlling the robot arm. In particular, this control unit considers a plan for a set trajectory and a treatment area defined by the user. Such a surface is defined on a body surface reconstructed according to the method of the present invention. Additionally, the control unit (CTRL) can adapt the movement of the robot arm (6) in real time by considering instructions transmitted from a collision detection module. Finally, the control unit (CTRL) can consider data transmitted from the head of the robot (6) using a module (CNTR_8), for example, when a specific image is detected and specific control of the robot is required.

[0209] According to one embodiment, the robot arm (6) includes a set of branches (61, 62, 63, 64, 65) connected to pivot joints (612, 623, 634, 645, 656). In this example, the robot arm (6) includes five branches connected by pivot links. According to one alternative, the robot arm includes six jointed branches, and according to another alternative, it may include seven, eight, or nine jointed branches. In practice, in this embodiment, each joint can implement at least one rotation between two adjacent branches.

[0210] According to one embodiment, each branch of the robot arm includes a force sensor. One advantage is that it prevents sudden collisions. When contact is made between a branch of the robot arm (6) and an object, the guiding can be modified so that the movement of the robot arm (6) is automatically stopped or the trajectory is changed.

[0211] According to one embodiment, a correction step of the robot arm (6) is performed prior to executing the method according to the present invention. The correction may include adjusting the relative position of the robot arm with respect to a reference point (R0) and the starting position of the robot arm.

[0212] Actuating device and optical part

[0213] According to one embodiment, the robot arm (6) preferably includes an actuation device (8) disposed at the end of the robot arm (6). According to the embodiment, the optical part (67) may be disposed to be located outside the axis of the head part (8) of the robot arm, or may be disposed at the end of the head part as in FIG. 6a.

[0214] One interest of the configuration illustrated in FIG. 3 is to provide the head portion of the robot arm with a device such as a laser device or an ultrasound generating device for local treatment and to displace the image acquisition optics. One interest of the configuration illustrated in FIG. 6a is to configure the robot arm (6) for dermatological image capture purposes. However, all configurations and combinations of the examples described above can be conceived.

[0215] According to one example, the actuating device (8) includes at least one sensor and / or at least one transmitter. The actuating device (8) is also referred to as the "head part" of the robot arm (6).

[0216] In the example of FIG. 5, the robot arm (6) includes one or more optical sensors (67) for collecting local images of the skin surface. One interest of capturing local images at the head portion of the robot arm (6) is to reconstruct the “colored” body surface, that is, to integrate the captured images with a parameterized body model (MOD_P) generated according to the method of the present invention. To this end, the optical device 67 or {83; 84} located at the end of the robot arm preferably includes two optical units to enable the reconstruction of a depth map associated with images collected simultaneously by the two optical units. The images can then be indexed onto the surface of the parameterized body.

[0217] Other transmission / collection devices

[0218] Additionally, the operating device (8) may include an ultrasonic, radio frequency, or laser type transmitter, or any other signal source capable of generating an incident beam. According to one embodiment, the operating device (8) includes a focused array of transmitters and means to control the emitted output and direction.

[0219] When an ultrasonic transmitter is placed at the end of the operating device (8), an ultrasonic receiver for measuring the reflected signal is associated to form an image of the skin and / or an image of the patient's skin thickness, or an image of a blood vessel or organ.

[0220] FIG. 6 illustrates another embodiment in which an operating device (8) comprises a receiving head (80) located at the end of the operating device. The receiving head (80) comprises two local optical units (83, 84) for capturing images of the body surface and a millimeter or microscopic optical unit capable of moving in two directions along the longitudinal direction (86). One interest is to generate an image with millimeter resolution, including resolution for dermatological use (e.g., diagnostic use).

[0221] Robot arm guiding

[0222] First, the movement generation of the robot arm (6) is defined by a trajectory set by the user using the component labeled PATH in FIG. 2. The user can select a region of interest on the surface of the patient's (1) body, and the region of interest corresponds to an area such as an "arm," "leg," or part of such area, or the entire body.

[0223] The selection of an area can be performed using a user interface (UI). An area may be indicated using a stylus or a mouse, or the user may select one or more predefined areas. According to other examples, selection may be made by issuing touch commands directly on the display or using voice commands.

[0224] The selection can be made on a parameterized body model (MOD_P) generated by the method (10) of the present invention. The module for performing region selection is indicated as ROI in FIG. 2. The parameterized body model (MOD_P) can be displayed on the display of a user interface (UI). Once a region is selected, extraction of the surface portion on which the trajectory of the operating device (8) is to be defined is performed. According to one example, the trajectory (30) can be automatically defined according to the dimensions of the selected region. Depending on the task to be performed using the operating device (8), such as image capture, laser treatment, and / or treatment using an ultrasound signal source, the automatic trajectory generation function can be executed in several different ways. When capturing an image of a pre-defined region, the purpose is to include the entire selected region. Thus, an optimized trajectory can be calculated according to the resolution of the optical unit and its field.

[0225] When the user defines a trajectory for a selected area, the trajectory is obtained by a component called PLAN_TRAJ, which calculates or optimizes movement parameters for the robot arm to move including the selected surface.

[0226] The method according to the present invention enables real-time guiding of a robot arm (6). To this end, a module (GEN_TRAJ) for generating movement of the robot arm (6) is executed. In order to calculate the trajectory in real time according to the patient's movement, the capture of a specific image, or control to stop movement when contact with an object occurs, the module enables the path of the robot arm (6) to be recalculated in real time.

[0227] Guiding of the robot arm (6) is established, for example, from a Cartesian coordinate system referenced within a reference point (R0). The direction of the end of the robot arm (6) can be established by controlling the direction of the normal (N) shown in FIG. 3 or FIG. 5. The normal (N) is defined by an angle of 90° with respect to the aiming point on the body surface. One interest of the corresponding parameterization is that each point of the reinforced meshing already contains an attribute corresponding to the normal for one point of the model. Thus, guiding of the robot arm head direction can be performed by aligning the direction of the robot with the normal (N) of the body surface.

[0228] According to one embodiment, the distance between the end of the head portion (8) of the robot arm (6) and the aiming point (35) on the modeled surface of the patient (1) body is automatically controlled to substantially maintain a certain distance (e.g., 5 cm). Accordingly, the guiding of the robot arm (6) is automatically controlled along the distance and direction from the patient (1)'s skin.

[0229] Next, a collision detection function (DET_COLLI) may be executed to transmit control instructions to the movement generation module (GEN_TRAJ) of the robot arm (6). Additionally, this module takes into account the initial trajectory established by the PLAN_TRAJ component. Finally, the movement of the robot arm may be managed according to the control function (CNTR_8) of the head portion (8) of the robot arm (6). If one or more optical devices are mounted on the head portion of the robot arm (6), automatic control of the robot arm may be required to activate the second optical unit.

[0230] According to one embodiment, the optical system (5) is configured to control each movement of the body (1) using a control loop. Through this control, it is possible to compare the difference between an image collected at a specific moment and a new image collected subsequently. One interest of this control is to modify the guiding of the robot arm (6) by adapting the starting path of the robot arm (6) to a new trajectory. To this end, a collision detection function (DET_COLLI) is executed to adapt the moving movement of the robot arm (6). The generation of the movement of the robot arm (6) is determined by the GEN_TRAJ function.

[0231] According to one embodiment, the head portion of the robot arm (6) is detachable so that the robot arm can be used for other purposes. The head portion, which includes one or more optical devices for dermatological use, may be replaced with a head portion that includes a laser for treating the skin surface, such as, for example, hair removal, tattoo removal, or Ultherapy for lifting.

[0232] Depending on the intended use, specific optical settings may be preferably selected.

[0233] According to one example, an array of optical parts having different focal points may be implemented at the head end of the robot arm (6). Depending on the embodiment, these optical parts may be connected by joints, or may move independently or together.

[0234] According to one example, at least one optical unit is a confocal microscope system. According to one example, the focal plane can be displaced to acquire images at different depth levels. A local three-dimensional image of the skin surface can be reconstructed. Such an application may involve reconstructing the volume of an object, such as a point, on the surface of the skin.

[0235] When using a confocal optical unit, illumination can be obtained from a light source such as a laser.

[0236] According to another example, at least one optical unit may be configured to acquire a series of cross-sectional images to perform tomographic technology. Depending on the given configuration, this technology may correspond to the type of optical coherence tomography (OCT). This technology often involves coupling between a sensor and a transmitter, such as a laser transmitter.

[0237] Another possible application relates to treatment by ultrasonic emission. For this purpose, at least one ultrasonic transmitter may be positioned at the level of the robot arm head. This transmitter is connected to a sensor to enable the measurement of echo signals or changes in the transmitted waveform. According to one example, this may be a transmitter matrix organized to generate a beam focused at a single point. The output may be configured to burn tissue or apply negative pressure to the skin surface to measure the elasticity of specific parts of the skin.

[0238] According to one embodiment, the transmitter may be configured to treat a subcutaneous area. Target point (P TARGET An image point can be defined by referring to ), and said image point is located on the surface of the body model. The image point is, for example, a target point (P TARGET It can be located several millimeters or even several centimeters below the skin surface along the normal of the model surface passing through it.

[0239] Guiding for the transmitter is the target point (P TARGET It can be performed according to the configured image point. Alternatively, if the image point is not defined relative to the target point (Ptarget), the transmitter's guiding can be performed directly by defining a point within the reachable space and defining an image point located beneath the patient's skin. Target point (P TARGET ) is used to collect images or plan the modified trajectory of a robot.

[0240] The head of the robot arm is also provided with a sensor to enable the investigation of mechanical properties such as skin elasticity, firmness, moisture content, or roughness.

[0241] According to another example, a fluorescent sensor is placed on the base of the robot arm (6). Such a sensor allows for monitoring, for example, a biological active ingredient for cosmetic use. According to another example, molecular characterization can be performed using spectroscopic techniques.

[0242] Dermatological use

[0243] According to one embodiment, the head portion (8) of the robot arm (6), also referred to as an "actuating device," includes an optical system for use in dermatological applications. In this case, a pair of optical devices may be provided. The first optical device (83, 84) includes a camera that performs a so-called "macroscopic" scan by the movement of the robot arm (6). Such a camera may perform image capture of interest on the surface of the skin. According to one example, a light source may be used to illuminate the captured area. The emitted light may be, for example, visible light, infrared light, or a fluorescent light source. According to one example, an LED system including a polarizer may be configured to emit polarized light. In the case of polarized light, a polarizing filter is applied to the LED-type light source. In that example, the camera provided at the end of the head portion of the robot arm is equipped with a polarizing filter. According to one example, the light source's filter and the camera are configured orthogonally.

[0244] When a peculiar part is detected, the robot arm (6) is controlled to modify the trajectory. The process of positioning the head portion on one point of the detected peculiar part, for example, the center or center of gravity of the area forming the peculiar part, is performed. Thus, guiding is stopped, and the starting trajectory is temporarily suspended.

[0245] The control unit activates the second optical device (85) to capture a millimeter or microscopic image with dermatological resolution. To obtain excellent resolution, the optical device (85) can be controlled to move along an axis (86) perpendicular to the plane of one of the optical units (83, 84, 85).

[0246] FIG. 4 shows a portion (9) of the body surface of a patient (1) and a trajectory (30) on said surface. The trajectory (30) can be obtained from a scanning strategy of a robot arm (6). According to one example, the generated trajectory (30) is obtained by a planning module (PLAN_TRAJ). In the example of FIG. 4, three dermatological specific parts (91) appear on a portion (9) of the skin of the patient (1). These may be, for example, scars, melanoma, moles, or other types of specific parts on the skin surface. When the optical field (87) of the first optical device (83, 84) crosses the specific part (91), the collected image is dynamically analyzed to automatically control the robot arm (6).

[0247] Accordingly, an image of a close-up area of ​​the patient (1) skin surface can be obtained through the second optical device. In one example, the second optical device (85) is a dermatoscope. The second optical device (85) may include a glass plate in contact with the skin. In one embodiment, the second optical device may include a polarization transmitter for obtaining depth image information of the skin epidermis.

[0248] To this end, the last branch (66) or actuation device (81) of the robot arm (6) may be equipped with a force feedback sensor to brake, stop, or stabilize the movement of the second optical device (85). When the second optical device (85) comes into contact with the skin of the patient (1), shot triggering may occur.

[0249] According to another example, each axis of the robot is equipped with a force sensor. The force value measured by the sensor is analyzed, and the pressure applied to the surface is automatically controlled using this information. Thus, the robot can automatically control the contact pressure between the work surface and the tool due to the torque sensors equipped on each axis, in which case the work surface is the patient's skin. This can also be used to "slide" the tool across the work surface.

[0250] According to one embodiment, a scanning device (87) is activated to remove hair from the optical unit (85). This scanning device (87) may be executed automatically while stabilizing the optical unit of the second optical unit (85). One concern is to prevent hair located near a specific part where a capture image is to be taken from being captured in the image. According to one embodiment, the scanning device (87) includes a rotating rod that activates a flexible brush proximal to the optical unit (85). One advantage is that the hair is rotated so that it is no longer between the optical unit (85) and the skin. According to one embodiment, a plurality of scanning devices (87) are placed around the optical unit (85). FIG. 6b illustrates two such devices.

[0251] After the second optical unit captures one or more images with a more precise resolution of the specific part, the second device (85) can perform a retraction movement to return to its original position. Accordingly, the robot arm can resume its path and return to the planned guiding trajectory.

[0252] According to one embodiment, guiding is performed at a distance of 1 to 20 cm from the body surface of the patient (1). According to one preferred embodiment, guiding is performed at a distance of 2 to 8 cm from the skin surface.

[0253] According to one embodiment, the field of view of the optical unit can be obtained at a specific distance from the robot surface, as the characteristics of the optical unit are known. Through these characteristics, the part of the complex surface of the patient's body that the collected image includes can be identified. One advantage is that the image can accurately correspond to the three-dimensional surface of the patient's body. Additionally, as the collected image can be indexed against the generated parameterized body model (MOD_P), the image can be placed at a specific location on the model. Thus, a three-dimensional representation of the body can be archived along with the user-accessible image to indicate specific parts of the skin. The process of associating the collected image with the generated parameterized body model can be performed using meshing M1 or M2. If the meshing includes, for example, a triangle, the image can be segmented so that the triangle corresponds to the surface of the image.

[0254] According to one embodiment, the present invention includes the step of referencing a captured image to a set of nodes of a meshing. According to one example, each triangle of the meshing includes an identifier. Accordingly, the image may be stored together with data that associates the image with a set of identifiers.

[0255] According to one embodiment, images collected by the optical units (83, 84, 85) are stored in memory (e.g., memory of the motion control unit of the robot arm (6)). According to one alternative, the operating device (8) includes an electronic board provided with at least one computational unit and memory. The collected images may first be stored locally before being transmitted to a remote electronic device.

[0256] According to one embodiment, photographs taken simultaneously by optical units 83 and 84 are associated with each other. The collected images may be associated, for example, with a point in the space of a reference point (R0) and a time stamp. The images may also be associated with a point that is referenced to a reference frame (R3) connected to the actuation device of the robot arm (6) or to a point that is referenced to a trajectory (30).

[0257] When an image is collected by the second optical device (85), the image may be associated with an image collected by the first optical device (83, 84), in particular, an image collected at a location on the same trajectory (30) as the location where the operating device (8) activated the movement of the second optical device (85).

[0258] One advantage of the present invention is that each collected image can be indexed into an image on a parametric model generated by a regression step (REG). Thus, the present method enables locating the position of a portion of an image on the skin and indexing it into the generated model. Another advantage is that it preserves an accurate trace of the captured area with respect to the geometric structure of the body. Therefore, when a patient changes location or revisits later, a new plotting of the already indexed region of interest for the newly collected image can be immediately generated using the indexing of the stored photos.

[0259] One interest is to construct an image bank with various resolutions for different parts of the body surface being considered.

[0260] In order to obtain an excellent response from the robot arm (6) while real-time collection of images captured by the operating device is being performed, the collected images can be compared with a characteristic image bank. According to another example, classification of the collected images can be performed using a learned neural network, and commands can be generated based on the result of the classification. For example, when a specific peculiarity is detected, there is no need to activate the second optical device (85), whereas when a different peculiarity is detected, image capture with a higher resolution is required.

[0261] According to another embodiment, the robot arm (6) performs a first trajectory (30) planned according to the range of images collected by the first device (83, 84). Next, the robot arm (6) performs a second image capture for a point where a peculiar part is detected. When the first trajectory (30) is performed to guide the robot arm (6) to a part selected by the user and the set of images collected by the first optical device is realized, the robot arm (6) performs a second trajectory (31). The trajectory may be automatically generated from the peculiar point identified while the first trajectory (30) is performed. To collect a second series of images from the identified point, the second device (85) is automatically activated at a characteristic point of the second trajectory (31).

[0262] Laser application

[0263] According to other application examples, the robot arm (6) includes a laser transmitter in its base. The laser transmitter may be placed, for example, in the base of the operating device (8). The laser transmitter may be used for hair removal, treatment of certain skin conditions, or tattoo removal. Additionally, the robot arm including the laser light source may be used to perform Ultherapy treatment or other laser-based skin treatments for medical or cosmetic purposes. To this end, the settings of the laser light source for parameters such as output, wavelength, pulsed emission, or continuous emission may be adapted according to the application.

[0264] In the case of laser light source operation, meshing (M2) is advantageously used to optimize the settings of the laser shots, particularly to manage shot output. In practice, for example, a uniform meshing including an equilateral triangle can be used to define a meshing of circles circumscribed or inscribed within the triangle. One advantage of a circular meshing circumscribed within the triangle is that it can define the overlap between different circles, that is, the overlap between shots. Due to this configuration, it is possible to define a shot target area adjacent to the target area under consideration. One interest in constructing tangent circles directly derived from the uniform meshing is to generate a target laser shot area so as to manage parameters regarding the degree of uniform overlap between two consecutive shots toward adjacent targets.

[0265] The present invention makes it possible to manage the degree of overlap to compensate for the non-uniformity of the amount of laser beam energy around the principal shot axis. Thus, by using a uniform type of meshing, a substantially uniform amount of energy can be emitted over a specific area.

[0266] When multiple laser emittings are performed to achieve the purpose of treating an area of ​​the skin, the guiding of the robot arm (6) can be set to be fixed at a set of points according to a given trajectory plan, thereby enabling continuous shots. The trajectory plan (PLAN_TRAJ) is determined using a user interface (UI). The trajectory of the robot arm (6) can be adapted in real time by taking into account, for example, data connected to the laser light source settings, the patient's movement, or collision detection of the robot arm.

[0267] The system of the present invention includes means for defining parameters of a laser shot, such as the shot output and / or diameter to reach the skin surface, and the degree of overlap. To achieve this purpose, a laser transmitter and a guiding trajectory may be established.

[0268] According to one embodiment, the resolution of the uniform meshing can be adapted according to a given shot setting. Generally, if the laser shot surface is defined according to the maximum radius to be reached on the skin surface, the meshing is set to define a circle having dimensions corresponding to the shot constraint.

[0269] In each example of application, while real-time guiding is being performed, a robot arm (6) moving along a given trajectory is induced to evaluate the trajectory in real time. This evaluation is performed by using a computational unit and executing the method of the present invention.

[0270] When the body is not moving, the image of the body surface recalculated to position the robot arm (6) includes only the rotation of the already calculated surface. That is, the transformation is equidistant and corresponds to one or more rotations of the robot arm based on the traveled path portion. The generated position of the body surface is generated for the position of the head portion of the robot arm.

[0271] When the body moves, the image of the body surface recalculated to position the robot arm (6) includes non-equidistant transformations, and the surface is recalculated according to the method of the present invention. In addition to considering potential rotations, it is possible to regenerate the body surface at the point in time of the robot arm head position by performing regression on the shape descriptor according to the method of the present invention.

[0272] According to one embodiment, particularly when a transmitter of the laser or ultrasonic type is placed on the head portion (8) of the robot arm (6), if the region of interest is smaller than a given surface threshold value, an optimization step for meshing of the surface of interest may be performed. This optimization may be performed so that the transformation within the region is considered equidistant. According to one embodiment, according to the method of the present invention, key points on the body surface to be considered for limiting the region of interest to which equidistant shape transformation is applied may be determined.

[0273] One advantage of the present invention is that it can be adapted for dermatological, surgical, and cardiac applications, etc.

Claims

Claim 1 A method for guiding a robot arm (6) in real time for processing body surface data comprises: a control unit generating a 3D body model (MOD_P) of the entire or part of a human body surface—the 3D body model (MOD_P) includes meshing of points (M1, M2), and the 3D body model (MOD_P) is generated from images collected by a first image acquisition system—; the control unit planning a treatment trajectory (TRAJ1) on the surface (MOD_S) of the 3D body model (MOD_P) using a calculator; the control unit calculating a guiding trajectory (TRAJ2) for the distal end of the robot arm (6), wherein the robot arm (6) is movable relative to the first image acquisition system (5); and the control unit [for a head portion] disposed at the distal end of the robot arm (6). A step of generating a kinematic (PILOT_ROBOT) of the robot arm (6) so that a point is automatically controlled by the guiding trajectory (TRAJ2); a step in which the control unit activates the kinematic of the robot arm (6) to move according to the guiding trajectory (TRAJ2) - the guiding trajectory (TRAJ2) is calculated in real time from the 3D body model (MOD_P) of the human body surface -; a step in which the control unit generates a new 3D body model (MOD_P) of the human body surface at regular time intervals to automatically manage the displacement of the robot arm (6) in real time, and calculates a new kinematic of the robot arm (6) accordingly - as the new surface (MOD_S) of the new 3D body model (MOD_P) is calculated, a new treatment trajectory (TRAJ1') and a new guiding trajectory (TRAJ2') are calculated -; the control The unit, target point (P) in space TARGET A step of receiving at least one location of ); a step in which the control unit enables the collection of a local near-field image of an area including the target point by at least one sensor of the transmitter of the actuator (8) - wherein the actuator (8) is positioned at the end portion of the robot arm (6), and the axial direction of the sensor is the target point (P TARGET A guiding method comprising: activation of the collection is performed when it coincides with a predefined straight line passing through ), and the target point is referenced on the generated 3D body model (MOD_P). Claim 2 A guiding method according to claim 1, characterized in that the treatment trajectory (TRAJ1) is generated within a treatment area on the surface (MOD_S) of the 3D body model (MOD_P). Claim 3 A guiding method according to claim 2, wherein the treatment trajectory (TRAJ1) is automatically generated by defining a treatment area (34, 35) on the surface (MOD_S) of the 3D body model (MOD_P). Claim 4 A guiding method according to claim 1, wherein the guiding trajectory (TRAJ2) is calculated from predetermined distance and direction instructions to generate the relative position of the original part of the robot arm (6) with respect to the treatment trajectory (TRAJ1) and the new treatment trajectory (TRAJ1'). Claim 5 A guiding method according to claim 1, wherein the robot arm (6) comprises at least three sets of branches (61, 62, 63, 64, 65, 81) connected in pairs by joints, each set of branches being connected to each other by pivot links (612, 623, 634, 645, 656), and each set of pivot links comprising a kinematic planned for a given guiding trajectory (TRAJ2) and a generated 3D body model (MOD_P). Claim 6 A guiding method according to claim 1, wherein the control unit generates the new 3D body model (MOD_P) at regular time intervals so that the robot arm (6) can be controlled in real time as the new guiding trajectory (TRAJ2') of the robot arm (6) is generated - wherein the new treatment trajectory (TRAJ1') is modified as the new surface (MOD_S) of the new 3D body model (MOD_P) is calculated - and further comprises the step of the control unit calculating a new kinematic of the robot arm (6) that is automatically controlled by the defined new guiding trajectory (TRAJ2'). Claim 7 A guiding method according to claim 1, characterized in that the control unit includes the step of calibrating the position of the robot arm (6) from a fixed position where an image is collected so as to generate the 3D body model (MOD_P). Claim 8 A guiding method according to claim 1, characterized by establishing a laser beam shot strategy of a laser transmitter fixed to the operating device (8) using the treatment trajectory (TRAJ1) and the new treatment trajectory (TRAJ1'). Claim 9 In claim 8, the body model comprises a homogeneous meshing (M2) for generating a new meshing (M3) defined by an array of circles (38), and at least one of the circles is a target point (P) in space. TARGET A guiding method characterized by defining the position of ). Claim 10 A guiding method according to claim 9, wherein the new meshing (M3) is generated according to at least one of a shot power instruction and an instruction regarding the degree of overlap between two shots. Claim 11 A guiding method according to claim 10, wherein the sampling of the number of shots performed is based on the new meshing (M3), the set shot output, and the degree of overlap defined between two adjacent (colocalized) circles. Claim 12 A guiding method according to claim 1, characterized by establishing an ultrasonic signal shot strategy of an ultrasonic transmitter fixed to the operating device (8) using the treatment trajectory (TRAJ1) and the new treatment trajectory (TRAJ1'). Claim 13 In claim 1, one set of images is obtained from optical devices (83, 84, 86) disposed in the operating device (8), and each image (IM1, IM2) is the target point (P) on the 3D body model (MOD_P). TARGET A guiding method characterized by being indexed to at least one node or mesh of the meshing (M1, M2) of the 3D body model (MOD_P) according to the position of ) and the viewing angle of the optical device (83, 84, 86). Claim 14 In paragraph 13, at least one pair of images is simultaneously collected from an optical device (83, 84) comprising two sensors embedded in the operating device (8) and positioned on different axes of the operating device (8), wherein each image (IM1, IM2) is the target point (P) on the 3D body model (MOD_P). TARGET A guiding method characterized by being indexed to at least one node or mesh of the meshing of the 3D body model (MOD_P) according to the position of ) and the viewing angle of the optical device (83, 84, 86), and the image being processed to reconstruct a 3D local image. Claim 15 A guiding method according to claim 14, characterized in that the above-mentioned three-dimensional local image is superimposed as a 3D representation on a generated body model. Claim 16 A guiding method according to claim 1, wherein a first image collected by the first optical device (83, 84) of the operating device is compared with an image set in real time, and the comparison operation includes, if applicable, generating an indicator of interest, and can generate a digital control command for operating the robot arm (6) due to the indicator of interest. Claim 17 In paragraph 16, by means of a guiding instruction, (i) a first operation of the robot arm (6) at the target location where the first image is extracted and (ii) a second operation of the actuator (8) to activate the second optical device (85) are performed, and due to the activation of the second optical device (85), the target point (P TARGET A guiding method characterized by obtaining a second image within a near-field of a region including ). Claim 18 A guiding method according to claim 16, characterized in that the above comparison operation is the classification of the collected images by a neural network trained on an image set. Claim 19 delete Claim 20 A guiding system for a robot arm (6), characterized by comprising: a device for collecting images (5) of a patient's body to obtain a 3D body model (MOD_P); and a robot arm (6) having an operating device (8) for executing the method of any one of claims 1 to 18.

Images