A vision-based robot self-modeling method, system, device, and medium

Abstract

The application discloses a vision-based robot self-modeling method, system, device and medium, comprising: acquiring initial pictures of various joint postures of a robot and corresponding joint angles; performing data preprocessing on the initial pictures to obtain first pictures; acquiring camera information of the initial pictures; performing robot self-modeling based on a three-dimensional Gaussian splatting algorithm according to the first pictures and the camera information of the robot in a standard state; constructing and training an initial deformation neural network to obtain a target deformation neural network; wherein the initial deformation neural network is used to change ellipsoid parameters of a three-dimensional Gaussian splatting model according to joint angles and camera information corresponding to each first picture of a random posture; and combining the three-dimensional Gaussian splatting model and the target deformation neural network into a robot model to complete robot self-modeling. The application does not need a depth camera, can reduce data acquisition difficulty, improve robot self-modeling precision, and can be widely applied in the technical field of robots.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and particularly relates to a robot self-modeling method and system based on vision, a device and a medium. BACKGROUND

[0002] Robots are important tools for assisting and replacing human labor, and can reduce burden and improve efficiency in various fields. A robot model is the basis for controlling a robot. According to the robot model, the robot can work in various different environments and complete various different tasks.

[0003] In related technologies, the self-modeling technology of a robot has problems such as high cost of training data acquisition, low quality of self-modeling results due to the influence of data acquisition accuracy, and low degree of freedom range of a self-modeling robot, which limits the use range. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a robot self-modeling method and system based on vision, a device and a medium, which can realize low-cost data acquisition, high model accuracy and wide application range of robot self-modeling.

[0005] To achieve the above purpose, one aspect of the embodiments of the present application provides a robot self-modeling method based on vision, which comprises the following steps.

[0006] Obtain initial pictures of various joint postures of a robot, and obtain joint angles corresponding to the initial pictures; wherein the joint postures include a standard state and a random posture;

[0007] Perform data preprocessing on the initial pictures to obtain first pictures;

[0008] Obtain camera information of a camera for shooting the initial pictures;

[0009] According to the first pictures and the camera information of the robot in the standard state, perform robot self-modeling based on a three-dimensional Gaussian splash algorithm to obtain a three-dimensional Gaussian splash model;

[0010] Construct and train an initial deformation neural network to obtain a target deformation neural network; wherein the initial deformation neural network is used to change ellipsoid parameters of the three-dimensional Gaussian splash model according to the joint angles and the camera information corresponding to each first picture of the random posture;

[0011] Combine the three-dimensional Gaussian splash model and the target deformation neural network into a robot model to complete the self-modeling of the robot.

[0012] In some embodiments, the data pre-processing of the initial picture to obtain the first picture comprises the following steps:

[0013] The image segmentation algorithm is used to segment the initial picture to obtain the intermediate picture including only the robot part;

[0014] According to the preset length and width, the scaling processing is performed on all the intermediate pictures to obtain the first picture.

[0015] In some embodiments, the camera information of the camera for shooting the initial picture comprises the following steps:

[0016] The input camera information is obtained;

[0017] Alternatively, the camera information is extracted according to the initial picture.

[0018] In some embodiments, the camera information is extracted according to the initial picture, comprising the following steps:

[0019] The initial pictures of each camera when the robot is at the same joint angle are selected to obtain the current joint angle picture;

[0020] The camera position and camera parameter of each camera are extracted according to the current joint angle picture by the motion-in-structure algorithm to obtain the camera information.

[0021] In some embodiments, the robot self-modeling based on the three-dimensional Gaussian splash algorithm according to the first picture and the camera information when the robot is in the standard state to obtain the three-dimensional Gaussian splash model comprises the following steps:

[0022] The Gaussian ellipsoid is randomly generated in the three-dimensional space to obtain the Gaussian ellipsoid parameter; wherein the Gaussian ellipsoid parameter comprises position, radius, rotation attitude, feature and transparency;

[0023] According to the camera parameter in the camera information, the imaging plane corresponding to the camera is obtained;

[0024] According to the distance from the Gaussian ellipsoid to the imaging plane, the Gaussian ellipsoid is projected from far to near to the imaging plane;

[0025] According to the transparency of each Gaussian ellipsoid, the occlusion degree is determined;

[0026] According to the feature parameter, the projection color is calculated by the spherical harmonic function;

[0027] According to the projection result, the occlusion degree and the projection color, the second picture is generated;

[0028] calculating a first error of the second picture and the first picture;

[0029] repeating the step of obtaining the imaging plane corresponding to the camera according to the camera parameter in the camera information until the first error meets a preset first condition, to obtain a three-dimensional Gaussian splash model.

[0030] In some embodiments, the constructing and training of the initial deformation neural network to obtain the target deformation neural network comprises the following steps:

[0031] initializing a skeletal structure according to the three-dimensional Gaussian splash model;

[0032] locking ellipsoid parameters of the three-dimensional Gaussian splash model in a standard state to initialize the initial deformation neural network;

[0033] training the initial deformation neural network once to obtain a first deformation neural network under the condition of locking the ellipsoid parameters and limiting the amplitude of the robot pose change to be less than a preset amplitude value;

[0034] training the first neural network once to obtain a second deformation neural network under the condition of locking the ellipsoid parameters and not limiting the amplitude of the robot pose change;

[0035] completely training the second deformation neural network to obtain a target deformation neural network under the condition of unlocking the ellipsoid parameters.

[0036] To achieve the above object, another aspect of the embodiment of the present application proposes a robot self-modeling system based on vision, which comprises:

[0037] A first module is configured to obtain initial pictures of various joint poses of a robot and obtain joint angles corresponding to the initial pictures, wherein the joint poses include a standard state and a random pose.

[0038] A second module is configured to perform data preprocessing on the initial pictures to obtain first pictures.

[0039] A third module is configured to obtain camera information of a camera used to shoot the initial pictures.

[0040] A fourth module is configured to perform robot self-modeling based on a three-dimensional Gaussian splash algorithm according to the first pictures and the camera information of the robot in a standard state to obtain a three-dimensional Gaussian splash model.

[0041] The fifth module is configured to construct and train an initial deformation neural network to obtain a target deformation neural network; wherein the initial deformation neural network is configured to change the ellipsoid parameters of the three-dimensional Gaussian splatting model according to the joint angles and the camera information corresponding to each of the first pictures in the random poses;

[0042] The sixth module is configured to combine the three-dimensional Gaussian splatting model and the target deformation neural network into a robot model to complete the self-modeling of the robot.

[0043] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method described above when executing the computer program.

[0044] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0045] The embodiment of the present application at least has the following beneficial effects: the present application provides a robot self-modeling method, system, device and medium based on vision, which performs three-dimensional Gaussian splatting algorithm modeling on a robot in a standard state by obtaining color pictures of various joint poses of the robot, and trains a deformation neural network according to robot pictures in random poses to change the parameters of the three-dimensional Gaussian splatting model, so that the robot self-modeling can be completed without a depth camera, the data acquisition difficulty is reduced, the modeling accuracy is improved, and the robot self-modeling can be applied to a high degree of freedom robot. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings are used to provide a further understanding of the technical scheme of the present application, and constitute a part of the specification, and are used to explain the technical scheme of the present application together with the embodiments of the present application, and do not constitute a limitation to the technical scheme of the present application.

[0047] Figure 1 is a schematic diagram of a triangular network model provided by the embodiment of the present application;

[0048] Figure 2 is a schematic diagram of a point cloud model provided by the embodiment of the present application;

[0049] Figure 3 is a schematic diagram of a voxel model provided by the embodiment of the present application;

[0050] Figure 4 is a step diagram of a robot self-modeling method based on vision provided by the embodiment of the present application;

[0051] Figure 5 is a flowchart of a robot self-modeling method based on vision provided by an embodiment of the present application;

[0052] Figure 6 is a schematic diagram of a 3DGS algorithm principle provided by an embodiment of the present application;

[0053] Figure 7 is a schematic diagram of a deformable neural network provided by an embodiment of the present application;

[0054] Figure 8 is an example diagram of a standard state robot self-modeling result provided by an embodiment of the present application;

[0055] Figure 9 is an example diagram of a robot self-modeling result based on vision provided by an embodiment of the present application;

[0056] Figure 10 is a schematic diagram of controlling robot motion according to a self-modeling result provided by an embodiment of the present application;

[0057] Figure 11 is a schematic diagram of a robot self-modeling system based on vision provided by an embodiment of the present application;

[0058] Figure 12 is a schematic diagram of a robot self-modeling system based on vision provided by an embodiment of the present application;

[0059] Figure 13 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application, but is only an example of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0061] Although the functional modules are divided in the system schematic diagram, and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the system or the sequence in the flowchart. The terms "first / S100", "second / S200", etc. in the specification and claims and the above drawings are used to distinguish similar objects and do not necessarily describe a specific sequence or order.

[0062] It can be understood that the terms "first", "second", etc. used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon determining" or "in response to determining".

[0063] The terms "at least one", "multiple", "each", "any", etc. used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0064] Reference to "embodiments" herein means that the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily all refer to the same embodiments, nor is it necessarily independent or alternative embodiments to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0066] In the related art, the expression of a three-dimensional model can be divided into two kinds of display expression and implicit expression. As shown in Figure 1 、 Figure 2 and Figure 3 , display expression generally adopts three kinds of ways of triangular mesh, point cloud and voxel. Triangular mesh uses a series of vertices in space and connecting lines between vertices to form a plurality of small triangular planes. Point cloud is a collection of a large number of points in three-dimensional space, which collectively form the model of an object. The voxel expression of a three-dimensional model can be compared with a two-dimensional image. A picture is usually composed of individual pixel points. A three-dimensional model can also be composed of a number of small cubes stacked in space.

[0067] The implicit expression of a three-dimensional model often uses a symbolic distance function. The symbolic distance function is represented by the following mapping relationship:

[0068]

[0069] Wherein, x is a three-dimensional vector, representing the coordinates of a point in space. s represents the nearest distance of point x to the surface of the three-dimensional model, and SDF represents the signed distance function. When point x is inside the three-dimensional model, s is negative; when point x is outside the three-dimensional model, s is positive; when point x is exactly on the surface of the model, s is 0, and these points on the surface of the model are also called level sets. Given a three-dimensional model expressed by a signed distance function, when the specific shape of the model is needed, the coordinates of the points in space are substituted into the function one by one to calculate, and all points with output 0 form the surface of the model, and all points with negative output form the internal entity of the model.

[0070] Some schemes are used to establish the signed distance function of the robot, which learns a neural network according to a large amount of collected data through deep learning, and uses the neural network as the signed distance function of the robot. When the points in space are input into the trained neural network, all points with output 0 form the surface of the robot model. When collecting data, several depth cameras that can capture color images and depth images at the same time are used to capture the robot from different directions. During the shooting process, the robot constantly changes the pose, records the joint angles of the robot corresponding to each pose; and according to the data collected by the camera, the point cloud forming the surface of the robot in the current pose is obtained, and the coordinates of all points forming the point cloud in space are recorded. These point clouds will be used as the true value needed in the subsequent neural network learning. When training the neural network, the joint angles of the robot and the coordinates of the points in space are input into the neural network at the same time. The angle value is randomly selected from all the angle values recorded when collecting data; the points in space are divided into "on the surface" and "not on the surface" two parts, one part is selected from the point cloud corresponding to the angle value collected, and the other part is randomly generated, but the coordinates of the points obtained in the first part are different. The output of the neural network should be 0 when the former part of points and joint angles are input into the neural network, and the output of the neural network should not be 0 when the latter part of points and joint angles are input into the neural network. Through this way, the neural network is supervised to learn, so as to obtain a neural network that can approximate the signed distance function of the robot, and realize the self-modeling of the robot.

[0071] However, the above related technology has the following shortcomings:

[0072] First, the cost of training data collection is high. Since the point cloud of the robot in different joint angle poses is needed when training the neural network, a depth camera that can obtain depth data is needed when collecting data, and shooting is needed from multiple angles. Depth camera is a professional equipment, compared with ordinary camera everywhere, the cost of acquisition is higher, the holding rate is lower, which hinders the application of the prior art.

[0073] Secondly, the self-modeling of the above technology is affected by the data collection accuracy and has low quality. Since the above method uses a depth camera to obtain point cloud as a real sample for neural network training, the fineness of the collected point cloud will directly affect the fineness of the model obtained by the final modeling. On the one hand, the point cloud is synthesized using the depth information collected by the depth camera, and the accuracy of the depth information of the depth camera is limited and related to the cost of the depth camera. On the other hand, the camera and the robot itself will inevitably have slight shaking when shooting, which will interfere with the collection of depth information and affect the quality of the obtained point cloud.

[0074] Thirdly, the above technology only realizes the self-modeling of a mechanical arm with 4 degrees of freedom (4 joints), which limits the use range of the technology. The higher the degree of freedom (the more the number of joints) of the robot, the more difficult the self-modeling. A method is needed to model the robot with high degree of freedom.

[0075] Therefore, in the embodiments of the present application, a robot self-modeling method, system, device and medium based on vision are provided. The scheme can be applied to various forms of robots. Taking a robot in the form of a common mechanical arm as an example, the mechanical arm is usually composed of multiple rigid links connected by rotating joints or translating joints. The model established in the embodiments of the present application mainly includes a physical model and a kinematic model of the robot. The physical model of the robot refers to the physical form of the robot, i.e. the three-dimensional model of the robot. The kinematic model describes the mapping relationship between the angles of each joint of the robot and the position and posture of a certain module or the entire robot in space. The scheme mainly obtains pictures of the robot in various joint postures taken at different fixed positions, obtains camera position and parameter information, constructs a three-dimensional Gaussian splat model in a standard state, trains a deformation neural network, changes the three-dimensional Gaussian splat model in the standard state according to the joint angles to reflect the posture of the robot at different joint angles, and finally obtains the robot self-modeling result including the three-dimensional Gaussian splat model and the deformation neural network.

[0076] The embodiment of the present application provides a kind of robot self modeling method based on vision, it is related to robot technical field.The robot self modeling method based on vision provided in the embodiment of the present application can be applied in terminal, also can be applied in server, it can also be software running in terminal or server.In some embodiments, terminal can be smart phone, tablet computer, notebook computer, desktop computer, smart speaker, smart watch and vehicle terminal etc., but not limited to this;Server end can be configured as independent physical server, can also be configured as the server cluster or distributed system of multiple physical servers, can also be configured as the cloud server of providing cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN and basic cloud computing services such as big data and artificial intelligence platform, server can also be a node server in blockchain network;Software can be the application of realizing a kind of robot self modeling method based on vision etc., but not limited to the above form.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computer, server computer, handheld device or portable device, tablet device, multi-processor system, microprocessor-based system, set-top box, programmable consumer electronics, network PC, small computer, mainframe computer, distributed computing environment including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0078] Figure 4 is an optional flowchart of the robot self modeling method based on vision provided by the embodiment of the present application, Figure 1 The method in the above embodiment can include but not limited to including steps S100-S600.

[0079] Step S100, obtain the initial picture of various joint postures of robot, and obtain the joint angle corresponding to the initial picture;Wherein, the joint posture includes standard state and random posture.

[0080] Step S200, data preprocessing is carried out to the initial picture, and first picture is obtained.

[0081] Step S300, obtain the camera information of the camera for shooting the initial picture.

[0082] Step S400, according to the first picture and the camera information that the robot is in the standard state, based on the three-dimensional Gaussian splash algorithm, the robot self-modeling is carried out, and a three-dimensional Gaussian splash model is obtained.

[0083] Step S500, an initial deformation neural network is constructed and trained to obtain a target deformation neural network; wherein the initial deformation neural network is used to change the ellipsoid parameters of the three-dimensional Gaussian splash model according to the joint angles and the camera information corresponding to each first picture of a random pose.

[0084] Step S600, the three-dimensional Gaussian splash model and the target deformation neural network are combined into a robot model, and the self-modeling of the robot is completed.

[0085] The steps S100 to S600 shown in the embodiments of the present application, by acquiring color pictures of various joint poses of the robot, the three-dimensional Gaussian splash algorithm modeling of the robot in the standard state is carried out, and a deformation neural network is trained according to the robot pictures of random poses, the parameters of the three-dimensional Gaussian splash model are changed, the self-modeling of the robot can be completed without depth camera, the data acquisition difficulty is reduced, the modeling accuracy is improved, and the self-modeling of the robot with high freedom can be applied.

[0086] In some embodiments, step S100 can include but is not limited to steps S110-S140:

[0087] Step S110, the robot is placed in a standard state, and a fixed camera is used to take pictures of the robot to obtain a standard state picture.

[0088] Step S120, the joint pose of the robot is randomly adjusted, and a fixed camera is used to take pictures of the robot to obtain a random pose picture.

[0089] Step S130, the standard state picture and the random pose picture are used as initial pictures.

[0090] Step S140, record the joint angle corresponding to the initial picture.

[0091] In some embodiments, step S200 includes but is not limited to steps S210-S220:

[0092] Step S210, an image segmentation algorithm is used to segment only the part of the robot from the initial picture to obtain an intermediate picture.

[0093] Step S220, according to the preset length and width, all the intermediate pictures are scaled to obtain a first picture.

[0094] In some embodiments, step S300 comprises, but is not limited to, step S310 or step S320.

[0095] Step S310, obtaining input camera information.

[0096] Step S320, extracting camera information according to the initial picture.

[0097] In some embodiments, step S320 comprises, but is not limited to, steps S321-S322.

[0098] Step S321, selecting initial pictures taken by each camera when the robot is at the same joint angle, to obtain a current joint angle picture.

[0099] Step S322, extracting camera positions and camera parameters of each camera according to the current joint angle picture by a motion-in-the-middle structure extraction algorithm, to obtain camera information.

[0100] In some embodiments, step S400 comprises, but is not limited to, steps S410-S480.

[0101] Step S410, randomly generating Gaussian ellipsoids in a three-dimensional space range to obtain Gaussian ellipsoid parameters; wherein the Gaussian ellipsoid parameters include position, radius, rotation attitude, feature, and transparency.

[0102] Step S420, obtaining an imaging plane corresponding to a camera according to a camera parameter in the camera information.

[0103] Step S430, projecting the Gaussian ellipsoid to the imaging plane from far to near according to a distance from the Gaussian ellipsoid to the imaging plane.

[0104] Step S440, determining an occlusion degree according to a transparency of each Gaussian ellipsoid.

[0105] Step S450, calculating a projection color by a spherical harmonic function according to a feature parameter.

[0106] Step S460, generating a second picture according to a projection result, the occlusion degree, and the projection color.

[0107] Step S470, calculating a first error of the second picture and the first picture.

[0108] Step S480, repeatedly performing the step of obtaining an imaging plane corresponding to a camera according to a camera parameter in the camera information until the first error meets a preset first condition, to obtain a three-dimensional Gaussian splatting model.

[0109] In some embodiments, step S500 includes but is not limited to steps S510-S550.

[0110] Step S510, according to the three-dimensional Gaussian splash model, initialize the bone structure.

[0111] Step S520, lock the ellipsoid parameters of the three-dimensional Gaussian splash model in the standard state, initialize the initial deformation neural network.

[0112] Step S530, under the condition of locking the ellipsoid parameters and limiting the amplitude of the robot pose change to be less than a preset amplitude value, train the initial deformation neural network once to obtain a first deformation neural network.

[0113] Step S540, under the condition of locking the ellipsoid parameters and not limiting the amplitude of the robot pose change, train the first neural network once to obtain a second deformation neural network.

[0114] Step S550, under the condition of unlocking the ellipsoid parameters, train the second deformation neural network completely to obtain a target deformation neural network.

[0115] Next, in combination with a specific robot self-modeling scene application example, the scheme of the embodiments of the present application is described in detail:

[0116] As shown in the flowchart of the robot self-modeling method based on vision proposed by the embodiments of the present application, the specific steps are as follows: Figure 5

[0117] (1) Take pictures and pre-process the input pictures. When taking pictures, the robot in various joint angles should be taken from multiple different fixed positions. Select a certain angle combination as the standard state. For the robot in the standard state, a mobile phone or other handheld camera can be additionally used to take pictures. Extract the part containing only the robot from the pictures, exclude the background, and process all the pictures to the same size.

[0118] (2) Camera information extraction. The input of the embodiments of the present application is a large number of pictures of the robot in different joint angles and the joint angle values corresponding to the pictures. Whether to provide the position of the camera in space when taking these pictures can be selected. For the case of unknown camera and parameters, camera information extraction is needed. Specifically, when the robot is in the same joint angle, the pictures taken from each camera are selected from all the pictures, and the position and parameter information of these cameras are extracted through the structure from motion (SFM) algorithm. The parameter information includes the number of pixels in the length and width directions and the focal length in the length and width directions. ​

[0119] (3) For the robot in the pre-set standard state, a three-dimensional Gaussian splash algorithm (3DGS) is used to realize self-modeling of the robot in the standard state. From all the collected pictures, the picture of the robot in the standard state is selected, the selected picture and the camera position and standard state joint angle values when the picture is taken are input, and a three-dimensional Gaussian splash model of the robot in the standard state is established according to the above input contents.

[0120] Further, with reference to Figure 6 , the following steps ①-④ are included:

[0121] ① A large number of Gaussian ellipsoids are randomly generated in a space range. The Gaussian ellipsoid has five parameters of position (x, y, z), radius, rotation attitude, feature and transparency. The first three parameters determine the geometric characteristics of the Gaussian ellipsoid, the feature parameter is used to represent the color of the synthesized Gaussian ellipsoid at different viewing angles, and the transparency affects the transparency of the color of the Gaussian ellipsoid. When generated, the position and radius of the Gaussian ellipsoid are random, and the rotation attitude and feature are initialized to 0.

[0122] Based on this, the expression of the Gaussian ellipsoid is:

[0123]

[0124] In the above expression, x is the position of the Gaussian ellipsoid in space; ∑ is the covariance matrix.

[0125] The calculation formula of the covariance matrix Σ is:

[0126] Σ = RSS T R T ;

[0127] Where R is the rotation attitude of the Gaussian ellipsoid; S is the radius of the ellipsoid in three dimensions; T represents matrix transposition.

[0128] ② The standard state robot arm picture, picture corresponding camera position and parameters are input, and the 3DGS algorithm is used for standard state modeling. The modeling process of the 3DGS algorithm is: according to the camera parameters, the imaging plane corresponding to the camera is obtained; according to the distance of all Gaussian ellipsoids to the plane, the ellipsoids are projected onto the plane from far to near, and the ellipsoids in the near place will cover the ellipsoids in the far place, and the degree of occlusion is determined by the transparency; according to the feature parameters, the color at the projection place is calculated through the spherical harmonic function; finally, a picture is obtained, which reflects the attitude of the current three-dimensional Gaussian splash model under the input camera viewing angle.

[0129] ③ Compare the picture obtained in step ② with the original input picture, calculate the error, and use the Pytorch deep learning framework to automatically optimize the Gaussian ellipsoid parameters in step ① according to the error. Also, split the oversized Gaussian ellipsoid into two new Gaussian ellipsoids.

[0130] ④ Repeat the process of steps ② to ③ to continuously optimize the Gaussian ellipsoid parameters until the picture generated by the Gaussian ellipsoid is very similar to the input picture and the error no longer decreases significantly, completing the self-modeling in the standard state.

[0131] (4) Train the neural network to change the three-dimensional Gaussian splash model in the standard state according to the joint angles to reflect the posture of the robot at different joint angles. Input the joint angles corresponding to each picture and the camera position when taking the picture into the neural network, and the neural network will deform the three-dimensional Gaussian splash model. After training, a deformation neural network is obtained.

[0132] Further, the training of the neural network is mainly through the deformation of the standard state model by the deformation neural network to obtain the posture of the robot model under various joint angle combinations. Referring to Figure 7 , the basis of the deformation neural network is a number of bones that make up the robot arm, which are represented by ellipsoids, and are determined by position, rotation posture, and three radii of the ellipsoid. Taking a 7-DOF robot arm as an example, 8 ellipsoids and 8 bones are needed to represent it. The deformation network takes joint angles as input and outputs the transformation mode of the position and rotation posture of each bone at the corresponding joint angle. After determining the bone transformation mode, the deformation network calculates the movement and rotation mode of each Gaussian ellipsoid in the standard state model. According to the calculated movement and rotation mode, the model in the standard state is deformed to obtain the model of the robot arm at the corresponding joint angle. Specifically, the following steps ⑤-⑩ are included:

[0133] ⑤ Initialize the bone structure according to the three-dimensional Gaussian splash model obtained in step (3). Still taking a 7-DOF robot arm as an example, the entire structure can be composed of 8 ellipsoids, each of which is determined by its own position, rotation posture, and radius. All parameters can be automatically updated during training by the Pytorch deep learning framework. At initialization, the rotation postures of all ellipsoids are initialized to 0, i.e. no rotation has been performed. The radius of each ellipsoid is limited to 0.001 to between, the middle value is taken at initialization. The position of the ellipsoid is obtained from the distribution of the Gaussian ellipsoids that make up the three-dimensional Gaussian splash model. Use the K-means algorithm to input the positions of all Gaussian ellipsoids and cluster them into 8 categories. The position of each cluster center is used as the initialization position of the ellipsoid. At this point, all bones are initialized.

[0134] ⑥Initialize the deformation neural network. Lock the 3D Gaussian splash model in the standard state, so that the Gaussian ellipsoid parameters in it are no longer automatically updated, and stop the splitting and copying of the Gaussian ellipsoid. Input the joint angles in the state to the deformation neural network, and the neural network outputs the displacement and rotation of each bone in the form of a 4x4 rigid transformation matrix T. According to the rigid transformation matrix of each bone, calculate the movement and rotation of each Gaussian ellipsoid. The specific calculation method is to take the weighted sum of the rigid transformation matrices of the 8 bones, and the weight of each Gaussian ellipsoid relative to each bone is different. When calculating the weight, first calculate the Mahalanobis distance from each Gaussian ellipsoid to each bone. Since the shape of the robot arm is not exactly the same as the ellipsoid, in order to better reflect the details of the shape, a correction neural network is used to correct the Mahalanobis distance, so that the distance is more consistent with the distance from the Gaussian ellipsoid to the real robot arm. The calculation method of the Mahalanobis distance is as follows:

[0135]

[0136] where x is the position of the Gaussian ellipsoid, μ is the center of the bone defined in D1, and Σ is the covariance matrix representing the ellipsoid of the bone, which is calculated in the same way as the calculation formula in ③, and Σ -1 represents the inverse of the covariance matrix.

[0137] The complete calculation method of the weight is as follows:

[0138] W i,j =softmax(-10×scale×(D M (x i ,μ j )+MLP(x i )));

[0139] where scale is a scale parameter that can be automatically optimized by the Pytorch framework, D M (x i ,μ j ) represents the Mahalanobis distance from the i-th Gaussian ellipsoid to the j-th bone, and MLP represents the deformation neural network. The calculation method of the Softmax function is as follows:

[0140]

[0141] where w i is the weight of a Gaussian ellipsoid to the i-th bone, and there are k bones in total.

[0142] According to the movement and rotation of each Gaussian ellipsoid, the latest position and posture of each Gaussian ellipsoid are calculated, and a picture in a standard state is rendered according to the rendering process and training process of the 3DGS algorithm in step ③. The deformed neural network is trained in the standard state by comparing the rendered picture with the real picture, so as to complete the initialization of the deformed neural network.

[0143] ⑦For pictures with joint angles between -30° and 30°, input the joint angles, camera positions and parameters corresponding to the pictures, calculate the movement and rotation of all Gaussian ellipsoids by the deformed neural network according to the same method in D2, and render the Gaussian ellipsoids in the new position and posture into pictures according to the rendering process of the 3DGS algorithm. According to the error between the rendered picture and the real picture, automatically optimize each parameter in the deformed neural network and each parameter of the skeleton. The parameters of the Gaussian ellipsoid itself remain unchanged and are not segmented or copied. The deformed neural network is trained under the condition that the arm posture changes slightly.

[0144] ⑧Train the deformed neural network without limiting the amplitude of the arm posture change. All collected data and pictures are used for training. Input the joint angles, camera positions and parameters corresponding to the pictures, calculate the movement and rotation of all Gaussian ellipsoids by the deformed neural network according to the same method in D2, and render the Gaussian ellipsoids in the new position and posture into pictures according to the rendering process of the 3DGS algorithm. According to the error between the rendered picture and the real picture, automatically optimize each parameter in the deformed neural network and each parameter of the skeleton. The parameters of the Gaussian ellipsoid itself remain unchanged and are not segmented or copied.

[0145] ⑨Complete the training of the Gaussian splashing model and the deformed neural network. Thaw the three-dimensional Gaussian model in the standard state, and update the position, rotation posture, radius size, features and transparency of the Gaussian ellipsoid parameters simultaneously with the skeleton parameters in the deformed neural network and the weight parameters of the neural network. Input the joint angles, camera positions and parameters corresponding to the pictures, calculate the movement and rotation of all Gaussian ellipsoids by the deformed neural network according to the same method in D2, and render the Gaussian ellipsoids in the new position and posture into pictures according to the rendering process of the 3DGS algorithm. According to the error between the rendered picture and the real picture, automatically optimize all the above parameters.

[0146] ⑩Continuously train the entire network until the rendered picture is similar enough to the input real picture, and the picture error no longer decreases significantly, stop training, and complete the entire self-modeling process. The model obtained by self-modeling is represented by the Gaussian splashing model of the 3DGS algorithm and the deformed neural network, and can display the posture of the robot arm under different viewing angles and different joint angle combinations.

[0147] In the above process, picture processing, 3D GS algorithm modeling standard state model, and training of deformation neural network can be completed using Python. The latter two can be completed using the Pytorch deep learning framework. For cases where the camera position needs to be calculated, the calculation process can be completed using COLMAP software.

[0148] As Figure 8 shown, is an example of modeling the mechanical arm in the standard state and a comparison with the real mechanical arm. It can be seen that the method of the present application completes high-quality modeling of the mechanical arm in the standard state. Figure 9 The results of the model obtained by self-modeling under different joint angle postures are shown, as well as the morphology of the real mechanical arm under the same joint angle posture. Comparison shows that the self-modeling results well reflect the morphology of the mechanical arm under different joint angles. In addition, in order to show the practicability of the model obtained by the self-modeling method of the present application, Figure 10 an example of controlling the mechanical arm to complete a simple task according to the self-modeling results is shown. The task content is to avoid the red obstacle while making the end of the mechanical arm reach the green position. The experimental results show that the mechanical arm always keeps a certain distance from the red ball, Figure 11 the distance changes.

[0149] In summary, the present application has at least the following beneficial effects:

[0150] 1. The color pictures of the robot can be used to complete the reconstruction of three-dimensional information, without the need for professional equipment to collect depth information, making it more convenient for popularization. Secondly, the present application can more accurately restore the geometric structure of the robot, reducing the influence of external interference on data collection and the accuracy of data collection equipment on the final self-modeling results. Finally, the present application can model high-degree-of-freedom robots and solve the self-modeling problem of more types of robots.

[0151] 2. Through the method of the present application, the robot can be self-modeled only by pictures.

[0152] 3. And since only different angle color pictures taken by the robot at different joint postures are used as raw data, and any camera can be used during the shooting process, data collection is very convenient and low in cost.

[0153] 4. Since common cameras have sufficient accuracy, it is easy to take clear pictures, which can solve the problem of poor self-modeling quality caused by the limited accuracy of depth cameras. The self-modeling accuracy is higher.

[0154] 5. It can be applied to self-modeling of higher degree of freedom robots and can be applied to more types of robots.

[0155] Referring to 12, the embodiment of the application further provides a visual-based robot self-modeling system, which can implement the visual-based robot self-modeling method described above, and the system comprises:

[0156] The first module 101 is configured to acquire initial pictures of various joint postures of the robot and acquire joint angles corresponding to the initial pictures, wherein the joint postures include a standard state and a random posture.

[0157] The second module 102 is configured to perform data preprocessing on the initial pictures to obtain first pictures.

[0158] The third module 103 is configured to acquire camera information of the photographed pictures.

[0159] The fourth module 104 is configured to perform robot self-modeling based on a three-dimensional Gaussian splash algorithm according to the first pictures of the robot in the standard state and the camera information, to obtain a three-dimensional Gaussian splash model.

[0160] The fifth module 105 is configured to construct and train an initial deformation neural network to obtain a target deformation neural network, wherein the initial deformation neural network is configured to change ellipsoid parameters of the three-dimensional Gaussian splash model according to the joint angles and the camera information corresponding to each of the first pictures of the random posture.

[0161] The sixth module 106 is configured to combine the three-dimensional Gaussian splash model and the target deformation neural network into a robot model, to complete the self-modeling of the robot.

[0162] It can be understood that the contents in the method embodiments described above are applicable to the system embodiments, the system embodiments specifically implement the same functions as the method embodiments described above, and achieve the same beneficial effects as the method embodiments described above.

[0163] The embodiment of the application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the visual-based robot self-modeling method described above when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0164] It can be understood that the contents in the method embodiments described above are applicable to the device embodiments, the device embodiments specifically implement the same functions as the method embodiments described above, and achieve the same beneficial effects as the method embodiments described above.

[0165] Referring to Figure 13 , Figure 12 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0166] The processor 201 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0167] The memory 202 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), and the like. The memory 202 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 202 and are called and executed by the processor 201 to implement a visual-based robot self-modeling method according to an embodiment of the present application.

[0168] The input / output interface 203 is configured to realize information input and output.

[0169] The communication interface 204 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0170] The bus 205 is configured to transmit information between the components (for example, the processor 201, the memory 202, the input / output interface 203, and the communication interface 204) of the device.

[0171] The processor 201, the memory 202, the input / output interface 203, and the communication interface 204 are connected to each other through the bus 205 to realize the communication connection between the components in the device.

[0172] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the visual-based robot self-modeling method.

[0173] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiments. The storage medium embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0174] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely from the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0175] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0176] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0177] The system embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0178] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system, and the device can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0179] The terms "first", "second", "third", "fourth" and the like used in the specification of the present application and the above-described drawings, if any, are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0180] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0181] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0182] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0183] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0184] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0185] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A vision-based robot self-modeling method, characterized in that, Includes the following steps: Acquire initial images of various joint poses of the robot, and obtain the joint angles corresponding to the initial images; wherein, the joint poses include standard states and random poses; The initial image is preprocessed to obtain the first image; Obtain camera information of the camera that took the initial image; Based on the first image of the robot in its standard state and the camera information, the robot self-modeling is performed using the three-dimensional Gaussian splashing algorithm to obtain a three-dimensional Gaussian splashing model. An initial deformable neural network is constructed and trained to obtain a target deformable neural network; wherein, the initial deformable neural network is used to change the ellipsoidal parameters of the three-dimensional Gaussian splash model according to the joint angle and the camera information corresponding to each of the first images of random poses; The three-dimensional Gaussian splash model and the target deformable neural network are combined into a robot model to complete the self-modeling of the robot; The process of constructing and training the initial deformable neural network to obtain the target deformable neural network includes the following steps: Initialize the skeletal structure based on the aforementioned three-dimensional Gaussian splash model; Lock the ellipsoidal parameters of the 3D Gaussian splash model under standard conditions and initialize the initial deformation neural network; With the ellipsoid parameters locked and the robot posture change amplitude limited to less than a preset value, the initial deformable neural network is trained once to obtain the first deformable neural network. With the ellipsoid parameters locked and the range of robot posture change not limited, the first deformable neural network is trained once to obtain the second deformable neural network. With the ellipsoid parameters unlocked, the second deformable neural network is fully trained to obtain the target deformable neural network.

2. The method according to claim 1, characterized in that, The process of acquiring initial images of various joint poses of the robot and obtaining the joint angles corresponding to the initial images includes the following steps: The robot is placed in a standard state, and a fixed camera is used to take pictures of the robot to obtain standard state pictures; The robot's joint postures are randomly adjusted, and images of the robot are taken using a fixed camera to obtain random posture images; Use the standard state image and the random pose image as the initial images; Record the joint angles corresponding to the initial image.

3. The method according to claim 1, characterized in that, The process of preprocessing the initial image to obtain the first image includes the following steps: An image segmentation algorithm is used to segment out the portion containing only the robot from the initial image to obtain an intermediate image; Based on the preset length and width, all the intermediate images are scaled to obtain the first image.

4. The method according to claim 1, characterized in that, The step of obtaining camera information of the camera that took the initial image includes the following steps: Obtain the input camera information; Alternatively, camera information can be extracted from the initial image.

5. The method according to claim 4, characterized in that, The step of extracting camera information from the initial image includes the following steps: Select the initial images taken by each camera when the robot is at the same joint angle to obtain the image of the current joint angle; By using a motion extraction algorithm, the camera positions and parameters of each camera are extracted based on the current joint angle image to obtain camera information.

6. The method according to claim 1, characterized in that, The step of performing robot self-modeling based on the first image of the robot in its standard state and the camera information to obtain a 3D Gaussian splash model includes the following steps: A Gaussian ellipsoid is randomly generated within a three-dimensional space to obtain Gaussian ellipsoid parameters; wherein, the Gaussian ellipsoid parameters include position, radius, rotation attitude, and transparency; Based on the camera parameters in the camera information, the imaging plane corresponding to the camera is obtained; According to the distance from the Gaussian ellipsoid to the imaging plane, the Gaussian ellipsoid is projected onto the imaging plane from far to near; The degree of occlusion is determined based on the transparency of each Gaussian ellipsoid; Calculate the projected color using spherical harmonic functions; A second image is generated based on the projection result, the degree of occlusion, and the projection color. Calculate the first error between the second image and the first image; Repeat the step of obtaining the imaging plane corresponding to the camera based on the camera parameters in the camera information until the first error meets the preset first condition to obtain the three-dimensional Gaussian splash model.

7. A system for implementing the vision-based robot self-modeling method as described in any one of claims 1-6, characterized in that, include: The first module is used to acquire initial images of various joint poses of the robot and to acquire the joint angles corresponding to the initial images; wherein, the joint poses include standard states and random poses; The second module is used to preprocess the initial image to obtain the first image; The third module is used to obtain camera information of the camera that took the initial image; The fourth module is used to perform robot self-modeling based on the first image of the robot in a standard state and the camera information, and to obtain a three-dimensional Gaussian splash model. The fifth module is used to construct and train an initial deformable neural network to obtain a target deformable neural network; wherein, the initial deformable neural network is used to change the ellipsoidal parameters of the three-dimensional Gaussian splash model according to the joint angle and the camera information corresponding to each of the first images with random poses; The sixth module is used to combine the three-dimensional Gaussian splash model and the target deformable neural network into a robot model, thereby completing the self-modeling of the robot.

8. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 6.

9. A computer storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to implement the method as described in any one of claims 1 to 6.

Images