Robot self-modeling method, system and equipment based on vision and medium

Abstract

The invention discloses a robot self-modeling method, system and device based on vision and a medium. The robot self-modeling method comprises the steps that initial pictures of various joint postures of a robot and corresponding joint angles are obtained; performing data preprocessing on the initial picture to obtain a first picture; acquiring camera information of the initial picture; robot self-modeling is carried out based on a three-dimensional Gaussian splash algorithm according to the first picture of the robot in the standard state and the camera information; constructing and training an initial deformation neural network to obtain a target deformation neural network; wherein the initial deformation neural network is used for changing ellipsoid parameters of the three-dimensional Gaussian splash model according to the joint angle and camera information corresponding to each first picture of the random attitude; and combining the three-dimensional Gaussian splashing model and the target deformation neural network into a robot model, and completing self-modeling of the robot. The method does not need a depth camera, can reduce the data collection difficulty, improves the self-modeling precision of the robot, and can be widely applied to the technical field of robots.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a vision-based robot self-modeling method, system, device and medium. Background Art

[0002] Robots are important tools to assist and replace human labor, and can reduce burdens and improve efficiency in various fields. The robot model is the basis for controlling the robot. According to the robot model, the robot can work in a variety of different environments and complete a variety of different tasks.

[0003] In the related technologies, the robot's self-modeling technology has problems such as high cost of training data collection, low quality of self-modeling results due to the influence of data collection accuracy, and low range of freedom of the self-modeling robotic arm, which limits the scope of use. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a vision-based robot self-modeling method, system, device and medium, aiming to achieve robot self-modeling with low data acquisition cost, high model accuracy and wide application range.

[0005] To achieve the above objectives, an embodiment of the present application provides a vision-based robot self-modeling method, the method comprising:

[0006] Obtaining initial images of various joint postures of the robot, and obtaining the joint angles corresponding to the initial images; wherein the joint postures include standard states and random postures;

[0007] Performing data preprocessing on the initial image to obtain a first image;

[0008] Obtaining camera information of the camera that took the initial image;

[0009] According to the first picture of the robot in a standard state and the camera information, robot self-modeling is performed based on a three-dimensional Gaussian splash algorithm to obtain a three-dimensional Gaussian splash model;

[0010] Constructing and training an initial deformation neural network to obtain a target deformation neural network; wherein the initial deformation neural network is used to change the ellipsoid parameters of the three-dimensional Gaussian splash model according to the joint angle and the camera information corresponding to each of the first pictures of the random posture;

[0011] The three-dimensional Gaussian splash model and the target deformation neural network are combined into a robot model to complete the self-modeling of the robot.

[0012] In some embodiments, performing data preprocessing on the initial image to obtain the first image includes the following steps:

[0013] Using an image segmentation algorithm, segmenting the portion including only the robot from the initial image to obtain an intermediate image;

[0014] All the intermediate images are scaled according to the preset length and width to obtain the first image.

[0015] In some embodiments, obtaining the camera information of the camera that took the initial picture includes the following steps:

[0016] Get the input camera information;

[0017] Alternatively, camera information is extracted according to the initial image.

[0018] In some embodiments, extracting camera information according to the initial image comprises the following steps:

[0019] Select the initial pictures taken by each camera when the robot is at the same joint angle to obtain the current joint angle picture;

[0020] By using a structure-from-motion algorithm, the camera position and camera parameters of each camera are extracted according to the current joint angle image to obtain camera information.

[0021] In some embodiments, performing robot self-modeling based on the first picture of the robot in a standard state and the camera information based on a three-dimensional Gaussian splash algorithm to obtain a three-dimensional Gaussian splash model includes the following steps:

[0022] Randomly generate a Gaussian ellipsoid within a three-dimensional space to obtain Gaussian ellipsoid parameters; wherein the Gaussian ellipsoid parameters include position, radius, rotation attitude, characteristics and transparency;

[0023] Obtaining an imaging plane corresponding to the camera according to the camera parameters in the camera information;

[0024] According to the distance from the Gaussian ellipsoid to the imaging plane, projecting the Gaussian ellipsoid from far to near onto the imaging plane;

[0025] Determining the degree of occlusion according to the transparency of each Gaussian ellipsoid;

[0026] According to the characteristic parameters, the projection color is calculated by spherical harmonics;

[0027] Generate a second picture according to the projection result, the occlusion degree and the projection color;

[0028] Calculating a first error between the second image and the first image;

[0029] The step of obtaining an imaging plane corresponding to the camera according to the camera parameters in the camera information is repeatedly performed until the first error meets a preset first condition, thereby obtaining a three-dimensional Gaussian splash model.

[0030] In some embodiments, constructing and training an initial deformable neural network to obtain a target deformable neural network comprises the following steps:

[0031] Initializing a skeleton structure according to the three-dimensional Gaussian splash model;

[0032] Lock the ellipsoid parameters of the three-dimensional Gaussian splash model under the standard state and initialize the initial deformation neural network;

[0033] Under the condition that the ellipsoid parameters are locked and the robot posture change amplitude is limited to be less than a preset amplitude value, the initial deformation neural network is trained once to obtain a first deformation neural network;

[0034] Under the condition that the ellipsoid parameters are locked and the range of robot posture change is not restricted, the first neural network is trained once to obtain a second deformed neural network;

[0035] When the ellipsoid parameters are unlocked, the second deformation neural network is fully trained to obtain a target deformation neural network.

[0036] To achieve the above purpose, another aspect of the embodiment of the present application provides a vision-based robot self-modeling system, the system comprising:

[0037] The first module is used to obtain initial images of various joint postures of the robot and obtain the joint angles corresponding to the initial images; wherein the joint postures include standard states and random postures;

[0038] The second module is used to perform data preprocessing on the initial image to obtain a first image;

[0039] The third module is used to obtain the camera information of the camera that took the initial picture;

[0040] A fourth module is used to perform robot self-modeling based on a three-dimensional Gaussian splash algorithm according to the first picture of the robot in a standard state and the camera information to obtain a three-dimensional Gaussian splash model;

[0041] A fifth module is used to construct and train an initial deformation neural network to obtain a target deformation neural network; wherein the initial deformation neural network is used to change the ellipsoid parameters of the three-dimensional Gaussian splash model according to the joint angle and the camera information corresponding to each of the first pictures of the random posture;

[0042] The sixth module is used to combine the three-dimensional Gaussian splash model and the target deformation neural network into a robot model to complete the self-modeling of the robot.

[0043] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0044] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0045] The embodiments of the present application include at least the following beneficial effects: The present application provides a vision-based robot self-modeling method, system, device and medium. The scheme models the robot in a standard state with a three-dimensional Gaussian splash algorithm by acquiring color pictures of various joint postures of the robot, and trains a deformable neural network based on robot pictures in random postures, and changes the parameters of the three-dimensional Gaussian splash model. The robot's self-modeling can be completed without a depth camera, which reduces the difficulty of data collection, improves modeling accuracy, and can be applied to self-modeling of robots with high degrees of freedom. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0047] Figure 1 is a schematic diagram of a triangular network model provided in an embodiment of the present application;

[0048] Figure 2 is a schematic diagram of a point cloud model provided in an embodiment of the present application;

[0049] Figure 3 is a schematic diagram of a voxel model provided in an embodiment of the present application;

[0050] Figure 4 is a step diagram of a vision-based robot self-modeling method provided in an embodiment of the present application;

[0051] Figure 5 It is a flowchart of a vision-based robot self-modeling method provided in an embodiment of the present application;

[0052] Figure 6 It is a schematic diagram of the principle of the 3DGS algorithm provided in the embodiment of the present application;

[0053] Figure 7 is a schematic diagram of a deformed neural network provided in an embodiment of the present application;

[0054] Figure 8 This is an example diagram of the standard state robot self-modeling result provided by the embodiment of the present application;

[0055] Fig. 9 This is an example diagram of a robot self-modeling result based on vision provided in an embodiment of the present application;

[0056] Fig.10 is a schematic diagram of controlling the movement of a robot according to the self-modeling result provided in an embodiment of the present application;

[0057] Fig.11 It is provided by the embodiments of the present application;

[0058] Fig.12 It is a module schematic diagram of a vision-based robot self-modeling system provided in an embodiment of the present application;

[0059] Fig.13 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0061] Although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0062] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0063] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0064] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] In related technologies, the expression of 3D models can be divided into two types: explicit expression and implicit expression. Figure 1 , Figure 2 and Figure 3 As shown, the display expression generally adopts three methods: triangular mesh, point cloud and voxel. Triangular mesh uses a series of vertices in space and multiple small triangular planes formed by the lines between the vertices. A point cloud is a collection of a large number of points in three-dimensional space. These points together form the model of the object. The voxel expression of the three-dimensional model can be compared with the two-dimensional image. A picture is usually composed of pixel points, so a three-dimensional model can also be composed of several small cubes stacked in space.

[0067] The implicit expression of the three-dimensional model is often represented by the signed distance function. The signed distance function is represented by the following mapping relationship:

[0068]

[0069] Among them, x is a three-dimensional vector, which represents the coordinates of a point in space. s represents the closest distance from point x to the surface of the three-dimensional model, and SDF represents the signed distance function. When point x is inside the three-dimensional model, s is a negative number; when point x is outside the three-dimensional model, s is a positive number. When the point is exactly on the surface of the model, s is 0. These points on the surface of the model are also called level sets. If a three-dimensional model expressed by a signed distance function is known and the specific shape of the model needs to be known, the coordinates of the points in space are only substituted into the function one by one for calculation. All points with an output of 0 constitute the surface of the model, and all points with a negative output constitute the internal entity of the model.

[0070] In order to establish this signed distance function that can express the robot, some solutions use deep learning to learn a neural network based on a large amount of collected data, and use this neural network as the signed distance function of the robot. When the points in the space are input into the trained neural network, all the points with an output of 0 constitute the surface of the robot model. When collecting data, use several depth cameras that can simultaneously capture color images and depth images to shoot the robot from different directions. During the shooting process, the robot constantly changes its posture, and the joint angles of the robot corresponding to each posture are recorded; and based on the data collected by the camera, the point cloud that constitutes the robot surface in the current posture is obtained, and the coordinates of all the points that constitute the point cloud in space are recorded. These point clouds will be used as the real values ​​needed for subsequent neural network learning. When training the neural network, the robot joint angle and the coordinates of the midpoint in space are simultaneously input into the neural network. The angle value is randomly selected from all the angle values ​​recorded when collecting data; the points in the space are divided into two parts: "on the surface" and "not on the surface". One part is selected from the collected point cloud corresponding to the selected angle value, and the other part is randomly generated, but it is guaranteed to be different from the point coordinates obtained in the first part. After the first part of the points and joint angles are input into the neural network, the output of the neural network should be 0. After the second part of the points and joint angles are input into the neural network, the output of the neural network should not be 0. In this way, the neural network is supervised for learning, thereby obtaining a neural network that can approximate the signed distance function of the robot and realize robot self-modeling.

[0071] However, the above-mentioned related technologies have the following shortcomings:

[0072] First, the cost of training data collection is high. Since point clouds of robots at different joint angles are needed to train neural networks, depth cameras that can obtain depth data and shoot from multiple angles are needed to collect data. Depth cameras are professional equipment. Compared with ordinary cameras that can be found everywhere, they have higher acquisition costs and lower ownership rates, which hinders the application of existing technologies.

[0073] Second, the self-modeling of the above technology is affected by the accuracy of data collection and has low quality. Since the above method uses a depth camera to obtain point clouds as real samples for neural network training, the fineness of the collected point clouds will directly affect the fineness of the model obtained by the final modeling. On the one hand, the point cloud is synthesized using the depth information collected by the depth camera, and the accuracy of the depth information of the depth camera is limited and related to the cost of the depth camera; on the other hand, the camera and the robot itself will inevitably have slight shakes during shooting, which will interfere with the collection of depth information and affect the quality of the obtained point cloud.

[0074] Third, the above technology only realizes the self-modeling of a robot arm with 4 degrees of freedom (4 joints), which limits the scope of application of the technology. The higher the degree of freedom (the more joints a robot has), the more difficult it is to self-model. A method is needed to model a robot with high degrees of freedom.

[0075] In view of this, a vision-based robot self-modeling method, system, device and medium are provided in the embodiment of the present application. The scheme can be applied to robots of various forms. Taking the robot in the form of a more common mechanical arm as an example, the mechanical arm is usually connected by multiple rigid links through a rotating joint or a translation joint. The model established in the embodiment of the present application mainly includes the physical model and kinematic model of the robot. The physical model of the robot refers to the physical form of the robot, that is, the three-dimensional model of the robot. The kinematic model describes the mapping relationship between the angle of each joint of the robot and the position and posture of a module or the entire robot in space. The scheme mainly obtains robot pictures in various joint postures taken at different fixed positions, and obtains parameter information such as camera position and parameters, constructs a three-dimensional Gaussian splash model in a standard state, and trains a deformed neural network. The three-dimensional Gaussian splash model in the standard state is changed according to the joint angle to reflect the posture of the robot at different joint angles, and finally obtains a robot self-modeling result including a three-dimensional Gaussian splash model and a deformed neural network.

[0076] A vision-based robot self-modeling method provided in an embodiment of the present application relates to the field of robot technology. A vision-based robot self-modeling method provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements a vision-based robot self-modeling method, etc., but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0078] Figure 4 is an optional flowchart of a vision-based robot self-modeling method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S600.

[0079] Step S100, obtaining initial images of various joint postures of the robot, and obtaining joint angles corresponding to the initial images; wherein the joint postures include standard states and random postures.

[0080] Step S200, performing data preprocessing on the initial image to obtain a first image.

[0081] Step S300, obtaining the camera information of the camera that took the initial picture.

[0082] Step S400 , performing robot self-modeling based on a three-dimensional Gaussian splash algorithm according to the first picture of the robot in a standard state and the camera information, to obtain a three-dimensional Gaussian splash model.

[0083] Step S500, constructing and training an initial deformation neural network to obtain a target deformation neural network; wherein the initial deformation neural network is used to change the ellipsoid parameters of the three-dimensional Gaussian splash model according to the joint angles and the camera information corresponding to each of the first pictures of random postures.

[0084] Step S600, combining the three-dimensional Gaussian splash model and the target deformation neural network into a robot model to complete the self-modeling of the robot.

[0085] In steps S100 to S600 shown in the embodiment of the present application, a three-dimensional Gaussian splash algorithm is used to model the robot in a standard state by acquiring color pictures of various joint postures of the robot, and a deformable neural network is trained based on pictures of the robot in random postures to change the parameters of the three-dimensional Gaussian splash model. This allows the robot to complete self-modeling without a depth camera, reduces the difficulty of data collection, improves modeling accuracy, and is suitable for self-modeling of robots with high degrees of freedom.

[0086] In some embodiments, step S100 may include but is not limited to steps S110 to S140:

[0087] Step S110, placing the robot in a standard state, and taking a picture of the robot using a fixed camera to obtain a standard state picture.

[0088] Step S120, randomly adjusting the joint postures of the robot, using a fixed camera to take pictures of the robot, and obtaining random posture pictures.

[0089] Step S130: taking the standard state picture and the random posture picture as initial pictures.

[0090] Step S140, recording the joint angle corresponding to the initial image.

[0091] In some embodiments, step S200 includes but is not limited to the following steps S210 to S220:

[0092] Step S210: using an image segmentation algorithm, segmenting the portion including only the robot from the initial image to obtain an intermediate image.

[0093] Step S220: scaling all the intermediate images according to a preset length and width to obtain a first image.

[0094] In some embodiments, step S300 includes but is not limited to the following steps S310 or S320:

[0095] Step S310, obtaining input camera information.

[0096] Step S320, extracting camera information according to the initial image.

[0097] In some embodiments, step S320 includes but is not limited to the following steps S321-S322:

[0098] Step S321, selecting the initial pictures taken by each camera when the robot is at the same joint angle, and obtaining the current joint angle picture.

[0099] Step S322, extracting the camera position and camera parameters of each camera according to the current joint angle image through a structure extraction algorithm in motion, and obtaining camera information.

[0100] In some embodiments, step S400 includes but is not limited to the following steps S410 to S480:

[0101] Step S410, randomly generating a Gaussian ellipsoid within the three-dimensional space to obtain Gaussian ellipsoid parameters; wherein the Gaussian ellipsoid parameters include position, radius, rotation attitude, features and transparency.

[0102] Step S420: obtaining an imaging plane corresponding to the camera according to the camera parameters in the camera information.

[0103] Step S430: projecting the Gaussian ellipsoid onto the imaging plane from far to near according to the distance from the Gaussian ellipsoid to the imaging plane.

[0104] Step S440: determining the degree of occlusion according to the transparency of each Gaussian ellipsoid.

[0105] Step S450: Calculate the projection color through spherical harmonics according to the characteristic parameters.

[0106] Step S460, generating a second picture according to the projection result, the occlusion degree and the projection color.

[0107] Step S470, calculating a first error between the second picture and the first picture;

[0108] Step S480, repeatedly executing the step of obtaining an imaging plane corresponding to the camera according to the camera parameters in the camera information, until the first error meets a preset first condition, and a three-dimensional Gaussian splash model is obtained.

[0109] In some embodiments, step S500 includes but is not limited to the following steps S510 to S550:

[0110] Step S510: Initialize the skeleton structure according to the three-dimensional Gaussian splash model.

[0111] Step S520, lock the ellipsoid parameters of the three-dimensional Gaussian splash model under the standard state and initialize the initial deformation neural network.

[0112] Step S530, while locking the ellipsoid parameters and limiting the robot posture change amplitude to be less than a preset amplitude value, training the initial deformation neural network once to obtain a first deformation neural network.

[0113] Step S540, while locking the ellipsoid parameters and not limiting the range of change of the robot's posture, training the first neural network once to obtain a second deformed neural network.

[0114] Step S550: When the ellipsoid parameters are unlocked, the second deformation neural network is fully trained to obtain a target deformation neural network.

[0115] Below, in conjunction with a specific example of a robot self-modeling scenario application, the solution of the embodiment of the present application is described in detail and explained:

[0116] like Figure 5 FIG. 1 is a flow chart of a vision-based robot self-modeling method proposed in an embodiment of the present application, and the specific steps are as follows:

[0117] (1) Take pictures and preprocess the input pictures. When taking pictures, the robot should be photographed from multiple different fixed positions in various joint postures. A specific angle combination is selected as the standard state. For the robot in the standard state, additional pictures can be taken using a handheld camera such as a mobile phone. Extract only the part containing the robot from the picture, exclude the background, and process all pictures to the same size.

[0118] (2) Camera information extraction. The input of the embodiment of the present application is a large number of pictures of the robot at different joint angles, and the joint angle values ​​corresponding to the pictures. You can choose whether to provide the position of the camera that took these pictures in space. For the case of unknown cameras and parameters, camera information extraction is required. Specifically, from all the pictures, select the pictures taken from each camera when the robot is at the same joint angle, and extract the position and parameter information of these cameras through the algorithm SFM (structure from motion). The parameter information includes the number of pixels of the camera in the length and width direction and the focal length in the length and width direction.

[0119] (3) For the robot in the preset standard state, the three-dimensional Gaussian splash algorithm (3DGS) is used to realize the self-modeling of the robot in the standard state. From all the collected images, the images of the robot in the standard state are selected, and the selected images, the camera positions of these images, and the joint angle values ​​in the standard state are input. Based on the above input content, the three-dimensional Gaussian splash model of the robot in the standard state is established.

[0120] Further, refer to Figure 6 , including the following steps ①~④:

[0121] ① Generate a large number of Gaussian ellipsoids randomly in the space. The Gaussian ellipsoid has five parameters: position (x, y, z), radius, rotation attitude, features, and transparency. Among them, the first three parameters determine the geometric features of the Gaussian ellipsoid, and the feature parameters are used to represent the color of the synthesized Gaussian ellipsoid at different viewing angles. Transparency affects the transparency of the Gaussian ellipsoid color. When generating, the position and radius of the Gaussian ellipsoid are random, and the rotation attitude and features are initialized to 0.

[0122] Based on this, the expression of the Gaussian ellipsoid is:

[0123]

[0124] In the above expression, x is the position of the Gaussian ellipsoid in space; ∑ is the covariance matrix.

[0125] The calculation formula of the covariance matrix Σ is:

[0126] Σ=RSS T R T ;

[0127] Among them, R is the rotation posture of the Gaussian ellipsoid; S is the radius of the ellipsoid in three dimensions; T represents the matrix transpose.

[0128] ② Input the image of the standard state of the robotic arm, the camera position and parameters corresponding to the image, and use the 3DGS algorithm to model the standard state. The modeling process of the 3DGS algorithm is as follows: according to the camera parameters, the imaging plane corresponding to the camera is obtained; according to the distance from all Gaussian ellipsoids to the plane, they are projected onto the plane from far to near. The near ellipsoid will cover the far ellipsoid, and the degree of occlusion is determined by the transparency; according to the characteristic parameters, the color of the projection is calculated by the spherical harmonic function; finally, a picture is obtained, which reflects the posture of the current three-dimensional Gaussian splash model under the input camera perspective.

[0129] ③Compare the image obtained in step ② with the original input image and calculate the error. The Pytorch deep learning framework can be used to automatically optimize the Gaussian ellipsoid parameters in step ① according to the error. In addition, the oversized Gaussian ellipsoid is split into two new Gaussian ellipsoids.

[0130] ④ Repeat the process from step ② to step ③, and continuously optimize the Gaussian ellipsoid parameters until the image generated by the Gaussian ellipsoid is very similar to the input image, and the error no longer decreases significantly, completing the self-modeling under the standard state.

[0131] (4) Train the neural network to change the three-dimensional Gaussian splash model in the standard state according to the joint angle to reflect the posture of the robot at different joint angles. The joint angle corresponding to each collected image and the camera position that took the image are input into the neural network. The neural network deforms the three-dimensional Gaussian splash model, and a deformed neural network is obtained after training.

[0132] Furthermore, the training of the neural network mainly involves deforming the standard state model through the deformation neural network to obtain the posture of the robot model under various joint angle combinations. Figure 7 The basis of the deformable neural network is the skeleton of several robotic arms. The skeleton is represented by an ellipsoid, which is determined by the position, rotation posture and three radii of the ellipsoid. Taking a 7-degree-of-freedom robotic arm as an example, 8 ellipsoids and 8 bones are needed to represent it. The deformation network takes the joint angle as input and outputs the transformation mode of the position and rotation posture of each bone at the corresponding joint angle. After determining the bone transformation mode, the deformation network calculates the movement and rotation mode of each Gaussian ellipsoid in the standard state model. According to the calculated movement and rotation mode, the model in the standard state is deformed to obtain the model of the robotic arm at the corresponding joint angle. Specifically, it includes the following steps ⑤~⑩:

[0133] ⑤ Initialize the skeleton structure according to the three-dimensional Gaussian splash model obtained in step (3). Still taking the 7-DOF robotic arm as an example, the entire structure can be composed of 8 ellipsoids, each of which is determined by the position, rotation posture, and radius of the device itself. All parameters can be automatically updated when trained by the Pytorch deep learning framework. During initialization, the rotation posture of all ellipsoids is initialized to 0, that is, no rotation is performed. The radius of each ellipsoid is limited to 0.001 to 0.001. The middle value is taken during initialization. The position of the ellipsoid is obtained from the distribution of the Gaussian ellipsoids that make up the three-dimensional Gaussian splash model. Using the K-means algorithm, the positions of all Gaussian ellipsoids are input and clustered into 8 categories. The position of each cluster center is used as the position of the ellipsoid initialization. At this point, all skeletons are initialized.

[0134] ⑥ Initialize the deformation neural network. Lock the three-dimensional Gaussian splash model in the standard state so that the parameters of the Gaussian ellipsoid are no longer automatically updated, and stop the splitting and copying of the Gaussian ellipsoid. The joint angles in the input state are input to the deformation neural network, and the neural network outputs the displacement and rotation of each bone, represented by a 4×4 rigid transformation matrix T. According to the rigid transformation matrix of each bone, calculate the movement and rotation mode of each Gaussian ellipsoid. The specific calculation method is to find the weighted sum of the rigid transformation matrices of the 8 bones, and each Gaussian ellipsoid has a different weight relative to each bone. When calculating the weight, first calculate the Mahalanobis distance from each Gaussian ellipsoid to each bone. Since the shape of the robotic arm is not exactly the same as the ellipsoid, in order to better reflect the morphological details, a correction neural network is used to correct the Mahalanobis distance so that its distance is more consistent with the distance from the Gaussian ellipsoid to the real robotic arm. The calculation method of the Mahalanobis distance is as follows:

[0135]

[0136] Where x is the position of the Gaussian ellipsoid, μ is the center of the bone defined in D1, and Σ is the covariance matrix of the ellipsoid representing the bone. The calculation method is the same as the calculation formula in ③. -1 represents the inverse of the covariance matrix.

[0137] The complete calculation of weights is as follows:

[0138] W i,j =softmax(-10×scale×(D M (x i ,μ j )+MLP(x i )));

[0139] Among them, scale is a scale parameter that can be automatically optimized by the Pytorch framework, D M (x i ,μ j ) represents the Mahalanobis distance from the i-th Gaussian ellipsoid to the j-th bone, and MLP represents the deformable neural network. The calculation method of the Softmax function is as follows:

[0140]

[0141] Among them, w i is the weight of a Gaussian ellipsoid to the i-th bone, and there are k bones in total.

[0142] According to the movement and rotation of each Gaussian ellipsoid, the latest position and posture of each Gaussian ellipsoid is calculated, and the image under the standard state is rendered according to the rendering process and training process of the 3DGS algorithm in step ③. Compare the rendered image with the real image, train the deformation neural network under the standard state, and complete the initialization of the deformation neural network.

[0143] ⑦ For images with joint angles between -30° and 30°, the joint angles, camera positions and parameters corresponding to the images are input, and the deformation neural network calculates the movement and rotation of all Gaussian ellipsoids in the same way as in D2, and renders the Gaussian ellipsoids in the new position and posture as images according to the rendering process of the 3DGS algorithm. According to the error between the rendered image and the real image, the various parameters in the deformation neural network and the various parameters of the skeleton are automatically optimized. During the process, the parameters of the Gaussian ellipsoid itself remain unchanged, and no segmentation or replication is performed. When the arm posture changes slightly, the deformation neural network is trained.

[0144] ⑧ Train the deformation neural network, and no longer limit the range of change of the robot's posture. All collected data and pictures are used for training. The joint angle, camera position and parameters corresponding to the picture are input, and the deformation neural network calculates the movement and rotation of all Gaussian ellipsoids in the same way as D2, and renders the Gaussian ellipsoids in the new position and posture as pictures according to the rendering process of the 3DGS algorithm. According to the error between the rendered picture and the real picture, the various parameters in the deformation neural network and the various parameters of the skeleton are automatically optimized. In the process, the parameters of the Gaussian ellipsoid itself remain unchanged, and no segmentation or copying is performed.

[0145] ⑨ Completely train the Gaussian splash model and the deformable neural network. Unfreeze the three-dimensional Gaussian model in the standard state, and update the position, rotation posture, radius size, features, and transparency of the Gaussian ellipsoid at the same time as the bone parameters in the deformable neural network and the weight parameters of the neural network. Input the joint angle, the camera position and parameters corresponding to the image, and the deformable neural network calculates the movement and rotation of all Gaussian ellipsoids in the same way as D2, and renders the Gaussian ellipsoid in the new position and posture as an image according to the rendering process of the 3DGS algorithm. Automatically optimize all the above parameters based on the error between the rendered image and the real image.

[0146] ⑩Continuously train the entire network until the rendered image is similar enough to the input real image and the error of the image no longer decreases significantly, then stop training and complete the entire self-modeling process. The model obtained by self-modeling is represented by the Gaussian splash model of the 3DGS algorithm and the deformation neural network, which can show the posture of the robot arm under different viewing angles and different joint angle combinations.

[0147] In the above process, image processing, 3DGS algorithm modeling standard state model, and training deformation neural network can all be completed using Python. The latter two can be completed using the Pytorch deep learning framework. If the camera position needs to be calculated, the calculation process can be completed using COLMAP software.

[0148] like Figure 8 As shown, it is an example of the result of modeling the robot arm in the standard state and a comparison with the real robot arm. It can be seen that the method of the embodiment of the present application completes high-quality modeling of the robot arm in the standard state. Fig. 9 The examples of the results of the self-modeling model at different joint angles and postures are shown, as well as the morphology of the real robot arm at the same joint angles and postures. The comparison shows that the self-modeling results well reflect the morphology of the robot arm at different joint angles. In addition, in order to demonstrate the practicality of the model obtained by the self-modeling method of the embodiment of the application, Fig.10 The example of controlling the robot arm to complete a simple task based on the self-modeling results is shown. The task is to make the end of the robot arm reach the green position while avoiding the red obstacle. The experimental results show that the robot arm always maintains a certain distance from the red sphere. Fig.11 The change of distance.

[0149] In summary, the embodiments of the present application have at least the following beneficial effects:

[0150] 1. The reconstruction of three-dimensional information can be completed using a color picture of the robot, without the need for professional equipment to collect depth information, which is more convenient for popularization. Secondly, this embodiment can more accurately restore the geometric structure of the robot, reduce the impact of external interference on data collection and the impact of data collection equipment accuracy on the final self-modeling results. Finally, this embodiment can model high-degree-of-freedom robots and solve the self-modeling problems of more types of robots.

[0151] 2. Through the method of this application, the robot can self-model only through pictures.

[0152] 3. And because only color pictures taken at different angles with the robot in different joint postures are used as raw data, and any camera can be used during the shooting process, data collection is very convenient and low-cost.

[0153] 4. Since common cameras have sufficient accuracy, it is easy to take clear pictures, which can solve the problem of poor quality of self-modeling due to limited accuracy of depth cameras, making the self-modeling accuracy higher.

[0154] 5. It can be applied to robot self-modeling with higher degrees of freedom and can be applicable to more types of robots.

[0155] Please refer to 12. The embodiment of the present application further provides a vision-based robot self-modeling system, which can implement the above-mentioned vision-based robot self-modeling method, and the system includes:

[0156] The first module 101 is used to obtain initial images of various joint postures of the robot and obtain the joint angles corresponding to the initial images; wherein the joint postures include standard states and random postures;

[0157] The second module 102 is used to perform data preprocessing on the initial image to obtain a first image;

[0158] The third module 103 is used to obtain the camera information of the taken picture;

[0159] A fourth module 104 is configured to perform robot self-modeling based on a three-dimensional Gaussian splash algorithm according to the first picture of the robot in a standard state and the camera information, so as to obtain a three-dimensional Gaussian splash model;

[0160] The fifth module 105 is used to construct and train an initial deformation neural network to obtain a target deformation neural network; wherein the initial deformation neural network is used to change the ellipsoid parameters of the three-dimensional Gaussian splash model according to the joint angle and the camera information corresponding to each of the first pictures of the random posture;

[0161] The sixth module 106 is used to combine the three-dimensional Gaussian splash model and the target deformation neural network into a robot model to complete the self-modeling of the robot.

[0162] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0163] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned vision-based robot self-modeling method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0164] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0165] See also Fig.13 , Fig.12 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0166] The processor 201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0167] The memory 202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 202 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 202, and the processor 201 calls and executes a vision-based robot self-modeling method in the embodiment of this application;

[0168] Input / output interface 203, used to implement information input and output;

[0169] The communication interface 204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0170] Bus 205 , which transmits information between various components of the device (e.g., processor 201 , memory 202 , input / output interface 203 , and communication interface 204 );

[0171] The processor 201 , the memory 202 , the input / output interface 203 and the communication interface 204 are connected to each other in communication within the device via the bus 205 .

[0172] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned vision-based robot self-modeling method.

[0173] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0174] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0175] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0176] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0177] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0178] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0179] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0180] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0182] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0184] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0185] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A robot self-modeling method based on vision, characterized in that: The following steps are involved: Obtaining initial images of various joint postures of the robot, and obtaining the joint angles corresponding to the initial images; wherein the joint postures include standard states and random postures; Performing data preprocessing on the initial image to obtain a first image; Obtaining camera information of the camera that took the initial image; According to the first picture of the robot in a standard state and the camera information, robot self-modeling is performed based on a three-dimensional Gaussian splash algorithm to obtain a three-dimensional Gaussian splash model; Constructing and training an initial deformation neural network to obtain a target deformation neural network; wherein the initial deformation neural network is used to change the ellipsoid parameters of the three-dimensional Gaussian splash model according to the joint angle and the camera information corresponding to each of the first pictures of the random posture; The three-dimensional Gaussian splash model and the target deformation neural network are combined into a robot model to complete the self-modeling of the robot.

2. The method according to claim 1, characterized in that The method of obtaining initial images of various joint postures of the robot and obtaining the joint angles corresponding to the initial images includes the following steps: The robot is placed in a standard state, and a fixed camera is used to take a picture of the robot to obtain a standard state picture; Randomly adjusting the joint postures of the robot, and taking pictures of the robot with a fixed camera to obtain random posture pictures; Using the standard state picture and the random posture picture as initial pictures; The joint angles corresponding to the initial image are recorded.

3. The method according to claim 1, characterized in that The step of performing data preprocessing on the initial image to obtain a first image includes the following steps: Using an image segmentation algorithm, segmenting the portion including only the robot from the initial image to obtain an intermediate image; All the intermediate images are scaled according to the preset length and width to obtain the first image.

4. The method according to claim 1, characterized in that: The step of obtaining the camera information of the camera used to take the initial image comprises the following steps: Get the input camera information; Alternatively, camera information is extracted according to the initial image.

5. The method according to claim 4, characterized in that The step of extracting camera information according to the initial image comprises the following steps: Select the initial pictures taken by each camera when the robot is at the same joint angle to obtain the current joint angle picture; By using a structure-from-motion algorithm, the camera position and camera parameters of each camera are extracted according to the current joint angle image to obtain camera information.

6. The method according to claim 1, characterized in that The method of performing robot self-modeling based on the first picture of the robot in the standard state and the camera information based on a three-dimensional Gaussian splash algorithm to obtain a three-dimensional Gaussian splash model includes the following steps: Randomly generate a Gaussian ellipsoid within a three-dimensional space to obtain Gaussian ellipsoid parameters; wherein the Gaussian ellipsoid parameters include position, radius, rotation attitude, characteristics and transparency; Obtaining an imaging plane corresponding to the camera according to the camera parameters in the camera information; According to the distance from the Gaussian ellipsoid to the imaging plane, projecting the Gaussian ellipsoid from far to near onto the imaging plane; Determining the degree of occlusion according to the transparency of each Gaussian ellipsoid; According to the characteristic parameters, the projection color is calculated by spherical harmonics; Generate a second picture according to the projection result, the occlusion degree and the projection color; Calculating a first error between the second image and the first image; The step of obtaining an imaging plane corresponding to the camera according to the camera parameters in the camera information is repeatedly performed until the first error meets a preset first condition, thereby obtaining a three-dimensional Gaussian splash model.

7. The method according to claim 1, characterized in that The step of constructing and training an initial deformable neural network to obtain a target deformable neural network includes the following steps: Initializing a skeleton structure according to the three-dimensional Gaussian splash model; Lock the ellipsoid parameters of the three-dimensional Gaussian splash model under the standard state and initialize the initial deformation neural network; Under the condition that the ellipsoid parameters are locked and the robot posture change amplitude is limited to be less than a preset amplitude value, the initial deformation neural network is trained once to obtain a first deformation neural network; Under the condition that the ellipsoid parameters are locked and the range of robot posture change is not restricted, the first neural network is trained once to obtain a second deformed neural network; When the ellipsoid parameters are unlocked, the second deformation neural network is fully trained to obtain a target deformation neural network.

8. A vision-based robot self-modeling system, characterized in that: include: The first module is used to obtain initial images of various joint postures of the robot and obtain the joint angles corresponding to the initial images; wherein the joint postures include standard states and random postures; The second module is used to perform data preprocessing on the initial image to obtain a first image; The third module is used to obtain the camera information of the camera that took the initial picture; A fourth module is used to perform robot self-modeling based on a three-dimensional Gaussian splash algorithm according to the first picture of the robot in a standard state and the camera information to obtain a three-dimensional Gaussian splash model; A fifth module is used to construct and train an initial deformation neural network to obtain a target deformation neural network; wherein the initial deformation neural network is used to change the ellipsoid parameters of the three-dimensional Gaussian splash model according to the joint angle and the camera information corresponding to each of the first pictures of the random posture; The sixth module is used to combine the three-dimensional Gaussian splash model and the target deformation neural network into a robot model to complete the self-modeling of the robot.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images