Information processing device, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program for generating a photorealistic image having consistency in appearance and geometry, and for generating an image reflecting a luminance change according to an observation viewpoint.SOLUTION: In an information processing system including a controller, an information processing apparatus and a learning device, the information processing apparatus includes: an acquisition unit for acquiring posture data of an end effector capable of gripping or manipulating an object, direction data of a line of sight when observing an object from a certain viewpoint, and position data of the viewpoint; a data conversion unit 242 for generating low-dimensional posture data in which the number of dimensions of the posture data is reduced; and an image generation unit 243 for generating, using a previously learned machine learning model, an image including density of the object, which is an image generated when a scene in which the object is gripped or manipulated by the end effector is observed from the viewpoint, based on the low-dimensional posture data, the direction data, and the position data.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] A computing system is known that uses a machine learning model including multilayer perceptrons (MLPs) to generate or synthesize an image (view) obtained when an object or scene is observed from an arbitrary viewpoint (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2022 / 026692 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional techniques, when multiple different types of data are input to a machine learning model, if the dimensionality of the data is imbalanced, the machine learning model may fail to learn. As a result, images may be generated that do not show realistic appearances or geometric shapes of objects or scenes, or images that do not reflect brightness changes depending on the observation viewpoint.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objects is to provide an information processing device, an information processing method, and a program that can generate realistic images that are consistent in both appearance and geometric shape, and images that reflect brightness changes according to the observation viewpoint. [Means for solving the problem]

[0006] The information processing device, information processing method, and program according to the present invention employ the following configurations. (1) A first aspect of the present invention is an information processing device comprising: an acquisition unit that acquires posture data of an end effector capable of grasping or manipulating an object, gaze direction data when observing the object from a certain viewpoint, and position data of the viewpoint; a first generation unit that generates low-dimensional posture data by reducing the number of dimensions of the posture data; and a second generation unit that uses a pre-trained machine learning model to generate, based on the low-dimensional posture data, the direction data, and the position data, an image that can be generated when a scene in which the object is grasped or manipulated by the end effector is observed from the viewpoint, the image including the density of the object.

[0007] (2) A second aspect of the present invention is the first aspect, wherein the posture data includes, as elements, the positions and orientations of each of a plurality of joints provided in the end effector, and the first generation unit generates the low-dimensional posture data by reducing the plurality of elements included in the posture data through multivariate analysis.

[0008] (3) A third aspect of the present invention is that in the first or second aspect, the second generation unit uses a pre-trained machine learning model to estimate the color and density of the scene along the line of sight from the low-dimensional posture data, the direction data, and the position data, and generates the image by visualizing the color and density of the scene along the line of sight.

[0009] (4) A fourth aspect of the present invention is the third aspect, wherein the machine learning model is a neural radiance field trained based on a training dataset in which the correct images are labeled for the low-dimensional pose data, the orientation data, and the position data.

[0010] (5) A fifth aspect of the present invention is that, in the fourth aspect, the second generation unit increases the dimensionality of each of the low-dimensional posture data, the direction data, and the position data, inputs the increased dimensionality of the low-dimensional posture data, the direction data, and the position data into the neural radiance fields, and generates the image based on the color and density of the scene output by the neural radiance fields in response to the input of the increased dimensionality of the low-dimensional posture data, the direction data, and the position data.

[0011] (6) A sixth aspect of the present invention is that, in the first or second aspect, the robot further includes a control unit that controls the posture of the end effector based on an image of the scene captured by a camera and an image generated by the second generation unit.

[0012] (7) A seventh aspect of the present invention is an information processing method using a computer, which includes acquiring posture data of an end effector capable of grasping or manipulating an object, gaze direction data when observing the object from a certain viewpoint, and position data of the viewpoint, generating low-dimensional posture data by reducing the number of dimensions of the posture data, and using a pre-trained machine learning model to generate, based on the low-dimensional posture data, the direction data, and the position data, an image that can be generated when a scene in which the object is grasped or manipulated by the end effector is observed from the viewpoint, the image including the density of the object.

[0013] (8) An eighth aspect of the present invention is a program to be executed by a computer, the program including: acquiring posture data of an end effector capable of grasping or manipulating an object; gaze direction data when observing the object from a certain viewpoint; and position data of the viewpoint; generating low-dimensional posture data by reducing the number of dimensions of the posture data; and using a pre-trained machine learning model, generating an image based on the low-dimensional posture data, the direction data, and the position data, which may be generated when a scene in which the object is grasped or manipulated by the end effector is observed from the viewpoint, the image including the density of the object. [Effects of the Invention]

[0014] According to the above aspect, it is possible to generate realistic images that are consistent in both appearance and geometric shape, and to generate images that reflect brightness changes according to the observation viewpoint. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system 1 according to a first embodiment. [Figure 2] FIG. 1 is a diagram schematically illustrating the appearance of a robot 10 according to a first embodiment. [Figure 3] 1 is a configuration diagram of a robot 10 and a control device 100 according to a first embodiment. [Figure 4] 1 is a diagram illustrating a configuration of an information processing device 200 according to a first embodiment. [Figure 5] FIG. 1 is a diagram for explaining an overview of NeRF. [Figure 6] 10 is a flowchart showing the flow of a series of processes by a processing unit 240 according to the first embodiment. [Figure 7] FIG. 1 is a schematic representation of P-NeRF. [Figure 8] FIG. 1 is a diagram illustrating an example of a P-NeRF network configuration. [Figure 9] FIG. 1 is a configuration diagram of a learning device 300 according to a first embodiment. [Figure 10] FIG. 1 is a diagram for explaining a learning method for a machine learning model. [Figure 11] FIG. 10 is a configuration diagram of a robot 10 and a control device 100A according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an information processing device, an information processing method, and a program according to an embodiment of the present invention will be described with reference to the drawings.

[0017] First Embodiment [System Configuration] 1 is a diagram illustrating an example of the configuration of an information processing system 1 according to the first embodiment. The information processing system 1 includes, for example, a control device 100, an information processing device 200, and a learning device 300. The control device 100, the information processing device 200, and the learning device 300 are connected via a network NW. The network NW includes a LAN (Local Area Network), a WAN (Wide Area Network), etc.

[0018] The control device 100 performs autonomous control so that the robot 10 behaves autonomously. The control device 100 is typically mounted on the robot 10 and directly controls the robot 10. Alternatively, the control device 100 may be installed at a location far away from the robot 10 and control the robot 10 remotely via a network NW.

[0019] The information processing device 200 uses a machine learning model to artificially generate or synthesize an image of a scene observed from a different viewpoint than the camera 11 mounted on the robot 10 or the surveillance camera 20 installed in the space (e.g., indoors) where the robot 10 works.

[0020] The learning device 300 learns the machine learning model used by the information processing device 200.

[0021] Either or both of information processing device 200 and learning device 300 may be a single device, or may be a system in which multiple devices connected via a network NW operate in cooperation with each other. In other words, either or both of information processing device 200 and learning device 300 may be realized by multiple computers (processors) included in a distributed computing system or a cloud computing system.

[0022] [Robot appearance] 2 is a diagram schematically illustrating the appearance of a robot 10 according to the first embodiment. The robot 10 is typically a humanoid robot that can grasp or manipulate an object OB using an end effector, but is not limited to this and may be any type of robot that can grasp or manipulate an object OB. For example, the robot 10 may be a quadrupedal animal-type robot, an industrial robot, a military robot, a household cleaning robot, or any other type of robot.

[0023] The end effector is also called a robot hand. Therefore, in the following description, the end effector may be simply referred to as a "hand." The end effector of the robot 10 may be provided with, for example, several fingers that function as grippers.

[0024] The robot 10 is equipped with, for example, a camera 11 for imaging the external environment as seen by the robot 10, and a control device 100 for controlling the operation of the robot 10, and performs the desired task according to the actions determined by the control device 100.

[0025] The task may be, for example, grabbing the object OB with one hand, transferring the object OB to the other hand, or moving the object OB. However, the task is not limited to these, and any task can be set.

[0026] The camera 11 is installed in a part of the body (typically, the head) of the robot 10. The camera 11 may be, for example, a depth camera (3D camera). For example, the camera 11 captures an image of a scene in which an object OB is grasped or manipulated by an end effector, and transmits image data of the scene to the control device 100.

[0027] [Robot and control device configuration] 3 is a configuration diagram of the robot 10 and the control device 100 according to the first embodiment. In addition to the camera 11 described above, the robot 10 further includes an actuator 12, a state sensor 13, and an operation control unit 14.

[0028] The actuators 12 are controlled by the motion control unit 14 to drive the various parts (arms, fingers, legs, head, torso, waist, etc.) of the robot 10. The actuators 12 include, for example, electromagnetic motors, gears, artificial muscles, etc.

[0029] The status sensor 13 is a sensor that detects the status (for example, joint angle, angular velocity, torque, etc.) of the robot 10. The status sensor 13 includes, for example, a rotary encoder that detects the degree of rotation of the joints of the robot 10 or the end effector, a tension sensor that detects the tension of the wires that rotate the joints, a torque sensor that detects the torque applied to the joint shafts, an acceleration sensor or a gyro sensor that detects the posture of the robot 10, etc.

[0030] The operation control unit 14 controls the actuator 12 based on the control command generated by the control device 100 .

[0031] The control device 100 includes, for example, a communication interface 110, a processing unit 120, and a storage unit .

[0032] The communication interface 110 communicates with an external device via a network NW, and with the robot 10 via a communication line such as a bus. The external device is, for example, a terminal device (such as a smartphone or a personal computer) that can be used by a user to request the robot 10 to execute a task. The communication interface 110 includes, for example, a wireless communication module including a receiver and a transmitter, a NIC (Network Interface Card), and the like.

[0033] The processing unit 120 includes, for example, an acquisition unit 121, a grip posture determination unit 122, a command generation unit 123, and a communication control unit 124.

[0034] The acquisition unit 121 acquires image data and status data from the robot 10 via the communication interface 110 .

[0035] The image data is image data generated when the camera 11 captures an image of the object OB. For example, when the camera 11 is installed on the head of the robot 10, the acquisition unit 121 acquires image data obtained by observing a scene in which the object OB is grasped or manipulated by the end effector from the viewpoint of the head of the robot 10.

[0036] The status data includes, for example, the detection values of the status sensors 13 of the robot 10. Specifically, the status data includes the angles of the joints of the robot 10 or the end effector detected by a rotary encoder, the wire tension of the joints of the robot 10 or the end effector detected by a tension sensor, the torque of the joint shafts of the robot 10 or the end effector detected by a torque sensor, the acceleration of the robot 10 or the end effector detected by an acceleration sensor, and the angular velocity of the robot 10 or the end effector detected by a gyro sensor.

[0037] When a surveillance camera 20 is present in the workspace of the robot 10, the acquisition unit 121 may acquire image data from the surveillance camera 20 in addition to or instead of acquiring image data from the robot 10. Hereinafter, image data of a scene observed by the camera 11 and / or the surveillance camera 20 will be referred to as "camera viewpoint image data."

[0038] Furthermore, the acquisition unit 121 acquires image data of a scene observed from a viewpoint different from that of the camera 11 or the monitoring camera 20 from the information processing device 200 via the communication interface 110. Hereinafter, image data of a scene observed from a viewpoint different from that of the camera 11 or the monitoring camera 20 will be referred to as "different viewpoint image data".

[0039] The gripping posture determination unit 122 determines a posture (pose) to be taken by the end effector based on the camera viewpoint image data and the different viewpoint image data. For example, the gripping posture determination unit 122 determines the posture of the end effector so that the object OB included in the different viewpoint image data and the object OB included in the camera viewpoint image data coincide with each other on the image plane.

[0040] The command generation unit 123 generates a control command for controlling each actuator 12 of the robot 10 based on the posture of the end effector determined by the grip posture determination unit 122 and the state data acquired by the acquisition unit 121.

[0041] The communication control unit 124 transmits the control command generated by the command generation unit 123 to the robot 10 via the communication interface 110. When the operation control unit 14 of the robot 10 receives the control command from the control device 100, it controls the actuator 12 based on the control command. This causes the robot 10 to act and perform a task such as grasping or manipulating an object OB.

[0042] The components of the processing unit 120 are realized by, for example, a central processing unit (CPU) or a graphics processing unit (GPU) executing programs or instructions stored in the storage unit 130. Some or all of these components may be realized by hardware such as a large-scale integration (LSI), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), or may be realized by a combination of software and hardware.

[0043] The storage unit 130 is realized by, for example, a hard disk drive (HDD), a flash memory, an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a random access memory (RAM), etc. The storage unit 130 stores firmware, application programs, calculation results of the processing unit 120, etc.

[0044] [Configuration of information processing device] The configuration of the information processing device 200 according to the first embodiment will be described below. Fig. 4 is a configuration diagram of the information processing device 200 according to the first embodiment. The information processing device 200 includes, for example, a communication interface 210, an input interface 220, an output interface 230, a processing unit 240, and a storage unit 250.

[0045] The communication interface 210 communicates with the control device 100 and the learning device 300 via the network NW. The communication interface 310 includes, for example, a wireless communication module including a receiver and a transmitter, and an NIC.

[0046] The input interface 220 accepts various input operations from the user, converts the accepted input operations into electrical signals, and outputs the signals to the processing unit 240. For example, the input interface 220 includes a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touch panel, and the like.

[0047] The output interface 230 includes, for example, a display and a speaker. The display displays images generated by the processing unit 240, a GUI (Graphical User Interface) for receiving various input operations from an operator, and the like. For example, the display is an LCD (Liquid Crystal Display), an organic EL (Electro Luminescence) display, or the like. The speaker outputs information input from the processing unit 240 as sound.

[0048] The processing unit 240 includes, for example, an acquisition unit 241, a data conversion unit 242, an image generation unit 243, and an output control unit 244. The data conversion unit 242 is an example of a "first generation unit," and the image generation unit 243 is an example of a "second generation unit."

[0049] The components of the processing unit 240 are realized by, for example, a CPU or a GPU executing a program stored in the storage unit 250. Some or all of these components may be realized by hardware such as an LSI, an ASIC, or an FPGA, or may be realized by a combination of software and hardware.

[0050] The storage unit 250 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, a RAM, etc. The storage unit 250 stores model data in addition to various programs such as firmware and application programs.

[0051] The model data is a program or algorithm that defines a machine learning model used when generating or synthesizing different-viewpoint image data. The machine learning model of this embodiment is a neural network including multiple intermediate layers. For example, the model data includes various information, such as connection information on how units included in each of the multiple layers constituting the neural network are connected to each other, and connection coefficients assigned to data input and output between the connected units. The connection information includes, for example, the number of units included in each layer, information specifying the type of unit to which each unit is connected, activation functions that realize each unit, and gates provided between units in the hidden layer. The activation functions of the units may be, for example, a rectified linear function (ReLU function), a sigmoid function, a step function, a Softplus function, or other functions. The gates selectively pass or weight data transmitted between units depending on, for example, the value (e.g., 1 or 0) returned by the activation function. The connection coefficients include, for example, weights assigned to output data when data is output from a unit in a certain layer to a unit in a deeper layer in the intermediate layers of the neural network. The coupling coefficient may include the inherent bias component of each layer.

[0052] For example, the machine learning model is a model that has been trained in advance to output information about a three-dimensional structure that a human would be able to recognize when observing the target scene from the specified viewpoint and direction, when an arbitrary viewpoint from which the target scene can be observed and a line of sight direction when the scene is observed from the specified viewpoint and direction are specified. Such a machine learning model can be implemented, for example, by Neural Radiance Fields (NeRF).

[0053] NeRF is a method proposed primarily to achieve the task of generating an image from an arbitrary (new) viewpoint from images from multiple viewpoints (called Novel View Synthesis).

[0054] Figure 5 is a diagram illustrating an overview of NeRF. Given a position vector x(→) = (x, y, z) indicating the three-dimensional position of a certain viewpoint and a direction vector d(→) = (θ, φ) indicating the line of sight when observing a scene from that viewpoint, NeRF determines N sampling points on the line of sight and outputs a color vector c(→) = (R, G, B) indicating the color of each sampling point and its density σ. The parenthesized arrow (→) represents a vector. For example, x may represent the width of the three-dimensional space, y may represent the depth, and z may represent the height. One of θ and φ may represent the elevation angle, and the other of θ and φ may represent the azimuth angle. R represents red, G represents green, and B represents blue. N is an arbitrary natural number.

[0055] In this embodiment, a NeRF conditioned by the joint postures of the end effector is used to control the posture of the end effector of the robot 10. Several NeRF models have already been proposed for controlling the pose of the human body, and the simplest model known is the Pose-conditioned NeRF (hereinafter referred to as P-NeRF). In this embodiment, as an example, a machine learning model is implemented using a P-NeRF conditioned by the joint postures of the end effector.

[0056] [Processing flow of information processing device] Hereinafter, a description will be given using a flowchart of each component of the processing unit 240 of the information processing device 200. Fig. 6 is a flowchart showing the flow of a series of processes by the processing unit 240 according to the first embodiment.

[0057] First, the acquisition unit 241 acquires a position vector x(→) that represents the three-dimensional position of the viewpoint and a direction vector d(→) that indicates the direction of the line of sight (step S100). The position vector x(→) is an example of "position data," and the direction vector d(→) is an example of "direction data."

[0058] For example, when a user inputs the three-dimensional position of the viewpoint and the direction of the line of sight to the input interface 220, the acquisition unit 241 may acquire the user's input to the input interface 220 as a viewpoint position vector x(→) and a line of sight direction vector d(→).

[0059] Furthermore, the three-dimensional position of the viewpoint and the direction of the line of sight may be determined depending on the task assigned to the robot 10. For example, if the object OB is smaller than the end effector and most of the object OB is covered by the end effector, the three-dimensional structure of the object OB may not be fully recognized on the camera viewpoint image data. Furthermore, even if the three-dimensional structure of the surface of the object OB can be recognized on the camera viewpoint image data, the three-dimensional structure of the back and side surfaces of the object OB is more difficult to recognize than the surface.

[0060] In such a case, the acquisition unit 241 acquires a task from the robot 10 or the control device 100 via the communication interface 210, and based on the acquired task, calculates a viewpoint position and a line of sight direction that are assumed to make it difficult to recognize the three-dimensional structure of the object OB when viewed from the camera 11 or the monitoring camera 20. Then, the acquisition unit 241 acquires the vectors of the calculated viewpoint position and line of sight direction.

[0061] Next, the acquisition unit 241 acquires a posture vector l(→) representing the posture of the joints of the end effector (step S102). The posture vector l(→) is calculated based on a 4×4 dimensional pose transformation matrix l consisting of the rotation matrix and translation vector of each of the joints provided in the end effector. i (i=1,...,P). Hereinafter, the rotation matrix and translation vector may be collectively referred to as the "transformation matrix." P is the number of joints. In other words, the posture vector l(→) has P×4×4 dimensions. For example, if the number of joints P is 17, the posture vector l(→) has 272 dimensions. Considering that the position vector x(→) is three-dimensional and the direction vector d(→) is two-dimensional, the posture vector l(→) is a vector with a much higher dimension than these vectors.

[0062] For example, the acquisition unit 241 acquires status data including detection values of the status sensor 13 from the robot 10 or the control device 100 via the communication interface 210. Then, the acquisition unit 241 extracts transformation matrices (rotation matrices and translation vectors) of the multiple joints provided in the end effector from the multiple detection values included in the status data, and combines these to generate a posture vector l(→). The posture vector l(→) is an example of "posture data."

[0063] Next, the data conversion unit 242 converts (compresses) the attitude vector l(→) into a low-dimensional representation (step S104).

[0064] The space of the range of motion consisting of the joints of the end effector can be expressed in a lower-dimensional space. For example, the acquisition unit 241 acquires N samples of posture vectors l(→) in which the transformation matrices of each joint are arranged horizontally for the number of joints P. The data conversion unit 242 converts the posture vectors l(→) of the N samples into six-dimensional vectors of (x, y, z, yaw, pitch, roll), and expresses a set D∈{l i} N i=1 The data conversion unit 242 converts (compresses) the P×4×4 dimensional orientation vector l(→) into a d-dimensional low-dimensional representation e(→) using the eigenvalues and eigenvectors of the covariance matrix obtained by the principal component analysis.

[0065] The P×4×4 dimensional posture vector l(→) increases in number of dimensions in proportion to the increase in the number of joints P. On the other hand, the low-dimensional representation e(→) is a fixed-length vector representation (a vector with a constant number of dimensions) that does not depend on the number of joints P. In the following explanation, the low-dimensional representation e(→) will be referred to as the "eigengrasp representation." The eigengrasp representation e(→) is an example of "low-dimensional posture data."

[0066] The data conversion unit 242 may perform other multivariate analyses, such as multiple regression analysis, independent component analysis, or factor analysis, instead of principal component analysis. Furthermore, instead of converting the posture vector l(→) into the eigenvalue representation e(→) by performing multivariate analysis, the data conversion unit 242 may convert the posture vector l(→) into the eigenvalue representation e(→) using an autoencoder. The autoencoder is a model trained in advance to compress the dimensionality of input data, and is implemented, for example, by a neural network including a convolutional layer.

[0067] Next, the data conversion unit 242 increases the dimensionality of each of the position vector x(→), the direction vector d(→), and the unique grasping expression e(→) (step S106). In other words, the data conversion unit 242 embeds each of the position vector x(→), the direction vector d(→), and the unique grasping expression e(→) in a high-dimensional space.

[0068] As mentioned above, a typical NeRF accepts a three-dimensional position vector x(→) and a two-dimensional direction vector d(→) as input, and outputs a color vector c(→) and a density σ. However, the input data dimensions are limited to five. While multilayer perceptrons (MLPs) can theoretically approximate any continuous function, they are known to be unable to represent high-frequency components of color and shape when using low-dimensional inputs. This phenomenon is known as frequency bias or spectral bias. This phenomenon is studied in the field of deep learning theory, specifically the neural tangent kernel, where neural networks train from low-frequency signals to high-frequency signals. In other words, they fit simple functions to complex functions. As a result, the output of multilayer perceptrons (MLPs) converges to low-frequency components, failing to capture the high-frequency components inherent in natural images, resulting in blurred images.

[0069] Therefore, the data conversion unit 242 embeds low-dimensional input data (position vector x(→), direction vector d(→), and eigenvector e(→)) into a high-dimensional space. For example, the data conversion unit 242 may embed the low-dimensional input data into a high-dimensional space using a technique called Positional Encoding. Positional Encoding is expressed by Equation (1).

[0070]

number

[0071] ρ represents the input data, i.e., the position vector x(→), the direction vector d(→), or the eigengrasp representation e(→), and L represents the number of dimensions of the high-dimensional space into which the input data ρ is embedded. L is a hyperparameter.

[0072] For example, the data conversion unit 242 converts the position vector x(→) into a higher-dimensional vector γ(x(→)), the direction vector d(→) into a higher-dimensional vector γ(d(→)), and the unique grasp representation e(→) into a higher-dimensional vector γ(e(→)) using Positional Encoding represented in formula (1).

[0073] Next, the image generation unit 243 uses P-NeRF, one of the machine learning models, to generate image data from a different viewpoint based on the vectors γ(x(→)), γ(d(→)), and γ(e(→)) that have been converted to higher dimensions by the data conversion unit 242 (step S108).

[0074] 7 is a diagram showing a schematic representation of a P-NeRF. As shown in the figure, the image generation unit 243 inputs the vectors γ(x(→)), γ(d(→)), and γ(e(→)) that have been increased in dimension by the data conversion unit 242 to the P-NeRF. In response to this, the P-NeRF outputs a color vector c(→) and density σ in response to the input of these increased-dimensional vectors. The image generation unit 243 generates different-viewpoint image data based on the color vector c(→) and density σ output by the P-NeRF.

[0075] 8 is a diagram showing an example of a P-NeRF network configuration. The P-NeRF (NeRF conditioned by the posture of the joints of the end effector) of this embodiment includes multiple hidden layers, as shown in the figure. The hidden layers are fully connected (FC) layers, and branch into an output layer that outputs density σ and an output layer that outputs a color vector c(→). For example, the number of dimensions of each hidden layer may be 256, and the activation function may be ReLU.

[0076] In P-NeRF, γ(x(→)) and γ(e(→)) are input to the first and middle hidden layers, respectively, and γ(d(→)) and γ(e(→)) are input to the hidden layer two layers before the output layer, which outputs the color vector c(→).

[0077] The P-NeRF conditioned by the eigengrasp representation e(→) is expressed by Equation (2). Hereinafter, the P-NeRF conditioned by the eigengrasp representation e(→) will be referred to as the “Eigen-NeRF (Eigengrasp-conditioned NeRF).”

[0078]

number

[0079] Eigen-NeRF outputs a color vector c(→) and density σ in response to input of γ(x(→)), γ(d(→)), and γ(e(→)). The image generation unit 243 performs volume rendering of the color vector c(→) and density σ output by Eigen-NeRF in the line of sight direction (direction vector d(→)), thereby generating different viewpoint image data of the scene that would be obtained when observed from the three-dimensional position indicated by the position vector x(→).

[0080] For example, as volume rendering, the image generation unit 243 determines pixel values in a different viewpoint image by accumulating colors in the line of sight (directional vector d(→)) according to density, so that the visual perception of humans is determined by the brightness of light incident on the retina.

[0081] In this case, the light traveling in the reverse direction from the light source to the viewpoint along the line of sight is called a ray r. At each sample point on the ray r traveling in three-dimensional space, the image generation unit 243 determines the pixel value in image space by volume rendering based on the color vector c(→) and density σ of each sample point determined by NeRF. This series of rendering procedures is formulated below. To determine the color C(r) in image space along the ray r, volume rendering sends the ray r traveling through the three-dimensional space where the object OB is located along a direction d that determines a point in image space. This direction d is determined from the parameters of the camera 11 (or surveillance camera 20). When the position of the camera 11 (or surveillance camera 20) is the origin o(→), any point r(t) on the ray r can be expressed as shown in Equation (3) using a parameter t that indicates how far it travels from the camera origin o(→) toward the direction d(→).

[0082]

number

[0083] The color on the image plane can be determined by integrating the density σ(r(→)(t)) on the ray and the color c(r(→)(t),d(→)) with respect to t. In this case, the evaluation of the ray r can be calculated from the camera origin o(→) to infinity, but in volume rendering, the lower limit t for evaluating the ray r is n and upper limit t f The integration process that determines the color C(r(→)) of a point in image space is defined by equations (4) and (5).

[0084]

number

[0085]

number

[0086] Here, the function T is called Accumulated Transmittance, and expresses the phenomenon in which the brightness of the light ray traveling from the light source to the observation point (camera) is attenuated by materials or objects closer to the observation point. As shown in formula (5), the integral term becomes larger when the density σ is high, and as a result, the output value of function T becomes smaller. This reduces the influence of the light ray r(t) at that point. In implementation, this integral is calculated in a discretized manner. For this purpose, the interval [t n ,t f ] samples the 3D positions of the N rays to be evaluated, determined by the hyperparameters. During rendering, only the sampled points on the rays are evaluated, affecting the pixel value. At this time, rays are sampled not at regular intervals, but with randomness based on a uniform distribution U, as shown in equation (6).

[0087]

number

[0088] This sampled t i For a set of , NeRF has density σi and color c i (→) is output. This density σ i and color c i By discretely integrating (→), the color C^(r) in the image space is determined. C^(r) is expressed by Equation (7). T in Equation (7) i is expressed by equation (8), and δ i is expressed by equation (9).

[0089]

number

[0090]

number

[0091]

number

[0092] where δ i indicates the distance between adjacent sample points. The image generation unit 243 generates a different viewpoint image by performing volume rendering to obtain the pixel value C^(r) on the image for all pixels.

[0093] In order to generate realistic alternative viewpoint images more effectively, the image generation unit 243 may sample light rays intensively in areas where the color changes rapidly, such as near the boundaries of objects OB, or in areas where objects OB are densely concentrated (Hierarchical Volume Sampling).

[0094] The image generation unit 243 may also render a depth image as different-viewpoint image data using the density σ output by Eigen-NeRF and the distance z to the sampling point t on the ray r. The depth can be calculated by Equation (10).

[0095]

number

[0096] The image generating unit 243 may generate a different viewpoint image by other methods such as maximum intensity projection or surface rendering instead of volume rendering. The image generating unit 243 may also generate a different viewpoint image by constructing a signed distance field (SDF) from the density σ output by Eigen-NeRF, and performing normal 3D / 2D projection after 3D reconstruction.

[0097] Returning to the description of the flowchart, the output control unit 244 then outputs the different-viewpoint image data generated by the image generation unit 243 (step S110). For example, the output control unit 244 may transmit the different-viewpoint image data to the control device 100 via the communication interface 210, or may cause the different-viewpoint image data to be displayed on the display of the output interface 230. This completes the processing of this flowchart.

[0098] For example, when the communication interface 110 of the control device 100 receives different-viewpoint image data from the information processing device 200, the gripping posture determination unit 122 determines the posture that the end effector should take based on the camera-viewpoint image data and the different-viewpoint image data, thereby generating a control command that causes the robot 10 to perform a target task.

[0099] [Learning device configuration] The configuration of the learning device 300 according to the first embodiment will be described below. Fig. 9 is a configuration diagram of the learning device 300 according to the first embodiment. The learning device 300 includes, for example, a communication interface 310, a processing unit 320, and a storage unit 330.

[0100] The communication interface 310 communicates with the information processing device 200 via the network NW. The communication interface 310 includes, for example, a wireless communication module including a receiver and a transmitter, and an NIC.

[0101] The processing unit 320 includes, for example, an acquisition unit 321, a learning unit 322, and a communication control unit 323.

[0102] The components of the processing unit 320 are realized by, for example, a CPU or a GPU executing a program stored in the storage unit 330. Some or all of these components may be realized by hardware such as an LSI, an ASIC, or an FPGA, or may be realized by a combination of software and hardware.

[0103] The storage unit 330 is realized by, for example, a HDD, flash memory, EEPROM, ROM, RAM, etc. The storage unit 330 stores various programs such as firmware and application programs, as well as model data that defines an untrained machine learning model (Eigen-NeRF). An untrained machine learning model may be a model that has never been trained, or may be a model that has been trained at least once but the training was insufficient.

[0104] The acquisition unit 321 acquires a training data set for training an untrained machine learning model.

[0105] The training dataset is a dataset in which correct different-viewpoint image data to be output by the machine learning model is labeled for the high-dimensional vectors γ(x(→)), γ(d(→)), and γ(e(→)) prepared for training. In other words, the training dataset is a dataset in which the high-dimensional vectors γ(x(→)), γ(d(→)), and γ(e(→)) are used as input data, and the correct different-viewpoint image data to be output by the machine learning model is used as output data.

[0106] For example, the acquisition unit 321 may acquire a training dataset from another device (e.g., a data source) via the communication interface 310. Furthermore, if the training dataset is already stored in the storage unit 330, the acquisition unit 321 may read the training dataset from the storage unit 330. Furthermore, if a non-transitory storage medium (e.g., flash memory) on which the training dataset is stored is connected to the drive device of the learning device 300, the acquisition unit 321 may read the training dataset from the storage medium.

[0107] The learning unit 322 uses the training data set acquired by the acquisition unit 321 to learn an untrained machine learning model.

[0108] 10 is a diagram illustrating a learning method for a machine learning model. For example, the learning unit 322 inputs high-dimensional vectors γ(x(→)), γ(d(→)), and γ(e(→)) included as input data in a training dataset into an unlearned machine learning model (Eigen-NeRF).

[0109] The untrained machine learning model outputs a color vector c(→) and density σ in response to input of high-dimensional vectors γ(x(→)), γ(d(→)), and γ(e(→)). The learning unit 322 generates different-viewpoint image data by volume-rendering the color vector c(→) and density σ output by the machine learning model in the line-of-sight direction (direction vector d(→)).

[0110] The learning unit 322 calculates an error L between the different viewpoint image data generated by volume rendering and the correct different viewpoint image data included as output data in the training data set. color The learning unit 322 calculates the error L color The weight coefficients and bias components, which are parameters of the machine learning model, are determined (updated) using methods such as stochastic gradient descent so that the error L color may be calculated, for example, by equation (11).

[0111]

number

[0112] where R is the set of rays and C t (r) is the pixel value of the correct different-view image. As shown in equation (11), the camera parameters are given and the squared error between the different-view image rendered from the Coarse Network and Fine Network and the correct different-view image is minimized. Only the image from the target viewpoint is required as training data; three-dimensional training data such as a 3D model is not required. This is the advantage of using a differentiable renderer. In addition, if you want the machine learning model to output a depth image as different-view image data, you need to consider an error term related to depth estimation in addition to the color error function above. The error in depth estimation is shown in equation (12).

[0113]

number

[0114] where D t (r) is the depth value of the correct image from another viewpoint, and D^ c ,D^ f are the depth values estimated by the Coarse Network and Fine Network, respectively. The final error function L depth If depth is not taken into account, L=L color When considering the depth, L = L color +L depth This can be done as follows.

[0115] The communication control unit 323 transmits model data that defines the machine learning model learned by the learning unit 322 to the information processing device 200 via the communication interface 310. This allows the information processing device 200 to generate different-viewpoint image data using the learned machine learning model.

[0116] According to the first embodiment described above, the information processing device 200 acquires the posture vector l(→) of the joint of the end effector capable of grasping or manipulating the object OB, the line-of-sight direction vector d(→) when observing the object OB from a certain viewpoint, and the position data x(→) of the viewpoint. The information processing device 200 generates, from the posture vector l(→), an eigengrasping representation e(→), which has a smaller number of dimensions than the posture vector l(→). Using a machine learning model implemented by Eigen-NeRF, the information processing device 200 generates different-viewpoint image data that is generated when a scene in which the object OB is grasped or manipulated by the end effector is observed from the above viewpoint, based on the eigengrasping representation e(→), the direction vector d(→), and the position data x(→). This processing makes it possible to generate realistic images that are consistent in both appearance and geometric shape, and images that reflect brightness changes according to the observation viewpoint.

[0117] Furthermore, according to the first embodiment described above, the information processing device 200 converts the pose vector l(→), which has a higher dimension than the position vector x(→) or the direction vector d(→), into a fixed-length eigenvalue representation e(→) using multivariate analysis or an autoencoder before positional encoding (before increasing the dimension). This reduces the bias in the ratio of the number of dimensions between multiple different types of data input to NeRF. More specifically, it is possible to prevent the number of dimensions of the pose vector l(→) from becoming significantly larger than the number of dimensions of the direction vector d(→) or the position data x(→). As a result, it becomes easier to tune the dimension of positional encoding, and the learning accuracy of NeRF can be improved.

[0118] Second Embodiment The second embodiment will be described below. In the first embodiment described above, the control device 100, the information processing device 200, and the learning device 300 included in the information processing system 1 were described as separate devices independent of each other. The second embodiment differs from the first embodiment described above in that some or all of these devices are integrated into one device. The following description will focus on the differences from the first embodiment, and will omit a description of the points in common with the first embodiment. In the description of the second embodiment, the same parts as in the first embodiment will be described using the same reference numerals.

[0119] 11 is a configuration diagram of a robot 10 and a control device 100A according to the second embodiment. A processing unit 120A of the control device 100A according to the second embodiment further includes a data conversion unit 242 and an image generation unit 243 in addition to the acquisition unit 121, grip posture determination unit 122, command generation unit 123, and communication control unit 124 described above. This allows the control device 100A to generate different-viewpoint image data by itself.

[0120] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0121] 1...information processing system, 10...robot, 11...camera, 12...actuator, 13...status sensor, 14...operation control unit, 100...control device, 110...communication interface, 120...processing unit, 130...storage unit, 200...information processing device, 210...communication interface, 220...input interface, 230...output interface, 240...processing unit, 250...storage unit, 300...learning device, 310...communication interface, 320...processing unit, 330...storage unit

Claims

1. an acquisition unit that acquires posture data of an end effector that can grasp or manipulate an object, line-of-sight direction data when observing the object from a certain viewpoint, and position data of the viewpoint; a first generation unit that generates low-dimensional posture data by reducing the number of dimensions of the posture data; a second generation unit that generates an image that can be generated when a scene in which the object is grasped or manipulated by the end effector is observed from the viewpoint based on the low-dimensional posture data, the direction data, and the position data using a pre-trained machine learning model, the image including the density of the object; and An information processing device comprising:

2. the posture data includes, as elements, the positions and orientations of each of a plurality of joints provided in the end effector; the first generation unit generates the low-dimensional posture data by reducing the plurality of elements included in the posture data through multivariate analysis. The information processing device according to claim 1 .

3. The second generation unit using the machine learning model to estimate color and density of the scene along the line of sight from the low-dimensional pose data, the orientation data, and the position data; generating the image by visualizing the color and density of the scene along the line of sight; 3. The information processing device according to claim 1 or 2.

4. The machine learning model is a neural radiance field trained based on a training dataset in which the images that are correct answers for the low-dimensional pose data, the orientation data, and the position data are labeled. The information processing device according to claim 3 .

5. The second generation unit The low-dimensional posture data, the orientation data, and the position data are each made into a high-dimensional data; inputting the increased-dimensional low-dimensional posture data, orientation data, and position data into the neural radiation fields; generating the image based on the color and density of the scene output by the Neural Radiance Fields in response to the input of the increased low-dimensional posture data, the orientation data, and the position data; The information processing device according to claim 4 .

6. a control unit that controls the posture of the end effector based on an image of the scene captured by a camera and the image generated by the second generation unit, 3. The information processing device according to claim 1 or 2.

7. An information processing method using a computer, comprising: Acquiring posture data of an end effector capable of grasping or manipulating an object, line-of-sight direction data when observing the object from a certain viewpoint, and position data of the viewpoint; generating low-dimensional posture data by reducing the number of dimensions of the posture data; generating an image that includes the density of the object, the image being generated when a scene in which the object is grasped or manipulated by the end effector is observed from the viewpoint, based on the low-dimensional posture data, the direction data, and the position data, using a pre-trained machine learning model; An information processing method including:

8. A program to be executed by a computer, Acquiring posture data of an end effector capable of grasping or manipulating an object, line-of-sight direction data when observing the object from a certain viewpoint, and position data of the viewpoint; generating low-dimensional posture data by reducing the number of dimensions of the posture data; generating an image that includes the density of the object, the image being generated when a scene in which the object is grasped or manipulated by the end effector is observed from the viewpoint, based on the low-dimensional posture data, the direction data, and the position data, using a pre-trained machine learning model; Programs including.

Images