Method and system for controlling artwork generating robot using robot interface system

Abstract

A method and system for controlling an artwork generating robot using a robot interface system. The method comprises the following steps: generating a digital prototype of the artwork by using a rendering algorithm of a robot interface system; setting a digital canvas by using a robot interface system; displaying the digital prototype of the artwork on a display of the robot interface system; calibrating the robot and the robot environment to enable the robot to generate a robot generated artwork corresponding to the digital prototype; using an artwork generation robot to convert the digital prototype into a physical robot to generate an artwork; a robot interface system is used to adjust settings and interact with the robot when converting the digital prototype to the robot-generated artwork. The step of calibrating the robot is performed by using a calibration tool of the robot interface system.

Description

Technical Field

[0001] The present invention generally relates to the fields of robotics, computer vision, artificial intelligence, and user interfaces. The technical field also includes the development of methods for automatically or semi-automatically generating digital images based on user input information and methods for implementing these images using a robotic system. Background Art

[0002] In all systems involving human-machine interaction, the elements that enable such interaction are called human-machine interfaces. Currently, many existing robotic systems come with user interfaces. For example, ABB's RobotStudio (https: / / new.abb.com / products / robotics / robotstudio) provides robotic modeling and programming software. Through this system, users can create and edit robotic models, as well as program and optimize their operation. However, the RobotStudio system does not have the function of enabling a robot to draw using a user interface, but allows designers to create applications for robots for various purposes.

[0003] Another example is the existing system Scribit (https: / / create.scribit.design), which is a robot that can draw on vertical surfaces. It comes with an application for managing and downloading images. However, Scribit does not provide drawing capabilities based on a user interface or cloud-based drawing capabilities.

[0004] Another example is the known system Line-us (https: / / www.line-us.com / ), which is a robot that can draw on flat surfaces. It comes with an application that allows users to upload images and draw on their own. However, Line-us does not provide the function of working using the cloud or drawing using a user interface.

[0005] Finally, the AxiDraw (https: / / www.axidraw.com / ) robot can also draw on flat surfaces. It comes with software that allows users to upload images and create drawings on their own. However, similar to Line-us, AxiDraw does not provide the function of working using the cloud or drawing using a user interface.

[0006] These similar systems and prototypes provide various solutions in the fields of robotics and drawing, but none of them can fully match the functions and capabilities disclosed in the present invention application. There is a need in the art for a system that allows users to interact with a robot, draw using a user interface, and work using the cloud. Summary of the Invention

[0007] The system and method described in the present invention is a unique control system for a robotic artist, designed to enable users to create images using an art-generating robot. The interface system provides an intuitive and user-friendly process, allowing users to easily manage the drawing process from loading images and selecting settings to the final stage of creating artworks.

[0008] In its most general aspect, the present invention is a method and system for controlling an art-generating robot using a robotic interface system. The method includes the following steps: generating a digital prototype of an artwork using the rendering algorithm of the robotic interface system; setting up a digital canvas using the robotic interface system; displaying the digital prototype of the artwork on the display of the robotic interface system; calibrating the robot and its robotic environment so that the robot can generate a robot-generated artwork corresponding to the digital prototype; using the art-generating robot to convert the digital prototype into a physical robot-generated artwork; and adjusting settings and interacting with the robot using the robotic interface system during the conversion of the digital prototype to the robot-generated artwork. The step of calibrating the robot is carried out using the calibration tool of the robotic interface system.

[0009] A main objective of this system is to enable any user, regardless of their level of technical training and experience with similar systems, to utilize and understand the process of using a robot for drawing. The system allows users to customize the entire technical part of creating artworks using the robot according to the users' preferences and requirements. The built-in control system of the robotic artist ensures the implementation of the user-specified settings and the control of the drawing process. After the settings are completed, the user can initiate the drawing process of the robot and is able to interrupt, pause, or change the drawing process.

[0010] In addition, the system provides a drawing scenario without a starting image, allowing users to select different image options. This broadens the application of the present invention and makes it applicable to various scenarios and situations.

[0011] Furthermore, the interface and action sequence for controlling the robotic artist are based on the principles of usability and intuitiveness. This ensures that people with different levels of technical training, including disabled persons (special groups), can easily operate. The present invention can be combined with various types of robots and thus becomes a general solution for controlling the drawing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present invention is illustrated by way of non-limiting examples and the accompanying drawings, in which reference numerals denote corresponding parts, wherein:

[0013] Figure 1 shows a representative user authorization page of the robotic interface system;

[0014] Figure 2Shows a representative page of the robotic interface system for naming digital prototypes of artworks;

[0015] Figure 3 Shows a representative page of the robotic interface system for selecting digital prototypes of artworks from the provided list;

[0016] Figure 4 Shows a representative page of the robotic interface system for uploading reference images;

[0017] Figure 5 Shows a representative page of the robotic interface system for controlling reference settings;

[0018] Figure 6 Shows a representative page of the robotic interface system for uploading an image as part of a pattern;

[0019] Figure 7 Shows a representative page of the robotic interface system for drawing an image as part of a pattern;

[0020] Figure 8 Shows a representative page of the robotic interface system for setting a signature;

[0021] Figure 9 Shows a representative page of the robotic interface system for setting the "lens" feature;

[0022] Figure 10 Shows another representative page of the robotic interface system for setting the "lens" feature;

[0023] Figure 11 Shows a representative page of the robotic interface system for setting color harmony;

[0024] Figure 12 Shows a representative page of the robotic interface system for setting the "historical style" feature;

[0025] Figure 13 Shows a representative page of the robotic interface system for setting the "random element generator" feature;

[0026] Figure 14 Shows a representative page of the robotic interface system for setting the "lighting effect" feature;

[0027] Figure 15 Shows a representative page of the robotic interface system for setting the "collage" feature;

[0028] Figure 16 Shows a representative page of the robotic interface system for setting the "mixed media" feature;

[0029] Figure 17 Shows a representative page of the robot interface system for setting the "interior" feature;

[0030] Figure 18 Shows a representative page of the robot interface system for setting the "virtual art assistant" feature;

[0031] Figure 19 Shows a representative page of the robot interface system for setting the "emotional color" feature;

[0032] Figure 20 Shows a representative page of the robot interface system for setting the "abstract concept generator" feature;

[0033] Figure 21 Shows a representative page of the robot interface system for setting the "musical inspiration" feature;

[0034] Figure 22 Shows a representative page of the robot interface system for setting the "personalized template" feature;

[0035] Figure 23 Shows a schematic diagram of system integration using a virtual reality device;

[0036] Figure 24 Shows a representative page of the robot interface system for setting the "split into floor plan" feature;

[0037] Figure 25 Shows a representative page of the robot interface system for setting the "algorithmic drawing" feature;

[0038] Figure 26 Shows a representative page of the robot interface system for setting the "fractal generator" feature;

[0039] Figure 27 Shows a representative page of the robot interface system for setting a digital canvas;

[0040] Figure 28 Shows a representative page of the robot interface system for selecting a palette type;

[0041] Figure 29 Shows a representative page of the robot interface system for selecting palette colors;

[0042] Figure 30 Shows a representative page of the robot interface system for selecting a drawing style;

[0043] Figure 31 Shows examples of digital prototypes of artworks in different styles;

[0044] Figure 32 Shows a representative page of the robot interface system for the "Start Rendering" feature;

[0045] Figure 33 Shows an example of the statistics of the digital prototype of the artwork;

[0046] Figure 34 Shows a representative page of the robot interface system for setting the downloaded digital prototype video of the artwork;

[0047] Figure 35 Shows an example of the uploaded digital prototype video of the artwork;

[0048] Figure 36 Shows a representative page of the robot interface system for calibration;

[0049] Figure 37 Shows a representative page of the robot interface system for selecting calibration from a list;

[0050] Figure 38 Shows a representative page of the robot interface system for editing the name of the selected calibration;

[0051] Figure 39 Shows a representative page of the robot interface system for brush calibration;

[0052] Figure 40 Shows a representative page of the robot interface system for calibrating the position of the art surface (such as a canvas);

[0053] Figure 41 Shows a representative page of the robot interface system for palette layout calibration;

[0054] Figure 42 Shows a representative page of the robot interface system for calibrating the brush wiping system;

[0055] Figure 43 Shows a representative page of the robot interface system for calibrating the automatic brush replacement system;

[0056] Figure 44 Shows a schematic diagram of the user interacting with the robot interface system;

[0057] Figure 45 Shows a representative page of the robot interface system for selecting the connection of a specific robot to the control system;

[0058] Figure 46 Shows a schematic diagram of the robot achieving connection through the robot interface system;

[0059] Figure 47Shows a schematic diagram of starting the drawing / art creation process;

[0060] Figure 48 Shows a schematic diagram of pausing the drawing / art creation process;

[0061] Figure 49 Shows a representative page of the robot interface system displaying statistics related to the drawing / art creation process;

[0062] Figure 50 Shows a schematic diagram of manually changing the paintbrush;

[0063] Figure 51 Shows a schematic diagram of automatically changing the paintbrush;

[0064] Figure 52 Shows a schematic diagram of adjusting the paintbrush height using the robot interface system;

[0065] Figure 53 Shows a representative page of the robot interface system for setting manual strokes;

[0066] Figure 54 Shows another representative page of the robot interface system for setting manual strokes;

[0067] Figure 55 Shows a representative page of the robot interface system for deleting manual strokes;

[0068] Figure 56 Shows a representative page of the robot interface system for selecting and applying a mask;

[0069] Figure 57 Shows a representative page of the robot interface system for setting mask interface characteristics;

[0070] Figure 58 Shows a representative page of the robot interface system for using a mask to select an area;

[0071] Figure 59 Shows a representative page of the robot interface system for resetting a mask;

[0072] Figure 60 Shows a representative page of the robot interface system for generating an image using text;

[0073] Figure 61 Shows a representative page of the robot interface system for generating 1 - 4 images using a text description;

[0074] Figure 62 Shows a representative page of the robot interface system for selecting a generated image based on a text description;

[0075] Figure 63 Shows a representative page of the robot interface system for generating art themes;

[0076] Figure 64 Shows another representative page of the robot interface system for generating art themes;

[0077] Figure 65 Shows a representative page of the robot interface system for inputting a recommended art theme into a field to generate an image based on a text description;

[0078] Figure 66 Shows a representative page of the robot interface system for splitting a large image into multiple parts;

[0079] Figure 67 Shows a schematic diagram of the horizontal mounting system for the art surface;

[0080] Figure 68 Shows a schematic diagram of the vertical mounting system for the art surface;

[0081] Figure 69 Shows a representative page of the robot interface system for implementing a random performance;

[0082] Figure 70 Shows a representative page of the robot interface system for implementing a special performance;

[0083] Figure 71 Shows an example of a robot performing a "performance";

[0084] Figure 72 Shows a schematic diagram of several robot manipulators of the same model working on a single piece of art;

[0085] Figure 73 Shows a schematic diagram of different types of robots generating art;

[0086] Figure 74 Shows a schematic diagram of creating art by combining various types of robot manipulators;

[0087] Figure 75 Shows a schematic diagram of creating art by combining different types of robots;

[0088] Figure 76 Shows a representative page of the robot interface system for computer vision interface features;

[0089] Figure 77 Shows a schematic diagram of creating art using computer vision;

[0090] Figure 78Displays a representative page of the robot interface system for one - key drawing;

[0091] Figure 79 Shows an example of a two - step collaborative art creation process;

[0092] Figure 80 Shows a schematic diagram of the robot and human creating art simultaneously;

[0093] Figure 81 Shows a schematic diagram of the options for the user to interact with the interface through voice control;

[0094] Figure 82 Shows a schematic diagram of the options for the user to interact with the interface using a neural headset;

[0095] Figure 83 Shows a schematic diagram of the parallel control of the robot. Detailed implementation mode

[0096] The terms used in the above description are as follows:

[0097] "Robot - assisted drawing algorithm" is an action sequence that involves the process of converting the creative vision expressed by the user in the form of a digital image, oral or written idea, or story into a visual artwork by a robot artist. This process involves selecting the desired style, color scheme, and other settings to create the final artwork. The drawing algorithm determines how the robot interprets and transforms the user's creative intention into a physical artwork.

[0098] "Digital prototype of an artwork" is a virtual representation of the art image expected to be created by the robot. It is generated by software and based on specified parameters such as color, style, and stroke direction. This prototype then becomes a stroke table and a set of coordinates that determine how the robot moves when creating the real artwork.

[0099] The "digital prototype of an artwork" also allows the user to visualize the final result of the robot's work before the robot starts creating. It gives the user the opportunity to adjust and optimize the image and enables the user to preview the final artwork. Then, the user can adjust and refine the parameters until a satisfactory result is obtained and transmit these parameters to the robot for creating the physical artwork. Therefore, the digital prototype is an important tool for planning, controlling, and visualizing the process of creating artworks using a robot.

[0100] In short, the "digital prototype of an artwork" is a virtual visualization of the future artwork, evaluated and adjusted by the user, and represents a set of coordinates that determine the subsequent movement of the robot.

[0101] After the digital prototype is finalized, the painting algorithm converts this virtual data into a physical image, transforming the "digital prototype of the artwork" into a "robot image", such as an actual artwork. The "robot image" can include: traditional paintings, such as portraits, landscapes, etc.; oil paintings on canvas; illustrations and drawings on any surface, such as a physical object, canvas, or clothing; unfinished paintings, such as sketches, outlines, and drafts; teaching sketches for art teaching; abstract paintings created as part of an educational program (e.g., to visualize mathematical formulas or as a result of the operation of an educational algorithm); decorative images that can be used for interior decoration or to create a desired atmosphere or environment, including murals, wall art, or interior decoration designs; commercial images, such as logos, brands, or other types of advertising images; and / or industrial designs, such as paintings on items like glass, tiles, porcelain, etc. Thus, the "robot image" can represent a wide range of forms and genres, reflecting the diversity of art and its applications.

[0102] The present invention illustrates a most typical scenario of using the method and system of the present invention for a single user. It is apparent from this specification that this scenario can be extended to any desired number of users. Further, some extended scenarios will be described in more detail later in this specification.

[0103] The interaction between the user and the interface includes the following main steps: rendering or applying the painting algorithm; calibrating the robot and the robot environment, such as the painting device; interacting with the robot during the creation of the physical artwork. Further, the step of rendering or applying the painting algorithm includes additional sub-steps: reference settings; advanced custom settings; digital canvas settings.

[0104] Rendering the painting algorithm refers to the process of converting a digital image into a complete set of robot commands sufficient for the robot to create a complete physical artwork (such as, but not limited to, an oil painting on canvas). Rendering also includes displaying the resulting digital prototype of the artwork on a computer screen or any other screen-equipped device used.

[0105] During the rendering process, the user preferably first starts the rendering interface by opening the rendering interface in any standard browser on the user's computer or another device and completing Figure 1 the authorization process shown.

[0106] When creating a digital prototype of an artwork, the user first assigns it a unique name, thereby creating and reserving storage space in the relevant database of the painting algorithm to accommodate the digital prototype of the artwork and all accompanying files. Figure 2 Alternatively, the user can select a previously created digital prototype of an artwork from a list previously stored in the database. Figure 3 .

[0107] One of the most common ways to create a digital prototype of a work of art is to upload a digital image as a reference. For example, the user can upload pictures in jpg, png, or any other format. The digital image can be replaced or deleted when the user makes a request. Figure 4 。

[0108] Then, the user applies a series of desired settings to the loaded image. Such settings are divided into reference settings, advanced professional settings, and digital canvas settings as appropriate. The reference settings preferably include brightness adjustment, contrast adjustment, color saturation change, and color shift. Brightness adjustment means being able to make the reference image darker or brighter. Contrast adjustment means being able to enhance or weaken the transition from one color to another. Color saturation change means being able to make the colors of the reference image brighter or softer. Color shift means changing all the colors of the reference image relative to the color wheel (left: counterclockwise; right: clockwise). For example, in the reference image, green can become blue, blue can become red, and so on. For the convenience of the user, the interface preferably displays two images: one is the image before the settings are applied, and the other is the image after the settings are applied. Figure 5 After performing the reference settings, the user can save the reference settings or reset the reference settings.

[0109] In a preferred embodiment, the interface system further includes advanced custom settings, which include a variety of tools for making more in-depth and detailed adjustments to the image. In these settings, the user can find professional functions that, when working with drawing algorithms and robots, can solve some specific user problems (such as art, design, and marketing), making the robot artist a multi-functional intelligent assistant.

[0110] There are also many settings for educational purposes, taking into account both the analytical and creative aspects. There are also some settings for art therapy and / or rehabilitation purposes, such as serving disabled people.

[0111] One feature of the interface system is the pattern setting. This feature can create or load patterns, such as repeating patterns or shapes, and then the robot can use these patterns to create or fill an image. For example, the user can draw a star in a graphic field or upload a star image, and then the drawing algorithm will automatically fill the drawing area with the star pattern, repeating the uploaded or drawn star image according to various predetermined parameters. Figure 6-7 。

[0112] The user can further customize the process by creating a custom signature using the interface system. This tool allows the user to incorporate the user's signature into the digital prototype, thus personalizing the digital prototype painting. The user can upload a digital image of their signature in the interface or directly create a signature using a graphic field, and then the robot will copy the signature to the real image at the position and color selected by the user or automatically selected.Figure 8 .

[0113] Another feature of the interface system allows users to customize the lens. Users can use the lens to add different optical effects to regular photos / images. For example, using the recommended tool, the selected part of the image can be distorted. Figure 9 . For example, as Figure 10 shown, a photo of a cat can be distorted to make its face look longer.

[0114] The interface system further preferably includes a color harmonizer setting tool. This tool analyzes the primary colors in the uploaded image and provides the user with a palette of coordinated digital prototype paintings. This helps to create an aesthetically pleasing color combination that reflects the tone or atmosphere of the original image. This tool analyzes and edits the reference color until it is coordinated, while automatically selecting the color palette required for drawing. Figure 11 .

[0115] The interface system further preferably includes a historical style setting tool. This tool allows users to select a style from various historical art styles such as Impressionism, Cubism, Surrealism, etc. and apply it to the reference image. For example, users can upload a modern photo, and the drawing algorithm will create a digital prototype of a painting in the style of Van Gogh or Picasso. Figure 12 .

[0116] The interface system also preferably includes a random element generator setting tool. This tool adds random elements or textures to the digital prototype painting, making each piece unique. This may include randomly dripping paint, creating background textures, or introducing unexpected images into the composition. Figure 13 .

[0117] The interface system further preferably includes a lighting effect customization tool. This tool allows users to create realistic lighting effects on the digital prototype painting, such as sunlight reflections, soft glows, or shadows. This gives depth and volume to the original image, making it more vivid and lively. Figure 14 .

[0118] The interface system also preferably includes a collage setting tool. Using this tool, users can create collages by combining different images into a single reference image. The drawing algorithm can analyze the photos uploaded by the user, cut out the individual elements according to the user's instructions, and place them on the digital canvas, allowing the user to create a composition that integrates different scenes, objects, or people. Figure 15 .

[0119] The interface system further preferably includes a mixed media mode setting tool. This tool allows the simultaneous use of multiple materials and artistic techniques, such as combining watercolor paintings, oil paintings, and pencil drawings. The robot will always apply different painting styles to create multiple layers of complex textures on a single drawing surface. Figure 16 .

[0120] The interface system also preferably includes a design setting tool. This feature allows users to upload interior photos for displaying the resulting artwork. Then, the rendering algorithm can adjust the color scheme and style of the rendered prototype (and the subsequent resulting artwork) so that the resulting artwork harmoniously integrates into the surrounding space. The algorithm analyzes the tones and design features of the room and then recommends the best settings for creating the digital prototype, which will become an integral part of the interior design. Figure 17 。

[0121] The interface system further preferably includes a virtual art assistant tool. This tool provides suggestions and opinions to the user based on the current state of the image, helping to improve the composition or color palette. For example, the assistant may propose adding emphasis (bright and / or rich details) to certain areas of the image to draw attention to the focal point. Figure 18 。

[0122] The interface system further preferably includes an emotional color tool. This tool analyzes the emotional content of the uploaded image (or the emotions expressed by the user through the webcam) and recommends color and texture settings that reflect the given emotional tone. Figure 19 。

[0123] The interface system also preferably includes an abstract concept generator tool. This tool allows users to create abstract artworks using an algorithm that randomizes shapes, lines, and colors, enabling users to experience abstract art without the necessary skills for manual creation of such works. Figure 20 。

[0124] Finally, the interface system further preferably includes a music inspiration setting tool. By using this tool, the drawing algorithm analyzes the selected music and creates images that visually reflect the rhythm, melody, and tone of the music work, allowing users to transform what they hear into paintings. Figure 21 。

[0125] In a preferred embodiment of the interface system, users can create and save personalized drawing templates, including color palettes, styles, and other settings. These personalized templates can later be used to quickly start new projects or shared with other users. Figure 22 。

[0126] Users can interact with the robotic artist and its virtual reality interface using virtual reality (VR) headsets to draw in 3D space or fully immerse themselves in observing the creation process of digital prototype artworks. Figure 23 。

[0127] Another tool of the preferred embodiment of the interface system allows the user to create a visual effect of depth and perspective in an image by dividing the image into foreground, middle ground, and background. The robotic artist draws objects with different artistic characteristics on various planes to create impressive and attractive images, such as images with a three-dimensional effect. Figure 24 。

[0128] The interface system also preferably includes an algorithmic drawing tool mainly for educational purposes. This tool allows the user to set an image through software commands or mathematical formulas. For example, for educational purposes, the user can input the equation of an Archimedean spiral, and the robot will draw the corresponding geometric shape. Figure 25 。

[0129] Similarly, the fractal generator tool creates images based on repetitive mathematical patterns (fractals). For example, for educational purposes, the user can try to create fractals with different parameters, and the robot then draws the fractals. Figure 26 。

[0130] The "Advanced Professional Settings" tool can be used to modify, improve, and supplement new instruments for all the listed tools and panels.

[0131] To enable the robot to create artworks on the canvas, the system includes a set of digital canvas setting tools. These tools are preferably divided into basic setting tools and artistic setting tools. The basic setting tools preferably include: determining the size of the digital canvas, for example, by entering the width value and height value of the canvas; trimming the digital canvas, for example, by reducing the size of one or more of its sides; selecting the background color of the digital canvas from several predetermined colors considering the rendering to be initiated. Then, the user can save the selected digital canvas settings. Figure 27 。

[0132] The artistic setting tools preferably include palette type selection, palette color selection, and artistic style selection tools. Through the palette type selection tool, the user can select the palette type from various preset options. For example, the standard palette provided by default consists of six primary colors: white, gray, black, red, blue, and yellow. One type of palette for "portraits" replaces black with deep warm and deep cool tones. Another type of palette generates palette colors based on the colors extracted from an image. The number of colors in the palette can vary from one to multiple. For the convenience of the user, all the colors of the palette are displayed side by side on the interface. In addition, by clicking on each color in the palette, the user can see the selected color in an enlarged view. The user can add colors to the palette and delete colors from the palette, or use the selected palette without making changes. Figure 28 。

[0133] The palette color selection tool allows users to adjust the hue of each color part of the palette by enabling the color picker by clicking on the corresponding cell. Users can add new colors to the palette, delete existing colors, or use the selected palette without making changes. The interface system preferably includes tips that give suggestions on selecting specific pigments for the desired artwork. Figure 29 。

[0134] The drawing style selection tool includes a large number of items available for use when the robotic artist is drawing, including various styles built into the drawing algorithm. By selecting a specific style, the user automatically enables certain corresponding parts of the drawing algorithm. A style is a set of parameters that define certain characteristics of the drawing process of the robotic artist, including but not limited to parameters such as the number of layers, brush size, stroke length of each layer, and the canvas paint mixing model. Figure 30-31 。

[0135] After all the above parameter setting steps are completed, the user can start the rendering process. During this process, the algorithm converts the reference image into a digital prototype of the artwork and generates a set of commands that enable the robot to complete the process of creating the actual physical artwork. Figure 32 。

[0136] After rendering is completed, the digital prototype of the artwork is preferably saved in the cloud. The saved digital prototype of the artwork can be used for drawing, deleting, or post - editing. In addition, for the convenience of users, the interface system can display useful statistics, such as the number of drawn strokes, drawing speed, and the number of layers and brushes required for drawing. Figure 33 。Users can also obtain a video showing the step - by - step process of creating the digital prototype of the artwork. Figure 34-35 。

[0137] The calibration of the robot and its environment is an important step in the process of user interaction with the interface system. This process is carried out through professional calibration features designed in the interface system for fine - tuning the parameters of the robot and its environment. Figure 36 。The purpose of calibration is to set such robotic working parameters so that it can correctly perceive and interpret the surrounding space and the objects therein. The calibration features of the interface system give users ample opportunity to configure and manage the calibration process of the robot and its environment. Figure 37 。

[0138] In a preferred embodiment, the user can create new calibration options by customizing all parameters to meet specific conditions and requirements. This enables the robot to adapt to specific working conditions as accurately as possible. Alternatively, the user can choose to edit existing calibration options. This option allows the user to change the parameters of the created calibration options at any time, clarify or adjust these parameters according to changes in conditions or requirements. Further, when a calibration option is no longer needed, the user can delete it, thus freeing up space to create new options. Finally, the user can choose to use prefabricated calibration options, i.e., a calibration option that has been created and saved, which can significantly speed up the preparation process and shorten the calibration time. Figure 38 Therefore, the calibration characteristics of the interface system can not only fine-tune the robot and its environment, but also manage the created calibration options, making the system highly flexible and user-friendly.

[0139] This system preferably includes a variety of calibration tools. Specifically, the interface system preferably includes a brush calibration tool. An important step in the calibration process is to set the correct brush angle. The user can use the brush calibration tool to precisely control the pressure and angle of the brush used by the robot, which will affect the thickness and characteristics of the line strokes during drawing. Figure 39 。

[0140] The robot must know the exact position and size of the surface used for the drawing process. Figure 40 。To set these parameters, the system provides the user with a drawing surface calibration tool.

[0141] The system further includes a palette calibration tool, which allows the robot to determine the position of the palette and pigments used for the drawing process. After determining the position, the robot can autonomously select and change colors during operation. Figure 41 。

[0142] The system also preferably includes a wiping system calibration tool. An important element of the environment is the wiping system, and its position must be accurately determined during the calibration process so that the robot can independently clean the pigments on the brush during the drawing / painting process, thereby improving the autonomy of the robot and the quality of the resulting image. Figure 42 。Optionally, the automatic brush replacement system can be calibrated. This allows the robot to independently replace the brush during the drawing process, thus expanding its functions. Figure 43 。

[0143] The above process of calibrating the robot and its environment includes multiple important steps to ensure the correct and autonomous operation of the robot during the drawing / painting process.

[0144] In this system, the user can also interact with the robot during the drawing / painting process. To achieve this interaction, the user preferably accesses the interface system through any standard browser on his computer or another communication device. The scenario at this time may be that the user is near the computer connected to the robot, or the user is far away from the robot and rendering remotely. In the latter case, the user can create a digital painting prototype and start drawing at any convenient time, or issue a command to another person (operator) to start drawing. Figure 44 .

[0145] The interaction with the robot during the drawing / painting process is a set of actions performed by the user using the control tools of the interface system. This process includes the following steps: selecting and connecting the robot; starting the drawing / painting process; pausing the drawing / painting process; viewing statistics. The user selects the robot from the list of available devices and connects it with the control tools of the interface system. This allows the user to customize the interaction between the interface and a specific robot. Figure 45-46 After connecting the robot and setting all necessary parameters, the user can start the drawing process. The robot starts to perform the specified actions according to the drawing algorithm. Figure 47 At any time during the drawing process, the user can pause the process. If necessary, the user can adjust the drawing process or change the robot's working parameters. Figure 48 The interface also allows users to view drawing statistics in real time. Users can track the progress of work: drawing time, material consumption and other parameters. Figure 49 .

[0146] The interface system allows the user to choose between automatic and manual brush changes during the drawing process. In automatic mode, the system independently determines when to change brushes based on the specified parameters of the digital prototype of the artwork and completely autonomously changes one brush to another. In manual mode, the user controls the process of changing brushes together with the interface system, so that the drawing process can be more finely managed and the desired results can be achieved. Figure 50-51 .

[0147] Users can adjust the height of the brush relative to the drawing surface in real time. This allows users to control the thickness and intensity of lines in the real image. Users can also adjust how deeply the brush is dipped into the paint, thereby controlling the amount of paint picked up and, in turn, the color saturation of the resulting image. Furthermore, users can adjust the height of the brush as it is wiped, thereby controlling the amount of paint removed from the brush. Figure 52 . Therefore, the interaction with the robot during drawing is a set of control actions that provide a high degree of control during the drawing process and allow the user to obtain the desired results.

[0148] Additional options for the user to interact with the interface system to enhance the artistic effect include the steps of the user adding manual strokes to the digital painting prototype and applying a mask before the rendering process. The system allows the user to add manual strokes using a mouse or a touch screen. Then, the robot repeats the strokes added by the user in the real image in the same order and direction as the strokes added by the human hand (or alternative signal) to the digital prototype of the artwork through the interface using a computer screen or another device. Such manual strokes can be used to: completely create the digital prototype of the painting by manual strokes; correct the defects of the digital prototype of the painting after rendering, or add more author strokes to the digital prototype of the painting if necessary. The custom script for adding manual strokes includes the following options: adding manual strokes before the basic scene image loading phase; adding manual strokes after the basic scene image loading step; adding manual strokes after the rendering phase of the basic scene.

[0149] When adding manual strokes, the user interacts with the interface system by first opening the digital prototype painting from a list. The user preferably selects the desired digital prototype painting from the list using any standard browser on the user's computer or another suitable device. Next, before creating the manual strokes, the user opens the manual stroke tool of the interface system and selects a number of manual stroke settings. Figure 53 . These settings preferably include: the line size of future strokes, the line color of future strokes through the color palette, the line type (such as a smooth curve or a straight line), and the background color selection. Figure 54 . After selecting the desired settings, the user adds strokes to the digital painting prototype using a mouse or a touch screen. If the manual strokes do not visually meet the expected goals and artistic vision, the user can also erase them separately or completely. If necessary, the user can also restore the manual strokes after erasing. After adding manual strokes to the digital painting prototype, the user can save the manual strokes. After saving, the manual strokes are loaded into the stroke table of the selected digital prototype picture, and then the user can start using them. The user can delete the manual strokes from the digital prototype of the painting after saving. Figure 55 . Next, the user returns to the basic scene according to the option of using manual strokes.

[0150] Sometimes, it is necessary to emphasize or clarify specific details of the digital prototype. This can be achieved by applying a mask before the rendering process. The process of using the mask is as follows: select the area with key details in the original image and display the selection result on the user's computer screen; display the application area of the mask on the user's screen; selectively add more brush stroke layers to the selected area during the rendering process to increase the visual impact of the image.

[0151] The user can also apply a mask before the basic scene rendering stage. When applying the mask, the user interacts with the interface system by opening a digital prototype painting from a list; then opens the mask tool of the interface system; and uses this mask tool to add a mask to the selected digital prototype painting. Figure 56 When setting up the mask tool, the user makes selections for a number of settings. Such settings preferably include: stroke type (straight line or freehand), stroke color, fill color within the selected area, and an invert button. Figure 57 After selecting the settings, the user selects the area within the original image where the changes will be applied and exits the detail selection mode. It should be noted that in the original image, the user can highlight as many key details as they deem necessary. For the convenience of the user, the interface system preferably displays the area with highlighted details as a separate image. Figure 58 After exiting the key feature selection mode, the user can apply the mask. When the selection area needs to be changed or the user wants to completely undo the mask, the user can also reset the mask. Figure 59 After applying the mask to the original image, the user can start the rendering process according to the above basic scene. During the rendering process, key details can be highlighted by adding more strokes to the selected area.

[0152] As described above, this creative system can require the interface system to generate an image using a text description to execute a drawing script without an initial image. In this case, except for the image loading stage, the user interacts with the interface in a manner similar to the above basic scene. Figure 60 During the process of generating an image using a text description, when the user interacts with the interface, they first access the image generation tool of the interface system and enter an image description. Figure 61 The user enters a text description of the image or its plot that they want to receive, such as "a robot in a flower bed, cartoon style". Then, the user starts image generation based on the entered text description, and the interface system automatically creates a digital image according to the description and displays the resulting image. The user can run the image generation as many times as they deem appropriate until they obtain the desired image. The user can also change the text description of the image and start the generation again. Figure 62 。

[0153] In another embodiment of the interface system, the interface system can provide the user with a theme for the resulting artwork. In this case, the user uses the interface system as an assistant to conceive ideas or plots for the future digital prototype of the painting. During the process of generating a drawing theme, the user opens the interface tool for generating a drawing theme and enters a command for generating the theme. Figure 63For example, the user may request the system to "propose five ideas for the painting 'Robot and the Future' and recommend one of them". Thus, the user starts the drawing theme generation process by entering a plain text description. Then, the user selects one or more themes from the generated drawing themes that best suit their request. The user can save the selected drawing theme using any convenient method, such as saving it by copying it to the clipboard. Figure 64 Next, the user enters an image description and selects an image based on the selected theme. Figure 65 。

[0154] The system disclosed herein is particularly suitable for creating large-format drawings / paintings. "Large-format drawing / painting" refers to the process of creating an image whose size exceeds the size of the robot's working area. In this case, more painting solutions are required, such as a device for moving the canvas or a technique for manually moving the robot to the desired position. By creating a large-sized image, deeper meaning can be injected into the artwork, thereby enhancing its appeal.

[0155] When the user interacts with the interface system, the robot can be used to draw a large image in two ways: splitting the large image into several small images, or using additional devices specifically designed to assist in creating large images, such as a large canvas feeding system or a system that enables the robot to move along and around the drawing surface.

[0156] In the system that splits a large image into several small images, the robot preferably performs subsequent drawing / painting on multiple artwork surfaces (such as several canvases). The steps of this process are basically the same as those in the above scenario. One difference is that during the process of creating a digital prototype of the painting, the user indicates in the interface system that the drawing will be performed on several surfaces. Then, the user specifies the number of surfaces on which the large image is to be created. The interface system then automatically splits the original image into the specified number of parts and displays the split result to the user, while numbering all the images. Next, the user performs the rendering process according to the basic scenario. At the end of the rendering process, the interface system displays the digital prototype of the painting divided into several parts and indicates the serial numbers of each part. In addition, before starting the drawing, the interface notifies the user of the part (serial number) of the digital prototype of the painting that will be drawn next. After a part of the large image is drawn, the interface notifies the user to replace the drawing surface with the next one and continue the rendering process. After the last part of the large image is drawn, the interface system notifies the user that the work is completed. It should be noted that for a large image split into multiple small images, no additional hardware is required for the drawing / painting process. Figure 66 。

[0157] When using a professional feeding system for a drawing surface in the process of drawing / painting large images, during the process of creating a digital prototype of a painting, the user indicates in the interface system that the drawing will be carried out through a large image drawing system. Further, during the rendering process, the interface automatically converts the image into a set of strokes and other robot commands, ensuring the smooth progress of the large image drawing process while taking into account the progressive feeding system of the drawing surface. Further actions are similar to those in the basic scenario.

[0158] Large images can be drawn on both horizontal and vertical planes. The user can flexibly choose the most suitable option according to their preferences and working conditions. When drawing on a surface located in the horizontal plane, especially for large sizes, for example, in combination with a drawing surface feeding system, it can ensure that the drawing surface is easy to handle and manipulate. This simplifies the robot calibration process, making the drawing more accurate and detailed, but may require more workspace. Figure 67 。

[0159] When drawing on a surface located in the vertical plane, for example, in combination with a system that enables the robot to move along the drawing surface, it can simulate a more traditional drawing process that may be more visually impactful to the user. Especially in limited space, this can make more efficient use of the space. Figure 68 。

[0160] Utilizing the large image drawing function of this system can expand the user's ability to create images using a robot, improve the autonomy of the robot's work, and enable the creation of larger-scale artworks. Through the large image drawing function, the system can draw large images with as little manual participation in the process as possible.

[0161] Another preferred function of this system is to enable and implement "performance" as part of the interaction of the interface system. In the above basic and alternative scenarios, when the user interacts with the interface system, the "performance" function can be turned on at any time. "Performance" refers to various movements of the robot within its technical capabilities, which can attract the attention of others but is not directly related to drawing. "Performance" can be manifested as the robot's movements such as moving forward, backward, left, right, up, down, and various turns. Individual parts of the robot can be moved separately or multiple parts can be moved simultaneously. Examples of performances can include imitating "stretching or kneading hands" as if taking a break during hard work, or imitating "deep thinking" or "a sudden inspiration" behavior. "Performance" is particularly suitable for academic environments or some special situations. Random "performances" can be carried out randomly during drawing, for example, once, or repeated at any configurable average interval between "performance" events. This provides an element of unpredictability, making the drawing process more vivid and interesting. For random "performances" during drawing, only a part of all the robot movements listed above can be used to avoid significantly increasing the total time of drawing the image. Figure 69 。

[0162] In addition, a special "performance" is a special performance that can be triggered by a user pressing a button or sending other signals. This enables a longer and more impactful wonderful performance, which can be used to attract attention at specific time points. For example, such a performance can be a combination of all the above movements of the robot and all its parts. Figure 70 。

[0163] During the process of starting a "performance", the user first selects the desired robot from the list and opens the "performance" option of the interface system through any standard browser on their computer or any other device used. Then, the user can use a switch to let the robot perform a random performance. For a random "performance", the user can also specify its repetition frequency. The user can also use a button or an alternative signal to start the special "performance" of the robot. After pressing the button, the special "performance" starts immediately. Figure 71 。The user can stop the execution of the "performance" through the interface when needed, or wait for it to complete automatically.

[0164] The system described in the present invention allows different types and quantities of robots to implement the basic and alternative drawing scenarios provided by the interface. When using multiple robots of the same type, the interface allows the user to use two or more robot manipulators to draw an image simultaneously, which improves the speed, accuracy, and effectiveness during the drawing process. Figure 72 。

[0165] The system also allows the user to use different types of robots. When drawing, the interface system supports the use of different types of robots, not just robotic arms. For example, unmanned aerial vehicles (UAVs) and robotic mobile platforms (robotic cars) can be used. The painting / drawing can be completed in various ways, including dropping dyes on the surface (e.g., spreading sand on concrete, splashing paint on a canvas, spraying dyes into the air) or using a paintbrush or other painting devices. Figure 73 。

[0166] In addition, different types of robots can be used in combination. The interface system allows multiple different types of robots to draw simultaneously. For example, a robotic mobile platform with paint on its pedal can create a part of the stroke, a UAV can create another part, and a robot manipulator can create a third part. Figure 74-75 。

[0167] The above-mentioned drawing / painting scenarios can be achieved through computer vision functions. Computer vision functions can be used to improve the accuracy and quality of the robot drawing process. In this case, the robot performs a certain number of strokes according to the basic scenario, and then uses a webcam installed at the desired position to record the current working state. Then, the algorithm generates the next set of strokes for drawing the image by referring to the information about this image received by the webcam.Figure 76-77 Through the application of computer vision, users can correct color and line errors in real time, and robots can also create images that better match the digital prototype of the painting. This feature improves the accuracy of the final artwork, ensuring that the image formed through the interface system matches the result of the rendering process. Applying computer vision in the above way makes the drawing process more vivid and adaptable, compensating for any deviations or errors in real time and improving the overall accuracy of the robot drawing process.

[0168] Users can also command the robot to create an image with the fewest steps. Figure 78 In this case, the user can use the interface to upload an image or provide commands to the algorithm to generate and select an image. After loading or generating and selecting an image, the interface system automatically selects all necessary default settings to ensure that these settings are optimally adapted to this image. After automatically selecting the settings, the user can use a neural headset or other convenient method to press the "draw" button or issue a command via a sound signal. The algorithm creates a digital prototype of the painting in real time based on the characteristics of the image and the selected settings. The robot then starts drawing the image, creating an artistic composition based on the generated data. In this mode, the user can interrupt, pause, or change the drawing process at any time, achieving full control over the created artwork.

[0169] Finally, the robot-human collaborative drawing function mode allows users to interact with the robot and create images together. This mode complements the basic scenario, bringing new possibilities to the creative process. The robot performs certain tasks according to the listed scenarios and modes, while the user contributes to and adds creativity to this process. Options for the robot-human collaborative drawing process can include two-step collaborative drawing. For example, the robot can start drawing, defining the overall outline of the work and determining its shape and proportion, and then the person can add colors, textures, and details to complement and refine the image. Figure 79 Alternatively, the robot and the person can process the image sequentially within a certain period. For example, the robot may first create the basic background or composition and then hand the work over to the person to add details, colors, and expressions. Finally, the robot and the human artist can work simultaneously. For example, the robot and the person can draw at the same time, complementing each other during the process and each starting to process a certain part of the image. By drawing at the same time, the two can jointly create an image, realizing the combination of creative ability and algorithmic ability. Figure 80 。

[0170] In addition to the above standard interface controls, the interface system can also be controlled by voice control or a neural headset. When using voice control, the user's voice can be used or all the commands described in this specification can be issued by imitating the voice. In addition, neural networks or other algorithms can be used for voice imitation. This may include photo upload, style selection, brightness setting selection, stroke direction, digital canvas cropping, robot calibration, and drawing supplies, etc. During the drawing process, the user can also use voice to issue commands such as "draw", "pause", and all other commands available in the drawing interface. This is particularly useful for disabled persons and can enhance comfort when preparing for drawing and painting. Voice commands support various languages and dialects, so the present invention can widely cover users. Figure 81 。

[0171] Furthermore, all the commands described in this specification can be issued using a neural headset. A neural headset is a head-mounted device for reading electrical signals from the brain. It may have different names, including: neural interface, neural helmet, neural ring, EEG headband, non-contact electroencephalograph (EEG), brain-computer interface headset, wireless electroencephalograph, etc. This device allows the user to control the interface according to the electrical activity of the brain signals and convert these signals into digital signals. This may be particularly suitable for disabled persons or those who prefer a more intuitive or novel interaction method. Figure 82 。

[0172] A key feature of the present invention is scalability and parallel control ability. The developed interface allows one or more robots to draw one or more images simultaneously. That is, for example, an operator user can control drawing on multiple robots through one or more laptops at the same time. This is achieved by means of a specially developed control system through which the user can coordinate the actions of multiple robots. Figure 83 。

[0173] It should be clear that the number of robots connected to the interface is not limited. In this case, the user can only control the robots connected to the interface that they own. This ability makes the system flexible and scalable, allowing the user to control robots around the world. No matter where the robots are located, they can be operated. The robots can be located in the same place or around the world. The user has the opportunity to use the developed interface to control all the robots they own at the same time. Therefore, the present invention can control multiple robots in parallel, which is beneficial for creating complex large-scale projects.

[0174] The present invention is preferably used in conjunction with standard interface elements. Such standard controls include "save", "cancel", "reset" buttons, "zoom out", "zoom in", "select", "mark" tools, etc. These elements are applied to the user interface according to standard logic. This ensures the convenience and familiarity of the user in using the interface.

[0175] The following sections of this application briefly introduce feasible solutions for controlling a robotic artist using an interface system and action sequences. These embodiments demonstrate the flexibility and adaptability of the present invention, which can be used in various situations and scenarios, including personal use, automatic image generation, text-based image generation, and sharing.

[0176] In a preferred embodiment, the user creates a digital prototype painting and then uses the interface to control the process of the robotic artist drawing it. This embodiment allows the user to have full control over the entire process from start to finish, providing maximum flexibility and creative freedom.

[0177] In another preferred embodiment, the user only needs to issue commands to the interface (such as "provide a picture" or "provide a picture scenario about penguins and give me 5 options"). The interface system then automatically creates a digital prototype painting, and then, as in the first scenario, the user controls the process of the robotic artist drawing the work.

[0178] In a further preferred embodiment, the user can create an image based on a text description. In this case, the user provides a text description of the desired image and its theme (such as "a robot in a flower bed, cartoon style"). The interface automatically creates a digital prototype painting based on this description, and then, as in the previous scenarios, the user controls the process of the robotic artist drawing the work.

[0179] In a further preferred embodiment, one user (such as an artist) creates a digital prototype painting, and another user controls the process of the robotic artist drawing the work. For example, this can be achieved in a remote studio that provides such services. This scenario enables you to combine the creative efforts of multiple people, increasing artistic and technological possibilities.

[0180] The above content of this specification describes the present invention with reference to specific exemplary embodiments of the present invention. However, various modifications and changes can obviously be made to the present invention without departing from the broader spirit and scope of the present invention as defined in the appended claims. Therefore, the nature of this specification and the drawings is illustrative rather than restrictive.

Claims

1. A method for controlling an art generation robot using a robot interface system, comprising: Generating a digital prototype of an artwork using a rendering algorithm of the robot interface system; Setting a digital canvas using the robot interface system; Displaying the digital prototype of the artwork on a display of the robot interface system; Calibrating the robot and the robot environment so that the robot can generate a robot-generated artwork corresponding to the digital prototype, wherein the step of calibrating the robot is performed by using a calibration tool of the robot interface system; Using an image generation robot to convert the digital prototype into a physical robot-generated artwork; and Adjusting settings and interacting with the robot using the robot interface system during the conversion of the digital prototype into a robot-generated artwork.

2. The method according to claim 1, wherein The step of generating the digital prototype includes: Uploading an external digital image using an upload input device of the robot interface system; and Converting the uploaded digital image into a digital prototype of an artwork using a rendering algorithm.

3. The method according to claim 1, wherein The step of generating the digital prototype includes: A user drawing a digital image using a digital drawing device of the robot interface system; and Converting the drawn digital image into a digital prototype of an artwork using a rendering algorithm.

4. The method according to claim 1, wherein The step of generating the digital prototype includes: Transmitting an oral description of a desired image to the robot interface system using an auditory input device of the robot interface system; and Converting the transmitted oral description of the desired image into a digital prototype of an artwork using a rendering algorithm.

5. The method according to claim 1, wherein, The step of generating the digital prototype includes selectively applying a predetermined image setting of the rendering algorithm to the digital prototype, thereby changing the digital prototype.

6. The method according to claim 5, wherein The predetermined image setting includes at least one of brightness adjustment, contrast adjustment, color saturation change, color shift, pattern, custom lens, and color harmonization.

7. The method according to claim 1, wherein The step of generating the digital prototype includes: Uploading an interior design image using an upload input device of the robot interface system; and Generating a digital prototype having features designed to fit the uploaded interior design image using a rendering algorithm.

8. The method according to claim 1, wherein The step of generating the digital prototype includes: Uploading a music work using an auditory input device; and Analyzing the uploaded music work using a rendering algorithm and creating a digital prototype that visually reflects the rhythm, melody, and tone of the uploaded music work.

9. The method according to claim 1, wherein The step of setting the digital canvas further includes setting the size of the digital canvas, trimming the digital canvas, and selecting the background color of the digital canvas.

10. The method according to claim 1, wherein, The step of setting the digital canvas further includes selecting a palette type.

11. The method according to claim 1, wherein, The step of setting the digital canvas further includes selecting the colors of the palette.

12. The method according to claim 1, wherein The step of setting the digital canvas further includes selecting a drawing style.

13. The method according to claim 1, wherein The step of calibrating the robot and the robot environment includes adjusting the angle and pressure of the art brush of the robot.

14. The method according to claim 1, wherein The step of calibrating the robot and the robot environment includes calibrating the drawing surface, and the step of calibrating the drawing surface includes determining the position and size of the surface of the physical robot-generated artwork to be generated.

15. The method according to claim 1, wherein The step of calibrating the robot and the robot environment includes determining the position of the palette and the paint to be used for generating the physical robot-generated artwork.

16. The method according to claim 1, wherein, The steps of calibrating the robot and the robot environment include the step of determining the position of the wiping system.

17. The method according to claim 1, wherein The steps of calibrating the robot and the robot environment include the steps of determining the position of the additional paintbrush, communicating the position to the robot, causing the robot to use the additional paintbrush, and automatically replacing one paintbrush with another.

18. The method according to claim 1, wherein, The step of adjusting settings and interacting with the robot using the robot interface system further includes the step of pausing the robot to adjust the robot operating parameters.

19. A system for controlling an art piece generating robot, comprising: A digital prototyping tool configured to generate a digital prototype of an art piece using a rendering algorithm; A canvas setting tool configured to manipulate a digital canvas; A display configured to display the digital prototype of the art piece thereon; A calibration system for calibrating the art piece generating robot and the robot environment so that the robot can generate a robot-generated art piece corresponding to the digital prototype; And A robot manipulator system configured to use the art piece generating robot to convert the digital prototype into a physical robot-generated art piece.

20. The system for controlling an art piece generating robot according to claim 19, further comprising an adjustment tool configured to adjust settings and interact with the robot during the conversion of the digital prototype into the robot-generated art piece.

Images