Robot drawing method, device, equipment and medium based on large model

Abstract

This invention provides a robot painting method, apparatus, device, and medium based on a large-scale model. The robot painting method includes: acquiring painting requirements and a target image; generating painting task information based on the painting requirements and target image; inputting the painting task information and target image into a multi-stage painting scheme generation model to obtain painting scheme descriptions and images corresponding to multiple painting stages; and sequentially inputting the painting scheme descriptions and images into a stroke planning model to obtain a painting stroke sequence for each painting stage, enabling the robot to paint based on the painting stroke sequence at each painting stage. This invention, based on a multi-stage painting scheme generation model and painting task information, divides the entire painting process into multiple different painting stages. The stroke planning model dynamically generates painting stroke sequences adapted to each painting stage, making the final robot painting effect closer to human level and more natural.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a robot painting method, apparatus, device, and medium based on a large model. Background Technology

[0002] With the rapid development of AI-generated content (AIGC), the intersection of artificial intelligence and art has received widespread attention, covering various art forms such as painting, poetry, music, and sculpture, and driving a great deal of work to explore the creative capabilities of machines. Currently, mainstream machine creation is result-oriented and cannot complete the creative process step by step. Existing robotic painting can only generate a single, fixed brushstroke planning strategy or use a predefined brushstroke strategy, generating painting trajectories through layered brushstroke planning and visual feedback, but this requires manual parameter predefinition. When the painting content and style change, the parameters need to be redefined, making it difficult to meet practical needs. Summary of the Invention

[0003] This invention provides a robot painting method, apparatus, device, and medium based on a large model, in order to overcome the deficiencies in the prior art.

[0004] This invention provides a robot painting method based on a large model, comprising: Obtain the painting requirements and the target image, and generate painting task information based on the painting requirements and the target image; The painting task information and the target image are input into a multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages; wherein, the multi-stage painting scheme generation model is based on a large model; The painting scheme description and the painting scheme image are sequentially input into the brushstroke planning model to obtain the painting brushstroke sequence for each painting stage, so that the robot can paint based on the painting brushstroke sequence in each painting stage; wherein, the brushstroke planning model is trained based on training data.

[0005] According to a robot painting method based on a large model provided by the present invention, the multi-stage painting scheme generation model includes a description generation module and an image generation module; the step of inputting the painting task information and the target image into the multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages includes: The painting task information is input into the description generation module to obtain multiple painting stages and a painting scheme description for each painting stage; The target image and the descriptions of each painting scheme are input into the image generation module to obtain the painting scheme images for each painting stage.

[0006] According to the present invention, a robot painting method based on a large model is provided, wherein the painting scheme description includes painting content, painting style, painting technique, and brushstroke characteristics, wherein the brushstroke characteristics are statistical features of the brushstrokes.

[0007] According to a large-model-based robot painting method provided by the present invention, the brushstroke planning model includes a language encoder, a visual encoder, a feature fusion layer, and a brushstroke prediction module; the step of sequentially inputting the painting scheme description and the painting scheme image into the brushstroke planning model to obtain the painting brushstroke sequence for each painting stage includes: The language encoder is used to encode the descriptions of each painting scheme to obtain the text feature vectors of the descriptions of each painting scheme. The visual encoder is used to encode each of the painting scheme images to obtain the image feature vector of each of the painting scheme images; Based on the feature fusion layer, the text feature vectors and image feature vectors of each painting stage are fused and logically associated to obtain the image-text fusion features of each painting stage; Based on the brushstroke prediction module, the image-text fusion features of each painting stage are processed sequentially to obtain the painting brushstroke sequence of each painting stage.

[0008] According to the robot painting method based on a large model provided by the present invention, the training process of the brushstroke planning model includes: Acquire multiple training data sets; wherein the training data sets include sample data and sample labels, the sample data sets include descriptions of painting sample schemes and image-text sample fusion features of painting sample scheme images, and the sample labels include sequences of painting brushstroke samples; The parameters in the language encoder, the visual encoder, and the feature fusion layer are fixed, and the parameters in the stroke prediction module are iteratively trained based on the training data until the training ends, thereby obtaining the stroke planning model.

[0009] According to the present invention, a robot painting method based on a large model, wherein generating painting task information based on the painting requirements and the target image includes: Extract the semantic information of the target image; The painting requirements and the semantic information are combined to obtain the painting task information.

[0010] According to a robot painting method based on a large model provided by the present invention, the extraction of semantic information of the target image includes: Segment the painting elements in the target image; Determine the descriptive information for each of the painting elements; The painting elements and the description information are used as the semantic information.

[0011] The present invention also provides a robot painting device based on a large model, comprising: The generation module is configured to acquire painting requirements and target images, and generate painting task information based on the painting requirements and target images; A painting scheme generation module is used to input the painting task information and the target image into a multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages; wherein, the multi-stage painting scheme generation model is based on a large model; The brushstroke planning module is configured to sequentially input the painting scheme description and the painting scheme image into the brushstroke planning model to obtain the painting brushstroke sequence for each painting stage, so that the robot can paint based on the painting brushstroke sequence in each painting stage; wherein, the brushstroke planning model is trained based on training data.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robot drawing method based on a large model as described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot drawing method based on a large model as described above.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robot drawing method based on a large model as described above.

[0015] The present invention provides a robot painting method, apparatus, device, and medium based on a large model. It generates painting task information based on painting requirements and target images, and adopts a multi-stage painting scheme generation model to divide the entire painting process into multiple different painting stages and generate detailed painting schemes accordingly. The brush stroke planning model dynamically generates painting brush stroke sequences adapted to each painting stage. Through this technical solution, different painting strategies can be adaptively adjusted according to the painting content and area of ​​each stage, flexibly responding to diverse painting scenarios, so that the final robot painting effect is closer to human level and more natural. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the robot painting method based on a large model provided by the present invention.

[0018] Figure 2 This is a simplified schematic diagram of the robot painting method based on a large model provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of the robot painting device based on a large model provided by the present invention.

[0020] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] Figure 1 This is a flowchart illustrating a robot painting method based on a large model, according to an exemplary embodiment. Figure 1 As shown in an exemplary embodiment, the robot painting method based on a large model includes steps 110 to 130, which are described in detail below.

[0023] Step 110: Obtain the painting requirements and target image, and generate painting task information based on the painting requirements and target image.

[0024] In this embodiment of the invention, the drawing requirement is a general textual description of the robot's drawing process. For example, the drawing requirement is to paint a portrait and landscape in the Impressionist style. The target image is the image that the robot needs to draw. In this embodiment, the drawing requirement and the target image can be directly provided by the user. In another embodiment, the target image can be generated directly based on the drawing requirement using a text-based image model. Drawing task information is generated based on the obtained drawing requirement and target image.

[0025] Step 120: Input the painting task information and the target image into the multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages; wherein, the multi-stage painting scheme generation model is obtained based on a large model.

[0026] In this embodiment of the invention, a multi-stage painting scheme generation model is pre-constructed based on a large model. First, sample data pairs are constructed, with the format of {painting sample task information, painting sample scheme}. Here, the painting sample scheme refers to the painting scheme instances corresponding to each painting stage of the painting sample task information determined by experts. The painting scheme instances include the corresponding painting sample scheme description and painting sample scheme image.

[0027] Because large models possess in-context learning capabilities, they can adapt to downstream tasks in the painting domain through question-and-answer examples. Therefore, we utilize the general painting knowledge contained in the large model to design structured painting schemes, provide the large model with painting sample task information and example data pairs of painting sample schemes, and optimize its understanding of the task through prompts, thereby obtaining a multi-stage painting scheme generation model.

[0028] After domain adaptation based on sample data pairs, the large model, based on its general painting knowledge learned from open network corpora, can generate structured, phased painting schemes from painting task information. The multi-stage painting scheme generation model, after obtaining the painting task information and target image, divides the painting process into a series of independent, time-sequential painting stages, generating painting scheme descriptions and images corresponding to each stage.

[0029] In this embodiment of the invention, the ChatGPT model and its iterative upgrades, or other open-source or closed-source large models, can be used as pre-trained large language models to construct a multi-stage painting scheme generation model.

[0030] Step 130: Input the painting scheme description and the painting scheme image into the brushstroke planning model in sequence to obtain the painting brushstroke sequence for each painting stage, so that the robot can paint based on the painting brushstroke sequence in each painting stage; wherein, the brushstroke planning model is trained based on training data.

[0031] In this embodiment of the invention, a brushstroke planning module is pre-trained. The drawing scheme description and drawing scheme image are sequentially input into the brushstroke planning model to obtain the drawing brushstroke sequence for each drawing stage. The robot executes the drawing brushstroke sequence for each drawing stage in sequence to complete the entire drawing process and obtain the final robot drawing.

[0032] The brushstroke sequence does not refer to the final painting itself, but rather to a complete record of the steps involved in the painting process within a corresponding stage. Each brushstroke sequence contains detailed information that tells the robot how to paint in specific locations on the canvas. The brushstroke sequence is a structured dataset containing the following information: Position and path: the starting point, ending point, and trajectory of a brushstroke, such as a straight line, a curve, or a dot. Brush attributes: Color, which is the color corresponding to each painting element; Size, which is the thickness or size of the brush; Shape, which is the shape of the brush head, such as round, square, oval, or more complex texture shapes. Direction / Angle, i.e., the angle at which the brush is placed; Sequential index: indicates which stroke this is.

[0033] In this embodiment of the invention, drawing task information is generated based on drawing requirements and target images. Based on a multi-stage drawing scheme generation model and drawing task information, the entire drawing process is divided into multiple different drawing stages. The brushstroke planning model dynamically generates drawing brushstroke sequences adapted to each drawing stage, thereby flexibly responding to diverse drawing scenarios and making the final drawing effect of the robot closer to human level and more natural.

[0034] The technical solution provided by this invention can be used for robotic art creation such as robotic painting and robotic calligraphy. Users can directly obtain physical paintings or calligraphy works completed by robots, such as oil paintings, acrylic paintings, gouache paintings, and watercolor paintings, by describing the target image in words or inputting a reference image. This allows for applications in art education, skills training, and personalized art customization. Furthermore, the technical solution provided by this invention can be extended to adaptive adjustment of robot operation strategies for dexterous robot skills.

[0035] In an exemplary embodiment of the present invention, generating painting task information based on the painting requirements and the target image includes: Extract the semantic information of the target image; The painting requirements and the semantic information are combined to obtain the painting task information.

[0036] In this embodiment of the invention, semantic information of the target image is extracted using a semantic segmentation model or a salient object extraction model. The semantic segmentation model can be a Bilateral Segmentation Network (BiSeNet), a Segmentation Transformer (SegFormer), a Deep Labeling (DeepLab) model, or a similar model. The salient object extraction model can be a nested U-shaped network (U2Net), an Interactive Segmentation Network (ISNet), a BiSalient Weight Network (BSWNet) model, or a similar model.

[0037] The extracted semantic information includes, but is not limited to, subject-object relationships and content attributes. For example, the target image is segmented into painting elements such as the subject (face, clothes, etc.), the foreground (trees, etc.), and the background (sky, etc.) to obtain the content attributes of the painting elements contained therein.

[0038] By concatenating the painting requirements with semantic information, we obtain the painting task information. For example, the painting requirements and semantic information can be concatenated into a text description in JSON format.

[0039] In an exemplary embodiment of the present invention, the extraction of semantic information from the target image includes: Segment the painting elements in the target image; Determine the descriptive information for each of the painting elements; The painting elements and the description information are used as the semantic information.

[0040] In this embodiment of the invention, the painting elements in the target image are segmented, and the descriptive information of each painting element is determined. For example, the target image is segmented into painting elements such as the main subject (face, clothing, etc.), foreground objects (trees, etc.), and background (sky, etc.), and the descriptive information of each painting element is determined. Furthermore, when the painting requirement is to paint a portrait landscape in the Impressionist style, the resulting JSON-formatted painting task information is as follows: { "Painting requirement": "Paint a portrait and landscape in the Impressionist style". "Semantic information": { "Portrait subject": { "Face": mask_array1, "Clothing": mask_array2 }, "Object foreground": { "Trees": mask_array3 }, "background": { "Sky": mask_array4, "Ground": mask_array5 } } }".

[0041] Here, mask_array represents the pixel mask matrix corresponding to each drawing element in the target image.

[0042] In an exemplary embodiment of the present invention, the multi-stage painting scheme generation model includes a description generation module and an image generation module; the step of inputting the painting task information and the target image into the multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages includes: The painting task information is input into the description generation module to obtain multiple painting stages and a painting scheme description for each painting stage; The target image and the descriptions of each painting scheme are input into the image generation module to obtain the painting scheme images for each painting stage.

[0043] In this embodiment of the invention, the multi-stage painting scheme generation model includes a description generation module and an image generation module. Painting task information is input into the description generation module, which breaks down the entire painting process into multiple painting stages and obtains a painting scheme description for each stage. The painting scheme description describes the painting content drawn in the corresponding painting stage, as well as the painting features required to complete the painting, in text form. The painting scheme descriptions for each painting stage can be in JSON format.

[0044] Using the target image as a reference, and the painting scheme descriptions for each painting stage, the images are sequentially input into the image generation module to obtain the painting scheme image corresponding to each painting stage. Each painting scheme image is generated based on the painting scheme image of the previous painting stage, combined with the corresponding painting scheme description.

[0045] In this embodiment of the invention, the description generation module is trained based on a large-scale Vision Language Action (VLA) model. Specifically, the VLA model can be a large-scale model such as Chatgpt, Gemini, Qwen, Deepseek, Claude, or Grok. The implementation of these models can be achieved either by calling APIs or by deploying the models locally.

[0046] The image generation module can use text-to-image models such as Stable Diffusion, VisionTransformer (ViT), Fast Lexically-Unconstrained Text-to-Image Generation (FLUX), SDXL Turbo (Stable Diffusion XLTurbo), and Kandinsky, as well as improved versions of each text-to-image model.

[0047] In an exemplary embodiment of the present invention, the painting scheme description includes painting content, painting style, painting technique, and brushstroke characteristics, wherein the brushstroke characteristics are statistical features of the brushstrokes.

[0048] In this embodiment of the invention, the description generation module, based on general painting knowledge learned from open network corpora, can generate structured, phased painting scheme descriptions from painting task information. The painting scheme description for each stage includes features such as painting content, painting style, painting techniques, and brushstroke characteristics. Brushstroke characteristics are represented by statistical features of the brushstrokes, including but not limited to average length, width, and curvature. The curvature value ranges from [0,1), with a larger value indicating a more curved brushstroke. Furthermore, the JSON-formatted painting scheme descriptions for each painting stage are as follows: { Phase 1: { "Painting Content": "Sky" Style: Impressionism Technique: Thick coating method "Stroke characteristics": { Average length: 15 Average width: 25 Curvature: 0.2 } }, Phase 2: { "Painting subject": "Trees" Style: Impressionism Technique: Pointillism "Stroke characteristics": { Average length: 25 Average width: 8 Curvature: 0.3 } } }".

[0049] In an exemplary embodiment of the present invention, the brushstroke planning model includes a language encoder, a visual encoder, a feature fusion layer, and a brushstroke prediction module; the step of sequentially inputting the painting scheme description and the painting scheme image into the brushstroke planning model to obtain the painting brushstroke sequence for each painting stage includes: The language encoder is used to encode the descriptions of each painting scheme to obtain the text feature vectors of the descriptions of each painting scheme. The visual encoder is used to encode each of the painting scheme images to obtain the image feature vector of each of the painting scheme images; Based on the feature fusion layer, the text feature vectors and image feature vectors of each painting stage are fused and logically associated to obtain the image-text fusion features of each painting stage; Based on the brushstroke prediction module, the image-text fusion features of each painting stage are processed sequentially to obtain the painting brushstroke sequence of each painting stage.

[0050] In embodiments of the present invention, such as Figure 2 As shown, the stroke planning model includes a language encoder, a visual encoder, a feature fusion layer, and a stroke prediction module, specifically including the following steps: The descriptions of the painting schemes for each painting stage are input into the language encoder to obtain the text feature vectors corresponding to each painting scheme description.

[0051] The language encoder can use language models such as GPT (Generative Pre-trained Transformer) or BERT (Bidirectional Encoder Representations from Transformers), or models such as Chatgpt, BERT, Gemini, Qwen, and Deepseek.

[0052] The images of the painting schemes for each painting stage are input into the visual encoder to obtain the image feature vectors corresponding to each painting scheme image.

[0053] The visual encoder can employ models such as Vision Transformer and EfficientNet.

[0054] The feature fusion module fuses the text feature vectors and image feature vectors corresponding to each painting stage to obtain image-text fusion features. Specifically, the FiLM layer (Feature-wise Linear Modulation) can be used to fuse the text feature vectors and image feature vectors and form a logical relationship to obtain the image-text fusion features corresponding to the painting stage.

[0055] The fused image and text features are sequentially input into the brushstroke prediction module to obtain the brushstroke sequence for each painting stage. The brushstroke prediction module is trained based on the Transformer model.

[0056] In an exemplary embodiment of the present invention, the training process of the stroke planning model includes: Acquire multiple training data sets; wherein the training data sets include sample data and sample labels, the sample data sets include descriptions of painting sample schemes and image-text sample fusion features of painting sample scheme images, and the sample labels include sequences of painting brushstroke samples; The parameters in the language encoder, the visual encoder, and the feature fusion layer are fixed, and the parameters in the stroke prediction module are iteratively trained based on the training data until the training ends, thereby obtaining the stroke planning model.

[0057] In this embodiment of the invention, training data is obtained, and the sequence of painting brushstroke samples in the training data is obtained based on expert painting data, which is the data formed by painting experts during the painting process.

[0058] When training the stroke planning model using training data, the parameters in the language encoder, visual encoder, and feature fusion layer are fixed. Only the weight parameters in the stroke prediction module are updated to implicitly adjust the painting scheme parameters, ensuring the stability of the original visual encoder, language encoder, feature fusion layer, and stroke planning model. The trained stroke planning model will be able to adaptively adjust the stroke planning strategy based on the painting scheme description based on structured text.

[0059] Specifically, during training, after fixing the parameters in the language encoder, visual encoder, and feature fusion layer, the initial weight parameters of the stroke prediction module are loaded. These initial weight parameters can be randomly generated or pre-set pre-trained weights. The optimizer is defined to optimize only the parameters in the stroke prediction module, so that in each subsequent step, gradient calculation and parameter updates will only occur in the stroke prediction module.

[0060] Training data is input into the model, flowing sequentially through the language encoder, visual encoder, feature fusion layer, and stroke prediction module to obtain the model's output. Although the parameters in the language encoder, visual encoder, and feature fusion layer are frozen, they still perform calculations normally, transforming the input training data into high-level features; this computational process is essential.

[0061] The loss value is calculated based on the output and corresponding sample labels. Based on this loss value, the computation graph propagates backward, and the gradient is calculated and flows through the stroke prediction module. Therefore, all parameters of the stroke prediction module receive a gradient value. When the gradient flows to the boundary between the stroke prediction module and the feature fusion layer, the gradient flow stops because the parameters of the feature fusion layer are fixed, and the gradients of the parameters in the language encoder, visual encoder, and feature fusion layer are not further calculated. The optimizer is configured to manage only the parameters of the stroke prediction module. Therefore, it looks up the attributes of the parameters in the stroke prediction module and updates the weights of the stroke prediction module based on their values ​​and hyperparameters such as the learning rate. After multiple rounds of iterative training, the trained stroke planning model is obtained.

[0062] The technical solution provided by this invention significantly reduces the number of gradients to be calculated and parameters to be updated, resulting in shorter training time for the brushstroke planning model in each training cycle. Simultaneously, since there is no need to store gradients for frozen parameters, the GPU memory usage is significantly reduced, allowing for the use of larger batch sizes or models with more parameters, thus improving training stability. In the technical solution provided by this invention, the VLA large model cooperates with multiple specialized models to adaptively adjust the painting strategy in different painting tasks, stages, and areas. This guides the brushstroke planning model to dynamically adjust and generate suitable robot painting brushstroke sequences, effectively solving the aforementioned problems, flexibly responding to diverse painting scenarios, and making the final robot painting effect closer to human level and more natural.

[0063] The following describes the robot painting device based on a large model provided by the present invention. The robot painting device based on a large model described below can be referred to in correspondence with the robot painting method based on a large model described above. It should be noted that the device provided in the following embodiments and the method provided in the above embodiments belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0064] In one exemplary embodiment of the present invention, please refer to Figure 3 , Figure 3 This is an exemplary embodiment of a large-model-based robotic painting device, comprising the following modules.

[0065] The generation module 310 is configured to acquire painting requirements and a target image, and generate painting task information based on the painting requirements and the target image. The painting scheme generation module 320 is used to input the painting task information and the target image into a multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages; wherein, the multi-stage painting scheme generation model is based on a large model; The brushstroke planning module 330 is configured to sequentially input the painting scheme description and the painting scheme image into the brushstroke planning model to obtain the painting brushstroke sequence for each painting stage, so that the robot can paint based on the painting brushstroke sequence in each painting stage; wherein, the brushstroke planning model is trained based on training data.

[0066] In an exemplary embodiment of the present invention, the multi-stage painting scheme generation model includes a description generation module and an image generation module; the painting scheme generation module 320 includes: The first input submodule is configured to input the painting task information into the description generation module to obtain multiple painting stages and the painting scheme description for each painting stage; The second input submodule is configured to input the target image and the descriptions of each painting scheme into the image generation module to obtain the painting scheme images of each painting stage.

[0067] In an exemplary embodiment of the present invention, the painting scheme description includes painting content, painting style, painting technique, and brushstroke characteristics, wherein the brushstroke characteristics are statistical features of the brushstrokes.

[0068] In an exemplary embodiment of the present invention, the stroke planning model includes a language encoder, a visual encoder, a feature fusion layer, and a stroke prediction module; the stroke planning module 330 includes: The first encoding submodule is configured to encode each of the painting scheme descriptions based on the language encoder to obtain the text feature vectors of each of the painting scheme descriptions. The second encoding submodule is configured to encode each of the painting scheme images based on the visual encoder to obtain the image feature vector of each of the painting scheme images; The fusion submodule is configured to fuse the text feature vectors and image feature vectors of each painting stage based on the feature fusion layer and form a logical association to obtain the image-text fusion features of each painting stage; The processing submodule is configured to process the image-text fusion features of each painting stage sequentially based on the brushstroke prediction module to obtain the painting brushstroke sequence of each painting stage.

[0069] In an exemplary embodiment of the present invention, the training process of the stroke planning model includes: The acquisition module is configured to acquire multiple training data sets; wherein the training data sets include sample data and sample labels, the sample data sets include descriptions of painting sample schemes and image-text sample fusion features of painting sample scheme images, and the sample labels include sequences of painting brushstroke samples; The iterative training module is configured to fix the parameters in the language encoder, the visual encoder, and the feature fusion layer, and iteratively train the parameters in the stroke prediction module based on the training data until the training ends, thereby obtaining the stroke planning model.

[0070] In an exemplary embodiment of the present invention, the generation module 310 includes: The extraction submodule is configured to extract semantic information from the target image; The splicing submodule is configured to splice the painting requirements and the semantic information to obtain the painting task information.

[0071] In one exemplary embodiment of the present invention, the extraction submodule includes: The segmentation unit is configured to segment the painting elements in the target image; The determining unit is configured to determine the description information of each of the painting elements; As a unit, it is configured to use the painting elements and the description information as the semantic information.

[0072] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a robot painting method based on a large model. The method includes: acquiring painting requirements and a target image, and generating painting task information based on the painting requirements and the target image. The painting task information and the target image are input into a multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages; wherein, the multi-stage painting scheme generation model is based on a large model; The painting scheme description and the painting scheme image are sequentially input into the brushstroke planning model to obtain the painting brushstroke sequence for each painting stage, so that the robot can paint based on the painting brushstroke sequence in each painting stage; wherein, the brushstroke planning model is trained based on training data.

[0073] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the robot painting method based on a large model provided by the above methods, the method including: acquiring painting requirements and a target image, and generating painting task information based on the painting requirements and the target image; The painting task information and the target image are input into a multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages; wherein, the multi-stage painting scheme generation model is based on a large model; The painting scheme description and the painting scheme image are sequentially input into the brushstroke planning model to obtain the painting brushstroke sequence for each painting stage, so that the robot can paint based on the painting brushstroke sequence in each painting stage; wherein, the brushstroke planning model is trained based on training data.

[0075] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the robot painting method based on a large model provided by the above methods, the method comprising: acquiring painting requirements and a target image, and generating painting task information based on the painting requirements and the target image; The painting task information and the target image are input into a multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages; wherein, the multi-stage painting scheme generation model is based on a large model; The painting scheme description and the painting scheme image are sequentially input into the brushstroke planning model to obtain the painting brushstroke sequence for each painting stage, so that the robot can paint based on the painting brushstroke sequence in each painting stage; wherein, the brushstroke planning model is trained based on training data.

[0076] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot painting method based on a large model, characterized in that, include: Obtain the painting requirements and the target image, and generate painting task information based on the painting requirements and the target image; The painting task information and the target image are input into a multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages; wherein, the multi-stage painting scheme generation model is based on a large model; The painting scheme description and the painting scheme image are sequentially input into the brushstroke planning model to obtain the painting brushstroke sequence for each painting stage, so that the robot can paint based on the painting brushstroke sequence in each painting stage; wherein, the brushstroke planning model is trained based on training data.

2. The robot painting method based on a large model according to claim 1, characterized in that, The multi-stage painting scheme generation model includes a description generation module and an image generation module; the step of inputting the painting task information and the target image into the multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages includes: The painting task information is input into the description generation module to obtain multiple painting stages and a painting scheme description for each painting stage; The target image and the descriptions of each painting scheme are input into the image generation module to obtain the painting scheme images for each painting stage.

3. The robot painting method based on a large model according to claim 2, characterized in that, The painting scheme description includes painting content, painting style, painting techniques, and brushstroke characteristics, wherein the brushstroke characteristics are the statistical features of the brushstrokes.

4. The robot painting method based on a large model according to claim 1, characterized in that, The stroke planning model includes a language encoder, a visual encoder, a feature fusion layer, and a stroke prediction module; The step of sequentially inputting the painting scheme description and the painting scheme image into the brushstroke planning model to obtain the painting brushstroke sequence for each painting stage includes: The language encoder is used to encode the descriptions of each painting scheme to obtain the text feature vectors of the descriptions of each painting scheme. The visual encoder is used to encode each of the painting scheme images to obtain the image feature vector of each of the painting scheme images; Based on the feature fusion layer, the text feature vectors and image feature vectors of each painting stage are fused and logically associated to obtain the image-text fusion features of each painting stage; Based on the brushstroke prediction module, the image-text fusion features of each painting stage are processed sequentially to obtain the painting brushstroke sequence of each painting stage.

5. The robot painting method based on a large model according to claim 4, characterized in that, The training process of the stroke planning model includes: Acquire multiple training data sets; wherein the training data sets include sample data and sample labels, the sample data sets include descriptions of painting sample schemes and image-text sample fusion features of painting sample scheme images, and the sample labels include sequences of painting brushstroke samples; The parameters in the language encoder, the visual encoder, and the feature fusion layer are fixed, and the parameters in the stroke prediction module are iteratively trained based on the training data until the training ends, thereby obtaining the stroke planning model.

6. The robot painting method based on a large model according to any one of claims 1 to 5, characterized in that, The process of generating painting task information based on the painting requirements and the target image includes: Extract the semantic information of the target image; The painting requirements and the semantic information are combined to obtain the painting task information.

7. The robot painting method based on a large model according to claim 6, characterized in that, The extraction of semantic information from the target image includes: Segment the painting elements in the target image; Determine the descriptive information for each of the painting elements; The painting elements and the description information are used as the semantic information.

8. A robotic painting device based on a large model, characterized in that, include: The generation module is configured to acquire painting requirements and target images, and generate painting task information based on the painting requirements and target images; A painting scheme generation module is used to input the painting task information and the target image into a multi-stage painting scheme generation model to obtain painting scheme descriptions and painting scheme images corresponding to multiple painting stages; wherein, the multi-stage painting scheme generation model is based on a large model; The brushstroke planning module is configured to sequentially input the painting scheme description and the painting scheme image into the brushstroke planning model to obtain the painting brushstroke sequence for each painting stage, so that the robot can paint based on the painting brushstroke sequence in each painting stage; wherein, the brushstroke planning model is trained based on training data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the robot drawing method based on a large model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot painting method based on a large model as described in any one of claims 1 to 7.

Images