Image-text publishing method and device and electronic equipment

Abstract

The invention provides an image-text publishing method and device and electronic equipment, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, natural language processing, computer vision, large models and the like. According to the specific implementation scheme, the method comprises the steps of obtaining image-text content and a publishing task; obtaining the current page content of the target software indicated by the release task; according to the image-text content, the publishing task, the current page content and an operation generation model, at least one piece of publishing operation information for the image-text content is determined, and then the publishing operation information is provided for a robot process automation (RPA) system to drive the RPA system to perform image-text content publishing processing; in combination with the operation generation model, the image-text task, the publishing task and the current page content of the target software, at least one piece of publishing operation information for the image-text content can be determined, then image-text content publishing processing is carried out, multiple interactions between the object and the target software are avoided, the image-text content publishing efficiency is improved, and the user experience is improved. And the image-text content publishing cost is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the technical fields of deep learning, natural language processing, computer vision, large models, and in particular to a method, device, and electronic device for publishing pictures and texts. Background Art

[0002] At present, when obtaining graphic content to be published to the target software, the object interacts with the target software multiple times to realize the graphic content publishing process. When there are many graphic contents to be published, the publishing efficiency of the graphic content is low and the publishing cost is high. Summary of the invention

[0003] The present invention provides a method, device and electronic device for publishing pictures and texts.

[0004] According to one aspect of the present disclosure, a method for publishing graphics and text is provided, the method comprising: obtaining graphics and text content and a publishing task; the publishing task instructs to perform graphics and text content publishing processing on a target software; obtaining the current page content of the target software; determining at least one publishing operation information for the graphics and text content according to the graphics and text content, the publishing task, the current page content and an operation generation model; providing the at least one publishing operation information and the graphics and text content to a Robotic Process Automation (RPA) system, and driving the RPA system to perform graphics and text content publishing processing.

[0005] According to another aspect of the present disclosure, a training method for an operation generation model is provided, the method comprising: obtaining training data; the training data comprising: each page content in at least one software, the jump relationship between each page content, and the operation information during the jump; obtaining an initial operation generation model; using each page content in at least one software, the jump relationship between each page content, and the operation information during the jump, to train the operation generation model to obtain a trained operation generation model.

[0006] According to another aspect of the present disclosure, a graphic and text publishing device is provided, the device comprising: a first acquisition module, used to acquire graphic and text content and a publishing task; the publishing task indicates that the graphic and text content publishing process is to be performed on the target software; a second acquisition module, used to acquire the current page content of the target software; a first determination module, used to determine at least one publishing operation information for the graphic and text content according to the graphic and text content, the publishing task, the current page content and an operation generation model; a first providing module, used to provide the at least one publishing operation information and the graphic and text content to a robot process automation (RPA) system, and drive the RPA system to perform graphic and text content publishing process.

[0007] According to another aspect of the present disclosure, a training device for an operation generation model is provided, the device comprising: a first acquisition module, used to acquire training data; the training data comprising: each page content in at least one software, the jump relationship between each page content and the operation information during the jump; a second acquisition module, used to acquire an initial operation generation model; a training processing module, used to use each page content in at least one software, the jump relationship between each page content and the operation information during the jump to perform training processing on the operation generation model to obtain a trained operation generation model.

[0008] According to another aspect of the present disclosure, a graphic and text publishing system is provided, including: intelligent agents corresponding to various fields and a control module; the control module determines the graphic and text content based on the prompt text determined by the publishing requirements and the intelligent agent corresponding to the target field to which the prompt text belongs; according to the graphic and text content, the publishing task indicated by the publishing requirements, the current page content of the target software indicated by the publishing task, and the operation generation model, at least one publishing operation information for the graphic and text content is determined to drive the robot process automation (RPA) system to perform graphic and text content publishing processing.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method for publishing pictures and texts of the present disclosure; or, execute the above-mentioned method for training the operation generation model of the present disclosure.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned method for publishing pictures and texts of the present disclosure; or, to execute the above-mentioned method for training an operation generation model of the present disclosure.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for publishing pictures and texts of the present disclosure; or, implements the steps of the above-mentioned method for training an operation generation model of the present disclosure.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0019] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0020] Figure 7 It is a block diagram of an electronic device used to implement the image and text publishing method or the training method of the operation generation model of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] At present, when obtaining graphic content to be published to the target software, the object interacts with the target software multiple times to realize the graphic content publishing process. When there are many graphic contents to be published, the publishing efficiency of the graphic content is low and the publishing cost is high.

[0023] In view of the above problems, the present disclosure provides a method, device and electronic device for publishing pictures and texts.

[0024] Figure 1 It is a schematic diagram according to the first embodiment of the present disclosure. It should be noted that the image and text publishing method of the embodiment of the present disclosure can be applied to an image and text publishing device, and the device can be configured in an electronic device so that the electronic device can perform the image and text publishing function.

[0025] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.

[0026] The image and text publishing device may also be software in an electronic device, such as image and text publishing software, etc. In the following embodiments, the execution subject is an electronic device as an example for description.

[0027] like Figure 1 As shown, the image and text publishing method may include the following steps:

[0028] Step 101, obtaining graphic content and publishing tasks; the publishing task indicates that the graphic content publishing process is performed on the target software.

[0029] In the disclosed embodiment, the graphic content may include at least one of the following: image, image title, image copy, text. In one example, the graphic content may include only images; or, include images and at least one of image titles and image copy. In another example, the graphic content may include only text. The image copy is a descriptive text of the content in the image. The image title is the text obtained by extracting the title of the image copy. The number of target software may be one or more.

[0030] Step 102, obtaining the current page content of the target software.

[0031] In the disclosed embodiment, the target software may be in a running state; or the target software may be in a non-running state. When the target software is in a running state, the electronic device may obtain the page content currently displayed in the target software; and determine the currently displayed page content as the current page content. The page content currently displayed in the target software may be any page content in the target software.

[0032] Wherein, when the target software is not in the running state, the electronic device can run the target software. After the target software is run, the first page will be displayed. At this time, the current page content can be the page content of the first page of the target software.

[0033] In the embodiment of the present disclosure, the current page content of the target software may be expressed in at least one of the following forms: a page, a page screenshot, and page element information. The page element information may include the location information, element type, element name, etc. of each element, which may be set according to actual needs and will not be described in detail here.

[0034] It should be noted that, when the current page content is in the form of a page or a page screenshot, element recognition processing can be performed on the page or the page screenshot to obtain page element information.

[0035] Step 103: Determine at least one publishing operation information for the graphic content according to the graphic content, the publishing task, the current page content, and the operation generation model.

[0036] In the disclosed embodiment, the process of the electronic device executing step 103 may be, for example, inputting the graphic content, publishing task, and current page content into the operation generation model, and obtaining at least one publishing operation information output by the operation generation model. The publishing operation information may include: operation position and operation type at the operation position.

[0037] Among them, the settings of the operation position and operation type in the published operation information enable the Robotic Process Automation (RPA) system to automatically process the elements at the operation position according to the operation type, thereby improving the efficiency of image and text publishing and reducing the cost of image and text publishing.

[0038] In the embodiment of the present disclosure, the operation generation model can be trained in combination with page content data in at least one software; the at least one software includes the target software; the page content data includes: each page content, the jump relationship between each page content and the operation information when jumping; the jump relationship between each page content and the operation information when jumping can be determined based on the historical operation process and process feedback opinions for the software.

[0039] The jump relationship between each page content, for example, you can jump from page content A to page content B; and from page content B to page content C. The jump operation information between each page content, for example, the element that needs to be operated when jumping from page content A to page content B, and the type of operation for the element; for another example, the element that needs to be operated when jumping from page content B to page content C, and the type of operation for the element.

[0040] Among them, the historical operation process may include multiple historical operation types, as well as the elements targeted by each historical operation type and the page content where the elements targeted by the historical operation type are located; the process feedback opinions can be used to adjust the historical operation process. For example, suppose the historical operation process includes: after performing historical operation A on page content 1, page content 2 is obtained; after performing historical operation B on page content 2, page content 3 is obtained; after performing historical operation C on page content 3, page content 4 is obtained. After adjusting the historical operation process in combination with the process feedback opinions, the obtained operation process may be, for example, after performing historical operation D on page content 1, page content 3 is obtained; after performing historical operation C on page content 3, page content 4 is obtained. Process feedback opinions can indicate to adjust the operation type for page content 1 to reduce the number of operations.

[0041] Among them, the operation generation model obtained by training with the page content data in at least one software can learn the page content in each software, the jump relationship between each page content and the operation information during the jump, thereby improving the accuracy of the determined publishing operation information, thereby further improving the efficiency of graphic content publishing.

[0042] Step 104: provide at least one publishing operation information and graphic content to the Robotic Process Automation (RPA) system, and drive the RPA system to perform graphic content publishing processing.

[0043] In the embodiments of the present disclosure, the page element information may include the location information, element type, element name, etc. of each element. When the current page content is expressed in the form of page element information, RPA can determine the target element to be operated based on the operation position and the location information of each element; perform operation processing on the target element based on the operation type to obtain the page content after the operation.

[0044] In an embodiment of the present disclosure, when there are changes in the page content of the target software, in order to ensure that the publishing operation information output by the operation generation model can be applicable to the target software with changes in the page content, and further improve the success rate of graphic content publishing, the electronic device can also perform the following process: when there are changes in the page content in the target software, obtain the first page content that has changed, the jump relationship between the first page content and the contents of each page in the target software, and the operation information when jumping; provide the first page content, the jump relationship between the first page content and the contents of each page in the target software, and the operation information when jumping to the operation generation model for fine-tuning training processing.

[0045] The graphic and text publishing method of the embodiment of the present invention obtains graphic and text content and a publishing task; the publishing task instructs to perform graphic and text content publishing processing on the target software; obtains the current page content of the target software; determines at least one publishing operation information for the graphic and text content according to the graphic and text content, the publishing task, the current page content and the operation generation model; provides the at least one publishing operation information and the graphic and text content to the Robotic Process Automation (RPA) system, and drives the RPA system to perform graphic and text content publishing processing; wherein, in combination with the operation generation model, the graphic and text task, the publishing task and the current page content of the target software, at least one publishing operation information for the graphic and text content can be determined, and then the graphic and text content publishing processing is performed, avoiding multiple interactions between the object and the target software, so that when there are many graphic and text contents to be published, the efficiency of graphic and text content publishing can be improved and the cost of graphic and text content publishing can be reduced.

[0046] In order to further improve the efficiency of publishing graphic content, the publishing tasks and the generation of graphic content can be determined in combination with the publishing requirements to improve the efficiency of obtaining graphic content. Figure 2 As shown, Figure 2 is a schematic diagram according to a second embodiment of the present disclosure, Figure 2 The illustrated embodiment may include the following steps:

[0047] Step 201, obtaining a publishing requirement; the publishing requirement indicates a publishing task and generates auxiliary information; the publishing task indicates performing graphic content publishing processing on the target software.

[0048] In the disclosed embodiment, the generation of auxiliary information may indicate any of the following: prompt text for the graphic content generation process; prompt text acquisition strategy. The prompt text for the graphic content generation process is, for example, "please generate a performance parameter comparison chart of car A and car B, and add a title and text".

[0049] Among them, the setting of generating auxiliary information in the publishing demand enables the electronic device to generate graphic content in combination with the publishing demand, avoiding object provision, thereby further improving the efficiency of obtaining graphic content.

[0050] In an embodiment of the present disclosure, a prompt text acquisition strategy may include at least one of the following: reading content that meets the first requirement from a target page as prompt text; capturing corresponding content whose attention level is greater than or equal to an attention level threshold from a web page as prompt text.

[0051] The image and text content publishing scenarios include, for example, the scenario of publishing an image, the scenario of replying to comments on an image, etc. Different acquisition strategies may be used for different image and text content publishing scenarios.

[0052] Among them, in the case where the graphic content publishing scenario is the scenario of publishing an image, the acquisition strategy of the prompt text can be, for example, to capture the corresponding content with a degree of attention greater than or equal to the degree of attention threshold from the web page as the prompt text. Correspondingly, the electronic device can perform content capture processing on any web page to capture the corresponding content with a degree of attention greater than or equal to the degree of attention threshold and use it as the prompt text.

[0053] In the case where the image and text content publishing scenario is a scenario of replying to comments on an image, the acquisition strategy of the prompt text may be, for example, to read content that meets the first requirement from the target page as the prompt text. In this case, the first requirement may be to obtain comments on the image. The obtained text that meets the first requirement may be a comment on the image.

[0054] Among them, the setting of multiple acquisition strategies allows the electronic device to flexibly select an acquisition strategy according to the graphic content publishing scenario; and then combine the selected acquisition strategy to obtain prompt text to generate graphic content, thereby improving the flexibility of graphic content acquisition and expanding the usage scenarios of graphic content publishing.

[0055] Step 202: Generate graphic content based on the generated auxiliary information.

[0056] In an embodiment of the present disclosure, when auxiliary information is generated to indicate an acquisition strategy for prompt text used for graphic content generation processing, the process of the electronic device executing step 202 may, for example, be to obtain the prompt text based on the acquisition strategy, and determine the target field to which the prompt text belongs; input the prompt text into an intelligent agent corresponding to the target field, and obtain relevant knowledge output by the intelligent agent; input the relevant knowledge and the prompt text into a graphic content generation model, and obtain the graphic content output by the graphic content generation model.

[0057] The target domain is, for example, the automotive domain, the communications domain, etc. Different domains may correspond to different intelligent agents. The intelligent agents corresponding to different domains may be trained using the knowledge of the corresponding domains; or, the intelligent agents corresponding to different domains may be provided with knowledge bases of the corresponding domains.

[0058] Among them, for the intelligent agent corresponding to the target domain, assuming that the intelligent agent is set with a knowledge text library of the target domain, the process of the intelligent agent acquiring relevant knowledge can be, for example, determining the representation vector of each knowledge in the knowledge base; determining the representation vector of the prompt text; for each knowledge in the knowledge base, combining the representation vector of the knowledge with the representation vector of the prompt text, determining the matching degree between the knowledge and the prompt text; when the matching degree is greater than or equal to the matching degree threshold, determining the knowledge as relevant knowledge; when the matching degree is less than the matching degree threshold, ignoring the knowledge.

[0059] It should be noted that the relevant knowledge may be expressed in at least one of the following forms: text, image, and audio. For the audio, speech recognition processing may be performed to obtain text.

[0060] Among them, the prompt text is obtained based on the acquisition strategy, and then the relevant knowledge is determined in combination with the intelligent agent corresponding to the target field to which the prompt text belongs, which is used to generate graphic content. This can ensure the accuracy of the generated graphic content and avoid generating unreasonable graphic content.

[0061] In the embodiment of the present disclosure, when auxiliary information is generated to indicate prompt text for graphic content generation processing, the process of the electronic device executing step 202 may, for example, be to determine the target field to which the prompt text belongs; input the prompt text into an intelligent agent corresponding to the target field to obtain relevant knowledge output by the intelligent agent; input the relevant knowledge and the prompt text into a graphic content generation model to obtain the graphic content output by the graphic content generation model.

[0062] Step 203, obtaining the current page content of the target software.

[0063] Step 204: Determine at least one publishing operation information for the graphic content according to the graphic content, the publishing task, the current page content, and the operation generation model.

[0064] Step 205: provide at least one publishing operation information and graphic content to the Robotic Process Automation (RPA) system, and drive the RPA system to perform graphic content publishing processing.

[0065] It should be noted that the details of steps 203 to 205 can be found in Figure 1 Steps 102 to 104 in the illustrated embodiment will not be described in detail herein.

[0066] The graphic and text publishing method of the embodiment of the present invention obtains publishing requirements; the publishing requirements indicate publishing tasks and generate auxiliary information; the publishing tasks indicate graphic and text content publishing processing on the target software; the graphic and text content is generated according to the generated auxiliary information; the current page content of the target software is obtained; according to the graphic and text content, the publishing tasks, the current page content and the operation generation model, at least one publishing operation information for the graphic and text content is determined; the at least one publishing operation information and the graphic and text content are provided to the Robotic Process Automation (RPA) system to drive the RPA system to perform graphic and text content publishing processing; wherein, determining the publishing tasks and generating the graphic and text content in combination with the publishing requirements can improve the efficiency of obtaining the graphic and text content, thereby further improving the efficiency of publishing the graphic and text content.

[0067] Among them, in order to further improve the accuracy of the generated publishing operation information, the electronic device can determine a publishing operation information in combination with the graphic content, publishing task, current page content and operation generation model, and determine the new current page content in combination with the RPA system, and then re-determine a publishing operation information to avoid the situation where the publishing operation information is incorrect due to inaccurate current page content. Figure 3 As shown, Figure 3 is a schematic diagram according to a third embodiment of the present disclosure, Figure 3 The illustrated embodiment may include the following steps:

[0068] Step 301, obtaining graphic content and publishing tasks; the publishing task indicates that graphic content publishing processing is performed on the target software.

[0069] Step 302, obtaining the current page content of the target software.

[0070] Step 303: input the graphic content, publishing task and current page content into the operation generation model, and obtain a publishing operation information for the current page content output by the operation generation model.

[0071] In the embodiment of the present disclosure, the operation generation model can be trained in combination with page content data in at least one software; the at least one software includes the target software; the page content data includes: each page content, the jump relationship between each page content and the operation information when jumping; the jump relationship between each page content and the operation information when jumping can be determined based on the historical operation process and process feedback opinions for the software.

[0072] Step 304: determine the publishing operation information as publishing operation information for graphic and text content.

[0073] Step 305: Provide a publishing operation information and graphic content to the Robotic Process Automation (RPA) system to drive the RPA system to execute the publishing operation information.

[0074] In the embodiment of the present disclosure, the published operation information may include: the operation position and the operation type at the operation position. Correspondingly, the RPA system may perform operation processing on the element at the operation position in the current page content of the target software according to the operation type.

[0075] Step 306 , determining the post-operation page content obtained after the RPA system operates the current page content based on the publishing operation information.

[0076] In the embodiment of the present disclosure, the page content obtained after the RPA system operates the element at the operation position in the current page content of the target software according to the operation type is the post-operation page content. The electronic device can interact with the RPA system to obtain the post-operation page content.

[0077] Step 307: if the post-operation page content is not the post-successful publishing page content, re-execute the generation process of the publishing operation information until it is determined that the post-operation page content is the post-successful publishing page content.

[0078] In the disclosed embodiment, the page content after successful publishing refers to the page content displayed after the graphic content is successfully published. In one example, the page content after successful publishing may include a specified element. The electronic device can determine whether the page content after the operation is the page content after successful publishing by judging whether the page content after the operation includes the specified element.

[0079] The designated element may be present in the page content after the successful publishing, and not present in other page content. The designated element may be, for example, "publish successfully", "publish completed", etc., which may be set according to actual needs, and will not be described in detail here.

[0080] In another example, the electronic device can use the page content after the operation as the new current page content. Afterwards, the electronic device can input the graphic content, the publishing task, and the current page content into the operation generation model to determine whether the publishing operation information output by the operation generation model is obtained; if the operation generation model outputs the publishing operation information, it is determined that the current page content is not the page content after the successful publishing; if the operation generation model does not output the publishing operation information, it means that the publishing is completed, and the current page content is determined to be the page content after the successful publishing.

[0081] In the disclosed embodiment, in order to further improve the accuracy of the determined publishing operation information, in the process of re-executing the generation process of publishing operation information, the electronic device can input the graphic content, publishing task, current page content, historical page content, and historical publishing operation information for historical page content into the operation generation model to obtain a publishing operation information output by the operation generation model.

[0082] The historical page content can be the page content that has been operated during the current graphic content publishing process; the historical publishing operation information can be the publishing operation information for the historical page content. The reference to the historical page content and the historical publishing operation information can avoid jumping to the historical page content after the publishing operation information output by the operation generation model is executed, thereby avoiding operation loops, avoiding an increase in the number of operations, and further improving the efficiency of graphic content publishing.

[0083] In the embodiment of the present disclosure, as an alternative to steps 303 to 307, the electronic device may also perform the following process: inputting graphic content, publishing tasks and current page content into the operation generation model, obtaining a publishing operation information for the current page content and post-operation page content output by the operation generation model; updating the current page content according to the post-operation page content; repeatedly executing the generation process of the publishing operation information and the post-operation page content until the post-operation page content is the page content after successful publishing; and determining each publishing operation information as publishing operation information for the graphic content.

[0084] Among them, during the operation generation model training process, the content of each page in the target software can be learned, so that the post-operation page content obtained after operating a page content can be understood.

[0085] Among them, the operation generation model outputs the publishing operation information and the page content after the operation, and then the next publishing operation information can be determined, thereby avoiding multiple interactions with the RPA system. Multiple publishing operation information can be provided to the RPA system through one interaction, thereby improving the operation execution efficiency of the RPA system, and further improving the efficiency of graphic content publishing.

[0086] It should be noted that the details of step 301 to step 302 can be found in Figure 1 Steps 101 to 102 in the illustrated embodiment will not be described in detail herein.

[0087] The graphic and text publishing method of the embodiment of the present disclosure obtains graphic and text content and a publishing task; the publishing task indicates that the graphic and text content publishing process is performed on the target software; the current page content of the target software is obtained; the graphic and text content, the publishing task and the current page content are input into the operation generation model, and a publishing operation information for the current page content output by the operation generation model is obtained; the publishing operation information is determined as the publishing operation information for the graphic and text content; the publishing operation information and the graphic and text content are provided to the robot process automation (RPA) system to drive the RPA system to execute the publishing operation information; the post-operation page content obtained after the RPA system operates on the current page content based on the publishing operation information; if the post-operation page content is not the post-successful publishing page content, the publishing operation information generation process is re-executed until it is determined that the post-operation page content obtained is the post-successful publishing page content; wherein, the new current page content is determined in combination with the RPA system and the publishing operation information, and then a publishing operation information is re-determined, which can avoid the situation where the publishing operation information is wrong due to inaccurate current page content, thereby further improving the accuracy of the determined publishing operation information and further improving the efficiency of graphic and text content publishing.

[0088] Figure 4 It is a schematic diagram according to the fourth embodiment of the present disclosure. It should be noted that the training method of the operation generation model of the embodiment of the present disclosure can be applied to the training device of the operation generation model. The device can be configured in an electronic device so that the electronic device can perform the training function of the operation generation model.

[0089] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.

[0090] The training device for the operation generation model may also be software in an electronic device, such as training software for the operation generation model, etc. In the following embodiments, the execution subject is an electronic device as an example for description.

[0091] like Figure 4 As shown, the training method of the operation generation model may include the following steps:

[0092] Step 401, obtaining training data; the training data includes: each page content in at least one software, the jump relationship between each page content and the operation information during the jump.

[0093] In the embodiments of the present disclosure, the jump relationship between the various page contents in at least one software and the operation information during the jump can be determined based on the historical operation process and process feedback opinions for the software; or can be determined based on the usage record of the software. The usage record can include the operation on the various page contents in the software and the page after the operation, etc.

[0094] Step 402: Obtain an initial operation generation model.

[0095] In the embodiment of the present disclosure, the operation generation model may be, for example, a large language model.

[0096] Step 403, using the page contents in at least one software, the jump relationship between the page contents, and the operation information during the jump, the operation generation model is trained to obtain a trained operation generation model.

[0097] In an embodiment of the present disclosure, in one example, the process of the electronic device executing step 403 may be, for example, determining, for each jump operation information in each page content, a functional description of the element operated by the jump operation information; inputting the functional description and the page content into an operation generation model to obtain predicted operation information output by the operation generation model; determining a value of the loss function based on the predicted operation information, the jump operation information, and the loss function of the operation generation model; and performing parameter adjustment processing on the operation generation model based on the value of the loss function to obtain a trained operation generation model.

[0098] In another example, the process of the electronic device executing step 403 may be, for example, determining, for each jump operation information in each page content, a functional description of the element operated by the jump operation information; inputting the functional description and the page content into the operation generation model, obtaining the predicted operation information output by the operation generation model and the predicted page content after the operation; determining the value of the loss function of the operation generation model based on the predicted operation information, the predicted page content after the operation, the post-jump operation information and the post-jump page content; and performing parameter adjustment processing on the operation generation model based on the value of the loss function to obtain a trained operation generation model.

[0099] Among them, after step 403, in order to improve the prediction accuracy of the operation generation model for each publishing operation in the publishing task, the electronic device can obtain the historical operation process under each historical publishing task; determine the content of each historical page and the corresponding jump operation information and the page content after the jump according to the historical operation process; and train the operation generation model in combination with the historical publishing tasks, the content of each historical page and the corresponding jump operation information and the page content after the jump.

[0100] The training method of the operation generation model of the embodiment of the present disclosure is through obtaining training data; the training data includes: each page content in at least one software, the jump relationship between each page content and the operation information when jumping; obtaining an initial operation generation model; using each page content in at least one software, the jump relationship between each page content and the operation information when jumping, the operation generation model is trained to obtain a trained operation generation model; wherein, the operation generation model obtained by training in combination with each page content in at least one software, the jump relationship between each page content and the operation information when jumping can perform at least one generation process of publishing operation information, thereby improving the accuracy of determining the published operation information.

[0101] In order to implement the above embodiment, the present disclosure also provides a graphic and text publishing device. Figure 5 As shown, Figure 5The fifth embodiment of the present disclosure is shown in FIG. 5. The image and text publishing device 50 may include: a first acquisition module 501 , a second acquisition module 502 , a first determination module 503 and a first providing module 504 .

[0102] Among them, the first acquisition module 501 is used to obtain graphic content and publishing tasks; the publishing task indicates the graphic content publishing process on the target software; the second acquisition module 502 is used to obtain the current page content of the target software; the first determination module 503 is used to determine at least one publishing operation information for the graphic content according to the graphic content, the publishing task, the current page content and the operation generation model; the first providing module 504 is used to provide the at least one publishing operation information and the graphic content to the robot process automation RPA system, and drive the RPA system to perform graphic content publishing process.

[0103] As a possible implementation method of the embodiment of the present disclosure, the first acquisition module 501 includes an acquisition unit and a generation unit; the acquisition unit is used to acquire the publishing requirements; the publishing requirements indicate the publishing task and generate auxiliary information; the generation unit is used to generate the graphic content according to the generated auxiliary information.

[0104] As a possible implementation manner of the embodiment of the present disclosure, the generation auxiliary information indicates any one of the following: prompt text used for graphic content generation processing; and a strategy for obtaining the prompt text.

[0105] As a possible implementation method of the embodiment of the present disclosure, the generation of auxiliary information indicates an acquisition strategy for prompt text used for graphic content generation processing; the generation unit is specifically used to acquire the prompt text based on the acquisition strategy, and determine the target field to which the prompt text belongs; input the prompt text into an intelligent agent corresponding to the target field, and acquire relevant knowledge output by the intelligent agent; input the relevant knowledge and the prompt text into a graphic content generation model, and acquire the graphic content output by the graphic content generation model.

[0106] As a possible implementation method of the embodiment of the present disclosure, the acquisition strategy includes one of the following: reading content that meets the first requirement from the target page as prompt text; grabbing corresponding content whose attention degree is greater than or equal to the attention degree threshold from the web page as prompt text.

[0107] As a possible implementation method of the embodiment of the present disclosure, the first determination module 503 is specifically used to input the graphic content, the publishing task and the current page content into the operation generation model, obtain a publishing operation information and a post-operation page content for the current page content output by the operation generation model; update the current page content according to the post-operation page content; repeatedly execute the generation process of the publishing operation information and the post-operation page content until the post-operation page content is the page content after successful publishing; and determine each of the publishing operation information as the publishing operation information for the graphic content.

[0108] As a possible implementation method of the embodiment of the present disclosure, the first determination module 503 is specifically used to input the graphic content, the publishing task and the current page content into the operation generation model, obtain a publishing operation information for the current page content output by the operation generation model; and determine the publishing operation information as the publishing operation information for the graphic content.

[0109] As a possible implementation method of the embodiment of the present disclosure, the device also includes: a second determination module and a re-execution module; the second determination module is used to determine the post-operation page content obtained after the RPA system operates the current page content based on the publishing operation information; the re-execution module is used to re-execute the generation process of the publishing operation information when the post-operation page content is not the page content after successful publishing, until it is determined that the post-operation page content obtained is the page content after successful publishing.

[0110] As a possible implementation method of the embodiment of the present disclosure, the operation generation model is trained in combination with page content data in at least one software; the at least one software includes the target software; the page content data includes: each page content, the jump relationship between each page content and the operation information when jumping; the jump relationship between each page content and the operation information when jumping are determined based on the historical operation process and process feedback opinions for the software.

[0111] As a possible implementation method of the embodiment of the present disclosure, the device also includes: a third acquisition module and a second providing module; the third acquisition module is used to obtain the first page content that has changed, the jump relationship between the first page content and each page content in the target software, and the operation information when jumping when the page content in the target software has changed; the second providing module is used to provide the first page content, the jump relationship between the first page content and each page content in the target software, and the operation information when jumping to the operation generation model for fine-tuning the training process.

[0112] As a possible implementation manner of the embodiment of the present disclosure, the publishing operation information includes: the page content to be operated, the operation position in the page content, and the operation type at the operation position.

[0113] The graphic and text publishing device of the embodiment of the present disclosure obtains graphic and text content and publishing tasks; the publishing task indicates that the graphic and text content publishing process is performed on the target software; the current page content of the target software is obtained; according to the graphic and text content, the publishing task, the current page content and the operation generation model, at least one publishing operation information for the graphic and text content is determined; the at least one publishing operation information and the graphic and text content are provided to the Robotic Process Automation (RPA) system to drive the RPA system to perform graphic and text content publishing process; wherein, in combination with the operation generation model, the graphic and text task, the publishing task and the current page content of the target software, at least one publishing operation information for the graphic and text content can be determined, and then the graphic and text content publishing process is performed, so as to avoid multiple interactions between the object and the target software, thereby being able to improve the efficiency of graphic and text content publishing and reduce the cost of graphic and text content publishing when there are a lot of graphic and text content to be published.

[0114] In order to implement the above embodiment, the present disclosure also provides a training device for operating a generation model. Figure 6 As shown, Figure 6 6 is a schematic diagram according to the sixth embodiment of the present disclosure. The training device 60 for operating the generation model may include: a first acquisition module 601 , a second acquisition module 602 and a training processing module 603 .

[0115] Among them, the first acquisition module 601 is used to obtain training data; the training data includes: the content of each page in at least one software, the jump relationship between each page content and the operation information during the jump; the second acquisition module 602 is used to obtain the initial operation generation model; the training processing module 603 is used to use the content of each page in at least one software, the jump relationship between each page content and the operation information during the jump to train the operation generation model to obtain a trained operation generation model.

[0116] The training method of the operation generation model of the embodiment of the present disclosure is through obtaining training data; the training data includes: each page content in at least one software, the jump relationship between each page content and the operation information when jumping; obtaining an initial operation generation model; using each page content in at least one software, the jump relationship between each page content and the operation information when jumping, the operation generation model is trained to obtain a trained operation generation model; wherein, the operation generation model obtained by training in combination with each page content in at least one software, the jump relationship between each page content and the operation information when jumping can perform at least one generation process of publishing operation information, thereby improving the accuracy of determining the published operation information.

[0117] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, comply with the relevant laws and regulations, and do not violate public order and good morals.

[0118] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0119] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0120] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0121] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0122] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a method for publishing images or a training method for operating a generation model. For example, in some embodiments, the method for publishing images or the training method for operating a generation model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method for publishing images or the training method for operating a generation model described above may be executed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the image and text publishing method or the training method of the operation generation model in any other appropriate manner (for example, by means of firmware).

[0123] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0125] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0127] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0128] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0129] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0130] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for publishing pictures and texts, the method comprising: Get graphic content and publish tasks; The publishing task instructs to perform graphic content publishing processing on the target software; Obtaining the current page content of the target software; Determine at least one publishing operation information for the graphic content according to the graphic content, the publishing task, the current page content, and the operation generation model; The at least one publishing operation information and the graphic content are provided to a Robotic Process Automation (RPA) system, and the RPA system is driven to perform graphic content publishing processing.

2. The method according to claim 1, wherein: The tasks of obtaining graphic content and publishing include: Obtaining a publishing requirement; the publishing requirement indicates a publishing task and generates auxiliary information; The graphic content is generated according to the auxiliary information.

3. The method according to claim 2, wherein: The generating auxiliary information indicates any one of the following: Prompt text for graphic content generation and processing; The acquisition strategy of the prompt text.

4. The method according to claim 2 or 3, wherein: The generation of auxiliary information indicates an acquisition strategy for prompt text used in the graphic content generation process; The step of generating the graphic content according to the auxiliary information includes: Acquire a prompt text based on the acquisition strategy, and determine a target field to which the prompt text belongs; Input the prompt text into the intelligent agent corresponding to the target domain to obtain relevant knowledge output by the intelligent agent; The relevant knowledge and the prompt text are input into a picture-text generation model to obtain the picture-text content output by the picture-text generation model.

5. The method according to claim 4, wherein: The acquisition strategy includes one of the following: Read the content that meets the first requirement from the target page as the prompt text; The corresponding content whose attention degree is greater than or equal to the attention degree threshold is captured from the web page as the prompt text.

6. The method according to claim 1, wherein: The determining, according to the graphic content, the publishing task, the current page content, and the operation generation model, at least one publishing operation information for the graphic content includes: Input the graphic content, the publishing task and the current page content into the operation generation model, and obtain a publishing operation information for the current page content and the page content after the operation output by the operation generation model; Updating the current page content according to the page content after the operation; Repeat the process of publishing the operation information and generating the post-operation page content until the post-operation page content is the page content after successful publishing; Each of the publishing operation information is determined as publishing operation information for the graphic content.

7. The method according to claim 1, wherein: The determining, according to the graphic content, the publishing task, the current page content, and the operation generation model, at least one publishing operation information for the graphic content includes: Input the graphic content, the publishing task and the current page content into the operation generation model, and obtain a publishing operation information for the current page content output by the operation generation model; The publishing operation information is determined as publishing operation information for the graphic content.

8. The method according to claim 7, wherein: The method further comprises: Determine a post-operation page content obtained after the RPA system operates the current page content based on the publishing operation information; In the case that the post-operation page content is not the post-successful publishing page content, the generation process of the publishing operation information is re-executed until it is determined that the post-operation page content obtained is the post-successful publishing page content.

9. The method according to claim 1, wherein: The operation generation model is trained by combining page content data in at least one software; The at least one software includes the target software; The page content data includes: each page content, the jump relationship between each page content and the operation information during the jump; The jump relationship between the contents of each page and the operation information during the jump are determined based on the historical operation process and process feedback opinions for the software.

10. The method according to claim 1 or 9, wherein: The method further comprises: When the page content in the target software is changed, obtaining the changed first page content, the jump relationship between the first page content and each page content in the target software, and the operation information during the jump; The first page content, the jump relationship between the first page content and each page content in the target software, and the operation information during the jump are provided to the operation generation model for fine-tuning training processing.

11. The method according to claim 1, wherein: The publishing operation information includes: the page content to be operated, the operation position in the page content, and the operation type at the operation position.

12. A training method for an operation generation model, the method comprising: Get training data; The training data includes: each page content in at least one software, the jump relationship between each page content and the operation information during the jump; Obtaining an initial operation generation model; The operation generation model is trained using the page contents, the jump relationship between the page contents, and the operation information during the jump in at least one software to obtain a trained operation generation model.

13. A device for publishing pictures and texts, comprising: The first acquisition module is used to acquire graphic content and a publishing task; the publishing task indicates that the graphic content publishing process is performed on the target software; A second acquisition module, used to acquire the current page content of the target software; A first determination module, configured to determine at least one publishing operation information for the graphic content according to the graphic content, the publishing task, the current page content, and an operation generation model; The first providing module is used to provide the at least one publishing operation information and the graphic content to the Robotic Process Automation (RPA) system, and drive the RPA system to perform graphic content publishing processing.

14. The device according to claim 13, wherein: The first acquisition module includes an acquisition unit and a generation unit; The acquisition unit is used to acquire a publishing requirement; the publishing requirement indicates a publishing task and generates auxiliary information; The generating unit is used to generate the graphic content according to the generating auxiliary information.

15. The device according to claim 14, wherein: The generating auxiliary information indicates any one of the following: Prompt text for graphic content generation and processing; The acquisition strategy of the prompt text.

16. The device according to claim 13 or 14, wherein: The auxiliary information generation indicates an acquisition strategy for the prompt text used for the graphic content generation process; the generation unit is specifically used to: Acquire a prompt text based on the acquisition strategy, and determine a target field to which the prompt text belongs; Input the prompt text into the intelligent agent corresponding to the target domain to obtain relevant knowledge output by the intelligent agent; The relevant knowledge and the prompt text are input into a picture-text generation model to obtain the picture-text content output by the picture-text generation model.

17. The device according to claim 16, wherein: The acquisition strategy includes one of the following: Read the content that meets the first requirement from the target page as the prompt text; The corresponding content whose attention degree is greater than or equal to the attention degree threshold is captured from the web page as the prompt text.

18. The device according to claim 13, wherein: The first determination module is specifically configured to: Input the graphic content, the publishing task and the current page content into the operation generation model, and obtain a publishing operation information for the current page content and the page content after the operation output by the operation generation model; Updating the current page content according to the page content after the operation; Repeat the process of publishing the operation information and generating the post-operation page content until the post-operation page content is the page content after successful publishing; Each of the publishing operation information is determined as publishing operation information for the graphic content.

19. The device according to claim 13, wherein: The first determination module is specifically configured to: Input the graphic content, the publishing task and the current page content into the operation generation model, and obtain a publishing operation information for the current page content output by the operation generation model; The publishing operation information is determined as publishing operation information for the graphic content.

20. The device according to claim 19, wherein The device further comprises: a second determination module and a re-execution module; The second determination module is used to determine the post-operation page content obtained after the RPA system operates the current page content based on the publishing operation information; The re-execution module is used to re-execute the generation process of the publishing operation information when the post-operation page content is not the post-successful publishing page content, until it is determined that the post-operation page content obtained is the post-successful publishing page content.

21. The device according to claim 13, wherein: The operation generation model is trained by combining page content data in at least one software; The at least one software includes the target software; The page content data includes: each page content, the jump relationship between each page content and the operation information during the jump; The jump relationship between the contents of each page and the operation information during the jump are determined based on the historical operation process and process feedback opinions for the software.

22. The device according to claim 13 or 21, wherein: The device further comprises: a third acquisition module and a second providing module; The third acquisition module is used to acquire the changed first page content, the jump relationship between the first page content and each page content in the target software, and the operation information during the jump when the page content in the target software is changed; The second providing module is used to provide the first page content, the jump relationship between the first page content and each page content in the target software, and the operation information during the jump to the operation generation model for fine-tuning training processing.

23. The device according to claim 13, wherein: The publishing operation information includes: the page content to be operated, the operation position in the page content, and the operation type at the operation position.

24. A training device for operating a generation model, the device comprising: A first acquisition module, used to acquire training data; The training data includes: each page content in at least one software, the jump relationship between each page content and the operation information during the jump; A second acquisition module is used to acquire an initial operation generation model; The training processing module is used to use the page contents, the jump relationship between the page contents and the operation information during the jump in at least one software to train the operation generation model to obtain the trained operation generation model.

25. A graphic and text publishing system, comprising: Intelligent agents and control modules corresponding to each field; The control module determines the graphic content based on the prompt text determined by the publishing requirement and the intelligent entity corresponding to the target field to which the prompt text belongs; determines at least one publishing operation information for the graphic content according to the graphic content, the publishing task indicated by the publishing requirement, the current page content of the target software indicated by the publishing task, and the operation generation model, so as to drive the robot process automation (RPA) system to perform graphic content publishing processing.

26. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11; Alternatively, the method of claim 12 is performed.

27. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 11; or, to execute the method according to claim 12.

28. A computer program product comprising a computer program, wherein the computer program When executed by a processor, the method according to any one of claims 1 to 11 is implemented; Alternatively, the method according to claim 12 is implemented.

Images