AIGC method and device in RPA scene and storage medium

By designing the RPA interface and writing UI logic on the QT5 framework, combining OCR and Embedding technology, using Milvus and pre-trained language models to generate reply content, solving the security and development complexity of Hook technology, achieving a more natural and stable Automated responses reduce the risk of malicious exploitation and detection.

CN119088356APending Publication Date: 2024-12-06BEIJING LIANYUN TIANXIA TECH CO LTD

Patent Information

Application Number
CN202411578025.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-07
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In the prior art, Hook technology requires in-depth internal modification of the application, which increases the risk of being exploited by malware, is low in security, is easily identified by the platform detection mechanism, resulting in account ban or restriction, and has high development cost and complexity.

Method used

Using the RPA method based on the QT5 framework, through the Qt Designer design interface, Qt Creator writes UI logic, uses OCR technology to analyze the screen image of the WeChat chat window, and Embedding technology converts the text content into a vector representation, stores it in the Milvus vector database, and uses pre-training to The language model generates reply content, simulates human interaction methods to automatically reply.

Benefits of technology

Reduces the risk of malicious exploitation, improves security and stability, reduces the possibility of detection, and the content generated through artificial intelligence is more natural, reducing the possibility of detection. RPA scripts can still be normal when the underlying system changes. Working, improving stability and reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088356A_ABST
    Figure CN119088356A_ABST
Patent Text Reader

Abstract

The invention relates to an AIGC method and device in an RPA scene and a storage medium, which are applied to the technical field of social media, and the method comprises the following steps: on the basis of a QT5 framework, designing an RPA interface by using a Qt Designer and compiling UI logic of the RPA interface by using a Qt Creator to obtain an RPA; the message content of the target equipment is obtained, the message content is converted into vector representation, similarity search is carried out on the vector representation, the label of the message content is obtained, the label and the message content are input into a pre-trained model together, reply content is obtained, and the existence of the label can further improve the accuracy of model output; according to the RPA technology, the task is executed by simulating the interaction mode of a human user, the bottom layer code of the application program does not need to be directly modified, compared with the Hook technology, the safety is higher, the content generated through artificial intelligence is more natural, the possibility that the content is detected by a platform is reduced, and the stability and the reliability are higher.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of social media technology, and in particular to an AIGC method, device and storage medium in an RPA scenario. Background Art

[0002] With the development of Internet technology, social media platforms such as WeChat have become an indispensable part of people's daily lives. Users have an increasing demand for efficient management and automated processing of daily social activities. For example, on the WeChat platform, users may want to automatically reply to messages, post status updates, and reply to comments to save time and maintain online activity. In this context, automation tools and technologies have been widely used.

[0003] However, the existing technology usually adopts Hook technology to achieve automatic reply. Since Hook technology needs to go deep into the application to make modifications, this increases the risk of being exploited by malware. If the Hook mechanism is taken over by malicious programs, it may lead to security issues such as user privacy leakage and remote control of the device. Different operating system versions or applications may have different Hook points, which means that the Hook mechanism needs to be developed and maintained separately for each version, increasing development costs and complexity. Many existing platforms and services have anti-cheating mechanisms that can detect abnormal automated behaviors. Hook technology is easily identified by these detection mechanisms, which may lead to the account being banned or restricted in use. Hook technology may cause the application to respond slowly or behave abnormally, affecting the user's normal usage experience. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an AIGC method, device and storage medium in an RPA scenario to solve the problems in the prior art: since the Hook technology needs to be modified in-depth inside the application, this increases the risk of being exploited by malware and has low security; the Hook technology is easily identified by the platform's detection mechanism, which in turn leads to the account being banned or restricted in use; and the problem of high development cost and complexity.

[0005] According to a first aspect of an embodiment of the present invention, an AIGC method in an RPA scenario is provided, the method comprising: Design RPA interface using Qt Designer based on QT5 framework; The UI logic of the RPA interface is written using Qt Creator based on the QT5 framework to obtain the RPA; The RPA is deployed on the target device, and the screen image of the WeChat chat window of the target device is obtained using the QScreen class of the QT5 framework; Analyze the text content in the screen image of the WeChat chat window of the target device through OCR technology; Convert the text content into a vector representation using Embedding technology, and store the vector representation in the Milvus vector database; The Milvus vector database performs a similarity search on the stored vector representations and outputs the data item that best matches the vector representation; Using the data item output by the Milvus vector database that best matches the vector representation as a label for the text content; The text content and labels are input into a pre-trained language model, the language model generates reply content according to the text content and labels, and the RPA automatically replies according to the generated reply content.

[0006] Preferably, The RPA interface designed by Qt Designer based on the QT5 framework includes: Create a new project in the Qt Designer and select the main window type; Select the desired controls from the toolbox and move them to the design area. Use the layout manager to organize the positions of the selected controls so that they can adapt to screens of different sizes. Set basic properties for each control; Save the designed interface as a .ui file.

[0007] Preferably, The Qt Creator based on the QT5 framework writes the UI logic of the RPA interface, and the RPA includes: Create a new Qt Widgets Application project in the Qt Creator, and import the .ui file designed by the QtDesigner into the project; According to the .ui file of the Qt Designer, the Qt Creator generates a corresponding header file and source file; In the header file and source file, write UI logic by defining logic processing functions; Use the connect function to connect the control's signal to the logic processing function; Use the built-in debugger of Qt Creator to debug the written UI logic to obtain RPA.

[0008] Preferably, The RPA automatically replies according to the generated reply content, including: The QApplication::sendEvent() function of the QT5 framework is used to simulate keyboard input and mouse click events to complete the operation of automatically replying to messages.

[0009] Preferably, The pre-trained language model includes: Get the raw text data for model training; Performing word segmentation processing on the acquired original text data through a word segmenter; Set the loss function during model training and train the pre-built Transformer architecture using the original text data after word segmentation. In the process of training the constructed Transformer architecture, future information is masked and predictions are made based on past context; Get the gradient of the loss function with respect to each weight, and adjust the weight through the optimizer until the loss function no longer decreases or reaches the preset number of training times, and obtain the pre-trained language model.

[0010] Preferably, it also includes: The QTimer class of the QT5 framework is used to periodically check whether a new message has arrived at the target device, and the automatic reply logic is triggered when a new message arrives.

[0011] According to a second aspect of an embodiment of the present invention, an AIGC device in an RPA scenario is provided, the device including: Interface design module: used to design RPA interface based on Qt Designer of QT5 framework; Logic writing module: used to write the UI logic of the RPA interface based on Qt Creator of QT5 framework to obtain RPA; Screen image acquisition module: used to deploy the RPA on the target device and use the QScreen class of the QT5 framework to acquire the screen image of the WeChat chat window of the target device; Text content recognition module: used to parse the text content in the screen image of the WeChat chat window of the target device obtained by OCR technology; Vector conversion module: used to convert the text content into vector representation using Embedding technology, and store the vector representation in the Milvus vector database; Vector search module: used for the Milvus vector database to perform similarity search on the stored vector representations and output the data item that best matches the vector representation; A label acquisition module: used to use the data item output by the Milvus vector database that best matches the vector representation as a label for the text content; Automatic reply module: used to input the text content and labels into a pre-trained language model, the language model generates reply content according to the text content and labels, and the RPA automatically replies according to the generated reply content.

[0012] According to a third aspect of an embodiment of the present invention, there is provided a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a main controller, each step in the above method is implemented.

[0013] The technical solution provided by the embodiments of the present invention may have the following beneficial effects: Based on the QT5 framework, this application uses Qt Designer to design the RPA interface and uses Qt Creator to write the UI logic of the RPA interface to obtain RPA; obtain the message content of the target device, and convert the message content into a vector representation using the Embedding technology, obtain the label of the message content by performing a similarity search on the vector representation, input the label and the message content into the trained model together to obtain the reply content, and the existence of the label can further improve the accuracy of the model output; this application uses RPA technology to perform tasks by simulating the interaction method of human users, without directly modifying the underlying code of the application, reducing the risk of malicious exploitation, and the content generated by artificial intelligence is more natural, reducing the possibility of detection. RPA is based on user interface (UI) level automation, which means that even if the underlying system changes, as long as the UI elements remain consistent, the RPA script can still work normally, improving stability and reliability.

[0014] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0016] Figure 1 is a flowchart of an AIGC method in an RPA scenario according to an exemplary embodiment; Figure 2 is a system schematic diagram of an AIGC device in an RPA scenario according to another exemplary embodiment; In the attached figure: 1-interface design module, 2-logic writing module, 3-screen image acquisition module, 4-text content recognition module, 5-vector conversion module, 6-vector search module, 7-label acquisition module, 8-automatic reply module. DETAILED DESCRIPTION

[0017] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0018] Embodiment 1 Figure 1 is a flowchart of an AIGC method in an RPA scenario according to an exemplary embodiment. Figure 1 As shown, the method includes: S1, design RPA interface based on Qt Designer of QT5 framework; S2, writing the UI logic of the RPA interface based on Qt Creator of QT5 framework to obtain RPA; S3, deploying the RPA on the target device, and using the QScreen class of the QT5 framework to obtain the screen image of the WeChat chat window of the target device; S4, parsing text content in the screen image of the WeChat chat window of the target device obtained by using OCR technology; S5, converting the text content into a vector representation using an Embedding technology, and storing the vector representation in a Milvus vector database; S6, the Milvus vector database performs a similarity search on the stored vector representations, and outputs the data item that best matches the vector representation; S7, using the data item output by the Milvus vector database that best matches the vector representation as a label for the text content; S8, inputting the text content and the label into a pre-trained language model, the language model generates a reply content according to the text content and the label, and the RPA automatically replies according to the generated reply content; It can be understood that the specific content of this embodiment is as follows: Design the interface using Qt Designer: Qt Designer is a visual tool that allows developers to drag and drop various controls to create user interfaces. It supports a variety of layouts and controls, such as buttons, text boxes, labels, lists, tables, etc. The following is the basic process of designing an interface using Qt Designer: Launch Qt Designer: Open the Qt Designer application; Create a new .ui file: Create a new project in Qt Designer and select "Main Window" as the main window type; Drag and drop controls: Drag the required controls from the toolbox to the design area; for example, you can add a QLineEdit for entering text, a QPushButton for triggering events, and a QTextEdit for displaying multi-line text output; Adjust the layout: Use layout managers (such as QVBoxLayout, QHBoxLayout) to organize the position of controls to adapt to screens of different sizes; Set properties: Set properties for each control, such as text, icon, color, font, etc.; Save .ui file: Save the designed interface as a .ui file.

[0019] Use Qt Creator integrated development environment to write UI logic: Qt Creator is a powerful IDE that integrates a code editor, build tools, debugger, and version control system. In Qt Creator, you can use C++ to write UI logic and combine the interface designed by Qt Designer with business logic. The following are the steps to write UI logic: Import .ui file: Create a new Qt Widgets Application project in Qt Creator and import the .ui file designed by Qt Designer into the project; Generate header and source files: Qt Creator will automatically generate a corresponding header and source file, which are generated based on the .ui file of Qt Designer; Write logic code: In the generated header files and source files, you can start writing UI-related logic processing functions; for example, define slot functions to respond to user operations, such as button click events; Connect signals and slots: Rewrite the part after the setupUi function in the generated class, and use the connect function to connect the control's signal (such as a button click signal) with the slot function (that is, the written logic processing function); Debugging and testing: Use Qt Creator's built-in debugger to debug UI logic and ensure that all functions work as expected; Build and deploy: Use Qt Creator to build your application and deploy it to the target platform for testing; Screen capture and OCR recognition: Use the QScreen class provided by QT5 to capture the screen image of the WeChat chat window, and use OCR technology (such as Tesseract OCR) to parse the text content in the image; Keyboard and mouse simulation: Use QT5's QApplication::sendEvent() function to simulate keyboard input and mouse click events to complete the automatic reply message operation; Scheduled tasks: Use QT5's QTimer class to periodically check whether new messages have arrived and trigger the automatic reply logic.

[0020] It is worth emphasizing that during the development process, it is important to note that the security and privacy policy of the WeChat client may limit the application of such automation tools. In addition, frequent automated operations may be identified as abnormal behavior by the WeChat backend, resulting in the account being banned. Therefore, when designing automation logic, consideration should be given to adding randomization factors to mimic the behavior patterns of real users and reduce the risk of detection.

[0021] AIGC (AI-Generated Content): In the field of Generative AI, combining AIGC (AI-Generated Content), Embedding technology and Milvus vector database can build an efficient content generation and retrieval system. The specific contents are as follows: Convert text content into vector representation using Embedding technology, and store the converted vector representation in the Milvus vector database, including: Embedding technology is a technology that converts discrete symbols (such as words or sentences) into continuous vectors. These vectors can capture the semantic relationship between symbols. In the field of natural language processing (NLP), word embedding is extremely important because it enables machines to understand and process text data. The following details the training process and implementation principle of word embedding: The implementation principle of Embedding: Word segmentation and vocabulary construction: Before starting training, you first need to segment the text data to split the sentences into individual words or tokens. Build a vocabulary containing all the unique words that appear in the training data. Each word is assigned a unique index number for subsequent vectorization; Vectorization: Each word is mapped to a fixed-length vector, which is called the embedding or embedding vector of the word. Initially, these vectors are usually randomly initialized. The spatial dimension of the embedding vector is much smaller than the size of the vocabulary, which allows words to be represented efficiently in the vector space and can capture similarities between words; Capturing semantic relationships Through the training process, the embedding vector can learn the semantic relationship between words. For example, the vectors of synonyms will be closer in the vector space, while different types of words will be distributed in different areas. Classic word embedding models such as Word2Vec and GloVe are based on the assumption that if two words often appear in the same context, they may be semantically related.

[0022] Embedding training process: Word2Vec: CBOW (Continuous Bag-of-Words): The CBOW model attempts to predict a word based on its context. Specifically, given the words in the context window around a central word, the model learns how to predict the central word.

[0023] Skip-Gram: In contrast to CBOW, the Skip-Gram model attempts to predict other words in its context based on a word. This method is more suitable for small-scale datasets. Both models learn word embeddings by maximizing the probability of word co-occurrence, usually using negative sampling or hierarchical softmax to simplify calculations; GloVe (Global Vectors for Word Representation): GloVe's approach is to learn word vectors based on global statistical information (co-occurrence matrix) rather than relying solely on local context windows.

[0024] It learns word embeddings by constructing a word co-occurrence matrix and using the elements in the matrix to optimize the objective function.

[0025] Training details: Loss function: For both Word2Vec and GloVe, the training process involves minimizing a loss function that measures the difference between the model's predictions and the actual co-occurrence frequencies. Regularization: To avoid overfitting, regularization techniques such as L1 or L2 regularization can be used; Optimizer: usually uses stochastic gradient descent (SGD) or other optimization algorithms such as Adam or Adagrad to update the embedding vectors and other model parameters; Through the above methods, the embedding layer can capture the complex relationship between words in high-dimensional space, so that the machine learning model can better understand and process text data. With the development of deep learning, people are also constantly exploring new methods to improve the design and training of the embedding layer to further improve the performance of the model.

[0026] Milvus: Milvus is an open source vector database management system designed for large-scale vector data storage and search. It supports a variety of vector similarity search algorithms (such as IVF_FLAT, IVF_SQ8, HNSW, etc.) and can efficiently process high-dimensional data. Milvus can be used to store vector data generated by Embedding and quickly retrieve the data item closest to the query vector. This closest data item will be used as a label for the text content of the chat to help the language model further understand the context, thereby generating more realistic automatic reply content. Pre-trained Transformer is a series of large language models based on the Transformer architecture, developed by OpenAI. The pre-trained model is pre-trained on a large amount of text data in an unsupervised manner, and can then be fine-tuned according to specific tasks. The following is a detailed description of the pre-trained model training process and its implementation principle: The implementation principle of the large model: Transformer Architecture: The core of the large model is the use of the Transformer architecture, which is a sequence-to-sequence model based entirely on the attention mechanism. Traditional RNN or LSTM models will encounter gradient vanishing or exploding problems when processing long sequences. The Transformer greatly improves training efficiency by processing each position in the sequence in parallel, and can better capture long-distance dependencies.

[0027] Transformer consists of two parts: encoder and decoder: The encoder is composed of multiple identical layers stacked together, each of which contains a multi-head self-attention mechanism and a feed forward neural network. The decoder contains similar components to the encoder, but adds an encoder-decoder attention layer, which allows the decoder to access the encoder's state. Autoregressive forecasting: The large model uses an autoregressive training method, which means that it can only see the past context when generating text, but not any future information. Specifically, during the training process, the model tries to predict the probability distribution of the next word in a given sequence. For the sequence "X1 X2 ... Xt", the model's goal is to predict "Xt+1"; The training process of the large model: Pre-training: Data preparation: Large models require a large amount of text data for pre-training. This data usually comes from text resources on the Internet, such as books, web pages, news, etc. Tokenization: The original text is segmented into tokens (subword units) by the tokenizer. This is to increase the generalization ability of the model. Large models use Byte Pair Encoding (BPE) or similar tokenization methods. Masking: During the training process, the input sequence will be passed to the model. Since the large model is an autoregressive model, future information will be masked during training to ensure that the model only makes predictions based on past context; Loss function: The model minimizes the loss function by maximizing the conditional probability of the next word given the context. The commonly used loss function is cross-entropy loss. Backpropagation: The model updates its weights through the backpropagation algorithm to better predict the next word. This process involves calculating the gradient of the loss function with respect to each weight and adjusting the weights through an optimizer such as Adam. Fine-tuning: Once the model has reached a certain level of performance after pre-training, it can be fine-tuned for specific tasks (such as question answering, translation, summarization, etc.). The fine-tuning stage involves continuing to train the model on a small amount of labeled data to adapt to the needs of the specific task. At this stage, some hyperparameters may be adjusted, such as the learning rate, and only the weights of some layers may be updated. After fine-tuning, the above-mentioned pre-trained language model is obtained.

[0028] The pre-trained language model is used to predict the input text content and labels to generate automatic reply content.

[0029] The large model achieves efficient parallel computing and can handle long-distance dependencies well by using the Transformer architecture based on the attention mechanism. Its training process involves unsupervised pre-training on large-scale text data and small-scale fine-tuning for specific tasks. This model can generate coherent and natural language output and has achieved remarkable success in a variety of NLP tasks. With the increase in model scale and technological advancement, the large model series models have significantly improved in language understanding ability and generation quality.

[0030] To summarize, to build a generative AI system based on the above techniques, you can follow these steps: Data preparation: First, a large amount of text data needs to be collected. This data will be used to train the generative AI model. The data preprocessing stage may include steps such as cleaning, word segmentation, and removal of stop words. Train the generative model: Use models such as big models to train data in a supervised or unsupervised manner to learn patterns in the data. Transfer learning may be used during training, that is, pre-training the model on a large corpus and then fine-tuning it for a specific task; Generate content embed: For each part of the generated content (such as sentences and paragraphs), the Embedding technology is used to convert it into a vector representation. This step can be completed offline, and the generated content and its vector are stored.

[0031] Construct a vector index: Use Milvus to store vector representations of generated content and build an efficient index structure so that the most relevant content can be found quickly when a new query is received. Retrieval and Generation: When the system receives a query request from a user, it also uses Embedding technology to convert the query into a vector form and uses Milvus to perform a vector similarity search. Based on the query results, the system can select the most matching existing content or further generate new content based on context information.

[0032] Dynamic update and optimization Over time, the model needs to continuously learn new data and regularly update the vector database and generative model, which helps keep the system accurate and up-to-date.

[0033] By integrating more advanced machine learning models, this embodiment can make more accurate and complex decisions based on large amounts of historical data and real-time information, thereby improving overall automation efficiency and accuracy; through the optimization of deep learning and natural language processing technology, it can more efficiently identify and parse key information in unstructured data and reduce the need for human participation; by adopting RPA technology in combination with AIGC technology to realize WeChat automatic reply to chat messages, automatic posting to Moments, automatic reply to Moments comments and other functions, it can not only effectively solve the security, stability and compatibility issues of existing Hook technology, but also ensure that the automation behavior is more natural and smooth, ultimately providing users with a safer, more reliable and efficient automation solution.

[0034] Embodiment 2 Figure 2 : is a system schematic diagram of an AIGC device in an RPA scenario according to another exemplary embodiment, the device including: Interface design module 1: Used to design RPA interface based on Qt Designer of QT5 framework; Logic writing module 2: used to write the UI logic of the RPA interface based on Qt Creator of QT5 framework to obtain RPA; Screen image acquisition module 3: used to deploy the RPA on the target device and use the QScreen class of the QT5 framework to acquire the screen image of the WeChat chat window of the target device; Text content recognition module 4: used to analyze the text content in the screen image of the WeChat chat window of the target device obtained by OCR technology; Vector conversion module 5: used to convert the text content into vector representation using Embedding technology, and store the vector representation in Milvus vector database; Vector search module 6: used for the Milvus vector database to perform similarity search on the stored vector representations and output the data item that best matches the vector representation; Label acquisition module 7: used to use the data item output by the Milvus vector database that best matches the vector representation as the label of the text content; Automatic reply module 8: used to input the text content and labels into a pre-trained language model, the language model generates reply content according to the text content and labels, and the RPA automatically replies according to the generated reply content.

[0035] Embodiment three: This embodiment provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a main controller, each step in the above method is implemented; It is understandable that the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0036] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0037] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0038] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0039] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0040] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0041] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0042] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0043] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0044] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An AIGC method in an RPA scenario, characterized in that: The method comprises: Design RPA interface using Qt Designer based on QT5 framework; The UI logic of the RPA interface is written using Qt Creator based on the QT5 framework to obtain the RPA; The RPA is deployed on the target device, and the screen image of the WeChat chat window of the target device is obtained using the QScreen class of the QT5 framework; Analyze the text content in the screen image of the WeChat chat window of the target device through OCR technology; Convert the text content into a vector representation using Embedding technology, and store the vector representation in the Milvus vector database; The Milvus vector database performs a similarity search on the stored vector representations and outputs the data item that best matches the vector representation; Using the data item output by the Milvus vector database that best matches the vector representation as a label for the text content; The text content and labels are input into a pre-trained language model, the language model generates reply content according to the text content and labels, and the RPA automatically replies according to the generated reply content.

2. The method according to claim 1, characterized in that: The RPA interface designed by Qt Designer based on the QT5 framework includes: Create a new project in the Qt Designer and select the main window type; Select the desired controls from the toolbox and move them to the design area. Use the layout manager to organize the positions of the selected controls so that they can adapt to screens of different sizes. Set basic properties for each control; Save the designed interface as a .ui file.

3. The method according to claim 2, characterized in that The Qt Creator based on the QT5 framework writes the UI logic of the RPA interface, and the RPA includes: Create a new Qt Widgets Application project in the Qt Creator, and import the .ui file designed by the QtDesigner into the project; According to the .ui file of the Qt Designer, the Qt Creator generates a corresponding header file and source file; In the header file and source file, write UI logic by defining logic processing functions; Use the connect function to connect the control's signal to the logic processing function; Use the built-in debugger of Qt Creator to debug the written UI logic to obtain RPA.

4. The method according to claim 3, characterized in that The RPA automatically replies according to the generated reply content, including: The QApplication::sendEvent() function of the QT5 framework is used to simulate keyboard input and mouse click events to complete the operation of automatically replying to messages.

5. The method according to claim 4, characterized in that The pre-trained language model includes: Get the raw text data for model training; Performing word segmentation processing on the acquired original text data through a word segmenter; Set the loss function during model training and train the pre-built Transformer architecture using the original text data after word segmentation. In the process of training the constructed Transformer architecture, future information is masked and predictions are made based on past context; Get the gradient of the loss function with respect to each weight, and adjust the weight through the optimizer until the loss function no longer decreases or reaches the preset number of training times, and obtain the pre-trained language model.

6. The method according to claim 1, characterized in that Also includes: The QTimer class of the QT5 framework is used to periodically check whether a new message has arrived at the target device, and the automatic reply logic is triggered when a new message arrives.

7. An AIGC device in an RPA scenario, characterized in that: The device comprises: Interface design module: used to design RPA interface based on Qt Designer of QT5 framework; Logic writing module: used to write the UI logic of the RPA interface based on Qt Creator of QT5 framework to obtain RPA; Screen image acquisition module: used to deploy the RPA on the target device and use the QScreen class of the QT5 framework to acquire the screen image of the WeChat chat window of the target device; Text content recognition module: used to parse the text content in the screen image of the WeChat chat window of the target device obtained by OCR technology; Vector conversion module: used to convert the text content into vector representation using Embedding technology, and store the vector representation in the Milvus vector database; Vector search module: used for the Milvus vector database to perform similarity search on the stored vector representations and output the data item that best matches the vector representation; A label acquisition module: used to use the data item output by the Milvus vector database that best matches the vector representation as a label for the text content; Automatic reply module: used to input the text content and labels into a pre-trained language model, the language model generates reply content according to the text content and labels, and the RPA automatically replies according to the generated reply content.

8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the main controller, each step of the AIGC method in an RPA scenario as described in any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Cross-platform software development method and device, electronic equipment and storage medium

    CN113238761A

  • Message reply method and device, storage medium and computer equipment

    CN115065652A

  • Multi-robot dialogue method and system for software-as-a-service platform

    CN117573834A

  • Intelligent dialogue method and device, computer equipment and storage medium

    CN117591649A

  • Man-machine interaction method and device based on large language model and electronic equipment

    CN117787409A

Cited By

  • Message processing method and device and storage medium

    CN120935131A