Social interface content perception and intelligent decision interaction method and system based on multi-modal large model and application of social interface content perception and intelligent decision interaction method and system

Abstract

The invention relates to the technical field of intelligent understanding and automatic interaction of a graphical user interface (GUI), and discloses a social interface content perception and intelligent decision interaction method and system based on a multi-modal large model and application of the social interface content perception and intelligent decision interaction method and system. Dynamic element extraction based on a feature icon group is carried out on a social interface screenshot, and a target pure image is obtained; performing multi-dimensional semantic analysis on the image content by using a multi-modal large model, generating a decision through a quantitative matching model, and driving LLM to generate customized comment content when comment is needed; performing accurate positioning on the interactive elements by adopting an OCR and GUI intelligent agent self-adaptive mixed strategy; and finally, the interaction action is executed through the RPA. According to the method, the defects of poor social interface recognition adaptability, weak semantic understanding, rigid decision and insufficient positioning robustness in the prior art are overcome, intelligent understanding and high-precision automatic interaction of a complex dynamic social interface are realized, and the execution efficiency and reliability of an RPA system in a social scene are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent understanding and automated interaction technology of graphical user interfaces (GUI), and in particular to a method, system and application of social interface content perception and intelligent decision-making interaction based on a multimodal large model. Background Technology

[0002] With the rapid development of information technology, Robotic Process Automation (RPA) has become a key technology for enterprises to improve operational efficiency and reduce labor costs. RPA uses software robots to simulate human operations on a graphical user interface (GUI), automating business processes and is widely used in areas such as financial reconciliation, data entry, customer service response, and report generation. In recent years, with the popularization of social media and the increasing prominence of social platforms in enterprise marketing, customer service, and brand management, the demand for automated interaction based on social interfaces has also grown rapidly. For example, enterprises need to use social media platforms for content publishing, user interaction, public opinion monitoring, and event promotion. These tasks typically involve the identification and interaction of elements such as posts, comments, buttons, and icons on social interfaces.

[0003] However, social interfaces are highly dynamic, diverse, and unstructured. Different social platforms (such as WeChat, Weibo, Douyin, Facebook, and Twitter) exhibit significant differences in interface design, layout style, and interaction logic; even on the same platform, the position, size, and form of interface elements may change on different devices (such as mobile phones, tablets, and computers) or different versions. Furthermore, social content itself possesses rich multimodal characteristics, including text, images, videos, and emoticons, with complex and dynamically changing semantics. Traditional RPA technologies, when dealing with such complex and ever-changing social interfaces, often rely on fixed rules, coordinate positioning, or simple image matching, making it difficult to achieve high-precision and robust automated interactions. This results in easily interrupted automated processes, low execution efficiency, and high maintenance costs.

[0004] Therefore, how to enable RPA systems to have intelligent perception, semantic understanding and adaptive interaction capabilities for social interfaces has become a technical challenge that current enterprise-level automation platforms urgently need to solve.

[0005] In existing technologies, automated interactions for social interfaces mainly rely on the following types of methods: Location methods based on coordinate or image template matching: These methods locate elements by recording screen coordinates or using pre-stored control image templates. However, this approach suffers from a high failure rate and poor adaptability when faced with changes in interface layout, resolution adjustments, or dynamic element loading. Text extraction methods based on Optical Character Recognition (OCR): This method identifies text information in the interface using OCR technology and then performs simple matching based on keywords. While this method is effective for plain text buttons, its accuracy drops significantly for icon buttons, complex layouts, multilingual content, and font style variations, and it lacks semantic understanding capabilities. The rule-based decision-making method uses pre-defined "if-then" rules for interactive decisions. However, this method cannot handle complex and varied social content, requires significant rule maintenance, and struggles to achieve personalized and intelligent interactions (such as generating context-appropriate comments). Traditional computer vision and GUI tree parsing methods identify controls by parsing the GUI tree structure of an application or using visual algorithms such as edge detection and feature matching. However, social platforms often use hybrid rendering or dynamically generated interfaces, resulting in unstable GUI tree structures and easily disturbed visual features, leading to insufficient robustness in recognition.

[0006] In summary, existing technologies have the following shortcomings in automating interactions on social interfaces: First, existing technologies suffer from low accuracy and poor adaptability in identifying interface elements. They cannot effectively cope with the diversity of layouts caused by different device parameters, platform differences, and interface updates. Element extraction is easily interfered with by factors such as occlusion, overlap, and dynamic loading. Secondly, existing technologies have weak semantic understanding capabilities and lack the ability to perform deep semantic analysis on interface content (especially multimodal content). They are unable to understand abstract information such as the theme, emotion, intent, and value of posts, resulting in interaction decisions remaining at the level of superficial keyword matching. Third, the existing interactive decision-making mechanism is rigid, and the decision-making logic relies on fixed rules, making it impossible to dynamically generate matching interactive strategies (such as likes, comments, and shares) and personalized interactive content (such as customized comments) based on the semantics of the content. Finally, existing technologies for locating interactive elements are limited to a single method, relying solely on coordinates, templates, or OCR. This results in a high failure rate in complex scenarios (such as pure icon buttons, text interference, and low image quality) and a lack of adaptive switching and fusion mechanisms.

[0007] In view of the above-mentioned technical problems, those skilled in the art urgently need an automated interaction method for social interfaces that can integrate visual perception, semantic understanding and intelligent decision-making. Summary of the Invention

[0008] The purpose of this invention is to solve the above problems by designing a method, system, and application for content perception and intelligent decision-making interaction of social interfaces based on a multimodal large model. This method creatively integrates multimodal large models, OCR technology, GUI models, and RPA technology. Through innovative dynamic element extraction, multi-dimensional semantic analysis, adaptive positioning, and decision execution mechanisms, it achieves accurate understanding and intelligent interaction of complex social interfaces, fundamentally improving the applicability, execution efficiency, and robustness of RPA systems in social scenarios. Among them, the Multimodal Large Language Model (MLM) has demonstrated powerful capabilities in image understanding, natural language processing, and cross-modal reasoning. The core of this invention's technical solution lies in how to deeply integrate the MLM with RPA technology to build a complete, robust, and adaptive automated interaction framework that extends from interface awareness, semantic analysis, decision localization to action execution.

[0009] The technical solution of the present invention to achieve the above objectives is a method for content perception and intelligent decision-making interaction of social interfaces based on a multimodal large model, which includes the following steps: S1: Dynamically extract interface elements from a screenshot of the social interface to obtain a clean image of the target element; S2: Perform content semantic analysis on the clean image and generate an interaction decision based on multimodal semantic matching. The interaction decision includes no interaction, like, comment, or share; S3: Based on the interaction decision, adaptively locate the corresponding interactive element in the screenshot of the social interface to obtain the interaction coordinates; S4: Based on the interaction decision and the interaction coordinates, execute the corresponding interactive action through Robotic Process Automation (RPA).

[0010] The specific process of dynamically extracting interface elements in step S1 includes: S11: identifying at least one feature icon group in the screenshot of the social interface, wherein the feature icon group consists of multiple icons with a fixed relative positional relationship; S12: dynamically calculating the number and boundary range of target elements based on the number and spatial coordinates of the identified feature icon groups; S13: calculating an adaptive offset based on the relative positional features of the feature icon groups and the boundary of the target elements, cropping the target elements, and obtaining the clean image.

[0011] The specific process of generating interactive decisions based on multimodal semantic matching in step S2 includes: S21: Inputting the clean image and the preset multimodal semantic matching prompt word template into the multimodal large model to obtain the multi-dimensional semantic analysis results of the content; S22: Calculating the fit weight between the content semantics and the preset multiple interactive behavior modes based on the semantic analysis results; S23: Selecting the corresponding interactive decision from the four-level system of no interaction, like, comment, and share based on the quantitative evaluation results of the fit weight.

[0012] When the interaction decision is a comment, the specific process of generating the interaction decision based on multimodal semantic matching in step S2 further includes: S24: generating customized prompt words based on the semantic parsing result and the preset interaction mode; S25: inputting the customized prompt words into the Large Language Model (LLM) to generate customized comment content.

[0013] In step S3, the adaptive positioning adopts a hybrid coordinate positioning strategy, which includes: if the interactive element is determined to be a button with text labels, then optical character recognition (OCR) technology is used to identify the corresponding text area and locate its center coordinates; if the interactive element is determined to be a pure icon button or the confidence level of optical character recognition (OCR) technology is lower than the threshold, then a GUI agent based on a multimodal large model is used to identify and locate the target icon through prompt words.

[0014] When the optical character recognition (OCR) technology is used for positioning, if the recognition of a single text is inaccurate, it further identifies a group of text icons that have a fixed relative position with the interactive element to assist in judgment and positioning.

[0015] The hybrid coordinate positioning strategy dynamically selects and switches between pure OCR positioning mode, pure GUI intelligent agent positioning mode, or a hybrid positioning mode that combines both, based on the image quality and content features of the social interface screenshot.

[0016] A social interface content perception and intelligent decision-making interaction system based on a multimodal large model includes the following functional modules: a dynamic interface element extraction module, used to process screenshots of the social interface to obtain clean images of target elements; a content semantic analysis and interaction decision module, used to perform multi-dimensional semantic analysis on the clean images and generate interaction decisions based on multimodal semantic matching; an adaptive localization module for interactive elements, used to locate interactive elements using a hybrid coordinate localization strategy according to the interaction decisions to obtain interaction coordinates; and an interaction action execution module, used to execute corresponding interactive actions based on Robotic Process Automation (RPA) technology, according to the interaction decisions and the interaction coordinates.

[0017] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method.

[0018] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0019] Compared with the prior art, the present invention has the following non-obvious technical features: First, this invention employs a dynamic boundary positioning algorithm for feature icon groups. Unlike existing fixed-area cropping or single-icon recognition methods, this invention innovatively proposes to identify "feature icon groups" with fixed relative positional relationships (such as the combination of like, comment, and share icons in a social post area), and dynamically infer the number and boundaries of target elements (such as a single post area) based on the number of icon groups and spatial coordinates. Combined with adaptive offset calculation, high-precision cropping is achieved. This method can effectively handle various states such as element integrity, coexistence, and partial occlusion, breaking through the dependence of traditional methods on layout stability. Secondly, this invention employs a multimodal semantic matching and a four-level interactive decision-making system. This system differs from keyword-based or simple rule-based decision-making; it combines semantic matching prompts from a multimodal large-scale model with a dynamic weight matching mechanism. The system performs multi-dimensional semantic analysis (theme, sentiment, value, etc.) on the extracted clean interface content and constructs a four-level quantitative interactive decision-making system for "no interaction, like, comment, and share." This decision-making process, based on a quantitative evaluation of semantic fit, achieves intelligent mapping from content understanding to interaction strategy generation. Third, this invention employs a customized comment content generation mechanism based on LLM. When the decision is "comment interaction," a Large Language Model (LLM) is introduced. Based on the parsed semantic features of the content and the preset interaction mode, customized prompt words driving the LLM are automatically generated, thereby producing comment content that conforms to the context, values, and preset style. This achieves a leap from "selection" to "generation" of interactive content, which is not available in existing technologies. Fourth, this invention implements an adaptive localization strategy that combines OCR and GUI intelligent agents. For the localization of interactive elements, this invention is not limited to a single technology, but innovatively constructs a hybrid mode of OCR technology and GUI intelligent agent technology based on a multimodal large model. The optimal localization method (pure OCR, pure GUI intelligent agent, or a combination of both) is dynamically selected based on image quality and content features (such as the presence of clear text labels), and corresponding fusion and switching logic is designed (such as using text icon groups for auxiliary judgment), which greatly improves the localization accuracy and robustness in scenarios with text buttons, pure icon buttons, and complex interference. Finally, this invention realizes a complete closed-loop architecture from perception, understanding, decision-making to execution. This invention does not improve a single link in isolation, but constructs a collaborative working framework that includes four core modules: "dynamic extraction of interface elements", "content semantic analysis and interactive decision-making", "adaptive positioning of interactive elements", and "execution of interactive actions". The modules are closely connected through innovative mechanisms to form an intelligent interactive closed loop of "perception-understanding-decision-execution". The overall solution has a high degree of systematicity and collaborative innovation.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention can significantly improve the accuracy and adaptability of interface element recognition. Through the dynamic boundary positioning algorithm based on feature icon groups, it can effectively adapt to the interface layout changes caused by different devices, platforms and versions, accurately extract target elements, reduce interference information, and provide reliable input for subsequent analysis. 2. This invention enables deep semantic understanding and intelligent decision-making. It utilizes the powerful understanding capabilities of multimodal large models to perform multi-dimensional and in-depth semantic analysis of social content, and combines quantitative matching models to achieve intelligent interactive decision-making, making automated interaction more in line with content context and business intent. 3. This invention enables the automatic generation of personalized interactive content. By integrating LLM, it can automatically generate customized and context-appropriate comment content in scenarios where comments are required, greatly expanding the depth and anthropomorphism of automated interaction and improving the quality of interaction. 4. This invention enhances the robustness of interactive element positioning by adopting a hybrid positioning strategy that combines OCR and GUI intelligent agent adaptation. This strategy can flexibly handle various buttons and complex scenarios, ensuring high-precision element positioning under different conditions and reducing the risk of automation process interruption. 5. This invention improves the overall execution efficiency and reliability of RPA systems. Through the above innovations, this method enables RPA systems to stably and accurately execute complex automated interactive tasks in highly dynamic and unstructured social interface environments, expanding the application scope of RPA technology and reducing deployment and maintenance costs. 6. This invention creates a technological synergy advantage: It creatively integrates multiple cutting-edge technologies such as multimodal large models, OCR, GUI models and RPA, forming a technological synergy effect of "1+1>2", and providing a systematic solution for the field of intelligent interface interaction. Attached Figure Description

[0021] Figure 1 This is a flowchart of a social interface content perception and intelligent decision-making interaction method based on a multimodal large model, as described in Embodiment 1 of the present invention; Figure 2This is a flowchart of the dynamic extraction process of interface elements as described in Embodiment 1 of the present invention; Figure 3 This is a flowchart of the content semantic analysis and interactive decision-making process described in Embodiment 1 of the present invention; Figure 4 This is a flowchart of the adaptive positioning of interactive elements as described in Embodiment 1 of the present invention; Figure 5 This is a flowchart of the interactive action execution described in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of a social interface content perception and intelligent decision-making interaction system based on a multimodal large model, as described in Embodiment 2 of the present invention. Detailed Implementation

[0022] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings; Example 1; A method for content perception and intelligent decision-making interaction in social interfaces based on a multimodal large model, such as Figure 1 As shown, the method includes the following steps: S1: Dynamically extract interface elements from a screenshot of the social interface to obtain a clean image of the target element; S2: Perform content semantic analysis on the clean image and generate an interaction decision based on multimodal semantic matching. The interaction decision includes no interaction, like, comment, or share; S3: Based on the interaction decision, adaptively locate the corresponding interactive element in the screenshot of the social interface to obtain the interaction coordinates; S4: Based on the interaction decision and the interaction coordinates, execute the corresponding interactive action through Robotic Process Automation (RPA).

[0023] In this method, Robotic Process Automation (RPA) is a technology that automates business processes by using software robots to simulate human actions on a graphical user interface (GUI). A typical RPA process includes four key stages: interface awareness, logical decision-making, element localization, and action execution. First, the system needs to identify and parse the current interface content (e.g., obtaining text and control information through OCR); second, it analyzes the identification results based on preset rules and makes interaction decisions; then, it accurately locates the target interactive elements (such as buttons, input boxes, etc.) in the interface; finally, it completes the automated task by simulating clicks, inputs, swipes, and other operations. Traditional RPA often relies on fixed rules and coordinate localization, resulting in poor robustness in complex and dynamically changing social interfaces.

[0024] This invention introduces a multimodal large model and adaptive mechanism into traditional RPA, providing an intelligent social interface interaction method based on a multimodal large model. By constructing an intelligent understanding and autonomous decision-making framework for interface content, it achieves accurate identification of interface elements, semantic analysis, and automated execution of interactive operations. This solution innovatively integrates multimodal large models, OCR recognition, GUI models, and RPA technology to construct a complete automated process from interface content parsing to interactive action execution, significantly improving the intelligence level and execution reliability of RPA in unstructured, highly dynamic social scenarios.

[0025] The method described in this invention mainly comprises the following four core parts: Part 1: Dynamic extraction of interface elements; This section primarily addresses the complexity of identifying target elements (such as post areas and images) in screenshots of social media interfaces. To address the diverse states of interface elements caused by different device parameters (single element completeness, multiple elements coexisting, partial element occlusion, etc.), an innovative dynamic boundary localization algorithm based on feature icon groups is proposed. Through deep analysis of interface features, multiple icon combinations with fixed relative positions (feature icon groups) are extracted, and their spatial coordinates are accurately identified using OCR technology. The number of target elements is dynamically estimated based on the number of icon groups (0 groups → 1 element, 1 group → 1-2 elements, 2 groups → 2-3 elements), and the complete element range is determined based on the vertical coordinate of the icon groups. Furthermore, by analyzing the relative positional features of the icon groups and element boundaries, adaptive offsets are calculated to achieve high-precision cropping of target elements, effectively eliminating interference information and providing clean input for subsequent analysis. This technology overcomes the limitations of traditional fixed-area cropping, achieving accurate extraction of target elements in complex interfaces. The dynamic extraction process of interface elements is as follows: Figure 2 As shown.

[0026] Part Two: Content Semantic Analysis and Interaction Decision-Making; This section constructs an interaction decision framework based on multimodal semantic matching to achieve precise correlation between content semantics and interactive behavior. It receives clean images output from the interface element extraction module, innovatively designs multimodal semantic matching prompt word templates, and combines the deep semantic understanding capabilities of a multimodal large-scale model to perform multi-dimensional analysis of the interface content. By extracting the core semantic features of the content (theme, sentiment, value, etc.), a dynamic weighted matching mechanism is constructed to quantitatively evaluate the fit between the content and preset interactive behaviors, generating interaction strategies.

[0027] The second part constructs the interactive behavior into a four-level system: 1) No interaction: The content does not match the preset interaction mode; 2) Like interaction: The content has moderate semantic value and is suitable for liking; 3) Comment interaction: The content has high semantic value and is suitable for commenting. Customized comment content needs to be generated.

[0028] 4) Sharing and Interaction: The content has high semantic value and aligns with shared values, making it suitable for sharing; For comment interaction, this section introduces an LLM content generation mechanism. When a comment is deemed necessary, the system automatically generates customized prompts based on content semantic features and preset interaction patterns, driving a multimodal big data model to generate comment content that conforms to the preset interaction patterns. The process of content semantic analysis and interaction decision-making is as follows: Figure 3 As shown.

[0029] Part Three: Adaptive Positioning of Interactive Elements; This section employs a hybrid coordinate positioning strategy to address the complexity of recognizing interactive elements across different social interfaces; A) For buttons with text labels (such as "Like"), OCR technology is used to achieve high-precision positioning and improve recognition efficiency: By recognizing all text areas in the screenshot, the text coordinates that match the characteristics of interactive buttons are filtered, and the center point of the area is taken as the final coordinates. If a single text cannot be correctly recognized, a group of text icons with fixed relative positions can be further recognized to assist in the judgment.

[0030] B) For scenarios with pure icons or severe interference, we innovatively introduce GUI intelligent agent technology. Based on the GUI interaction capabilities of multimodal large models, we achieve accurate recognition of pure icon buttons through customized prompts.

[0031] The core innovation of this part lies in constructing an adaptive switching mechanism between OCR and GUI agents: dynamically selecting the optimal localization method (pure OCR, pure GUI, or hybrid mode) based on image quality and content features, effectively improving coordinate localization accuracy and significantly enhancing the system's robustness in complex scenarios. The adaptive localization process for interactive elements is as follows: Figure 4 As shown.

[0032] Part Four: Execution of Interactive Actions; This section leverages RPA technology to achieve highly reliable action execution, translating decision results into precise interface operations. By parsing interaction types and corresponding coordinates, it achieves compatible operations across different interfaces and application versions. The interaction action execution flow realizes an automated closed loop for interface interaction. The interaction action execution flow is as follows: Figure 5 As shown.

[0033] This method, through the collaborative work of four parts, achieves automatic analysis of interface content, intelligent semantic understanding, precise interactive decision-making, and automated execution, forming a complete intelligent interface interaction closed loop and greatly improving the usability of intelligent RPA systems. Compared with existing technologies, this solution has significant innovations in the dynamic adaptability of interface element extraction, multi-dimensional quantification of semantic analysis, automatic generation of interactive content, and adaptive mechanisms for coordinate positioning, providing an efficient and accurate technical solution for intelligent interface interaction.

[0034] Example 2; A social interface content perception and intelligent decision-making interaction system based on a multimodal large model, such as... Figure 6 As shown, the system includes the following functional modules: a dynamic interface element extraction module, used to process screenshots of social interfaces to obtain clean images of target elements; a content semantic analysis and interaction decision module, used to perform multi-dimensional semantic analysis on the clean images and generate interaction decisions based on multi-modal semantic matching; an adaptive positioning module for interactive elements, used to locate interactive elements using a hybrid coordinate positioning strategy based on the interaction decisions to obtain interaction coordinates; and an interaction action execution module, used to execute corresponding interactive actions based on Robotic Process Automation (RPA) technology, according to the interaction decisions and the interaction coordinates.

[0035] Example 3; An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method.

[0036] Example 4; A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0037] The above technical solutions only embody the preferred technical solutions of the present invention. Any modifications that may be made by those skilled in the art to certain parts thereof embody the principles of the present invention and fall within the protection scope of the present invention.

Claims

1. A multi-modal large model based social interface content perception and intelligent decision interaction method, characterized in that, The method comprises the following steps: S1: dynamically extracting interface elements of a social interface screenshot to obtain a pure image of a target element; S2: performing content semantic analysis on the pure image to generate an interaction decision based on multi-modal semantic matching; S3: adaptively positioning a corresponding interaction element in the social interface screenshot according to the interaction decision to obtain an interaction coordinate; S4: executing a corresponding interaction action through robotic process automation according to the interaction decision and the interaction coordinate.

2. The method of claim 1, wherein, The process of interface element dynamic extraction in step S1 comprises: S11: identifying at least one feature icon group in the social interface screenshot, the feature icon group being composed of multiple icons with fixed relative position relationships; S12: dynamically calculating the number and boundary range of target elements according to the number and spatial coordinates of the identified feature icon groups; S13: calculating an adaptive offset based on the relative position characteristics of the feature icon group and the boundary of the target element, cropping the target element to obtain the pure image.

3. The method of claim 1, wherein, The specific process of generating an interaction decision based on multi-modal semantic matching in step S2 comprises: S21: inputting the pure image and a preset multi-modal semantic matching prompt word template into a multi-modal large model to obtain a multi-dimensional semantic analysis result of the content; S22: calculating the fitting degree weight between the content semantics and a plurality of preset interaction behavior modes according to the semantic analysis result; S23: quantitatively evaluating the result based on the fitting degree weight and selecting a corresponding interaction decision.

4. The method of claim 3, wherein, When the interaction decision is a comment, the specific process of generating an interaction decision based on multi-modal semantic matching in step S2 further comprises: S24: generating a customized prompt word based on the semantic analysis result and a preset interaction mode; S25: inputting the customized prompt word into a large language model to generate customized comment content.

5. The method of claim 1, wherein, The adaptive positioning in step S3 adopts a hybrid coordinate positioning strategy, which comprises: if it is judged that the interaction element is a button with text identification, an optical character recognition technology is used to identify the corresponding text area and locate the center coordinates thereof; if it is judged that the interaction element is a pure icon button or the confidence of the optical character recognition technology is lower than a threshold, a GUI agent based on a multi-modal large model is used to identify and locate the target icon through a prompt word.

6. The method of claim 5, wherein, When the optical character recognition technology is used for positioning, if single text recognition is not accurate, a text icon group with a fixed relative position to the interaction element is further identified to assist in judgment and positioning.

7. The method of claim 5, wherein, The hybrid coordinate positioning strategy dynamically selects and switches between a pure optical character recognition technology positioning mode, a pure GUI agent positioning mode or a hybrid positioning mode of cooperation between the two according to the image quality characteristics and content characteristics of the social interface screenshot.

8. A multi-modal large model based social interface content perception and intelligent decision interaction system, characterized in that, The system comprises the following functional modules: an interface element dynamic extraction module, configured to process a social interface screenshot to obtain a pure image of a target element; a content semantic analysis and interaction decision module, configured to perform multi-dimensional semantic analysis on the pure image and generate an interaction decision based on multi-modal semantic matching; an interactive element adaptive positioning module, configured to position an interactive element according to the interaction decision by using a hybrid coordinate positioning strategy to obtain an interaction coordinate; and an interactive action execution module, configured to execute a corresponding interactive action according to the interaction decision and the interaction coordinate based on a robot process automation technology.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 7.

Images