Multimodal perception based cross-language social intelligent interaction method and system

Abstract

The application provides a cross-language social intelligent interaction method and system based on multi-modal perception, relates to the technical field of automatic human-computer interaction, and solves the problems of high maintenance cost, poor robustness and weak cross-language adaptability of the prior art. In the method, text elements and icon elements in a social application interface screen image are extracted, and a structured interface element description set is generated by fusion; a natural language task instruction input by a user is received, a large language model is called for semantic reasoning in combination with the foregoing set, the screen image and a social semantic mapping library, the task instruction is parsed into an ordered atomic operation sequence; corresponding target interaction elements are repositioned, matching device operation instructions are generated for execution, and the process from screen extraction to instruction execution is cyclically executed until the interaction task is completed. The application reduces the automatic operation and maintenance cost of the social platform and improves the cross-language adaptability of the social platform.

Description

Technical Field

[0001] This invention relates to the field of automated human-computer interaction technology, specifically to a cross-language social intelligent interaction method and system based on multimodal perception. Background Technology

[0002] With the rapid development of the global social media ecosystem, businesses are increasingly demanding automated operations for online marketing, customer service, and other services. Automated interaction technologies, as a key means of reducing costs and increasing efficiency, have demonstrated significant value in improving response speed and service coverage. However, the complexity of real-world application environments still poses serious challenges to existing technologies.

[0003] Current mainstream automation implementation paths mainly rely on two types of technologies: one is calling the platform's private application programming interfaces (APIs), which is limited by the closed nature and frequent changes of the API policies of various social platforms, making cross-platform adaptation difficult and posing compliance risks; the other is script-based solutions based on graphical user interface control structure recognition, simulating user operations by parsing underlying control properties or preset coordinates. These methods can be effective in the short term when the interface structure is stable, but they struggle to cope with the high-frequency iteration of social media applications.

[0004] Social media interfaces frequently undergo layout adjustments, element restructuring, or interaction logic optimization to adapt to user habits and market strategies. Once a version update occurs, scripts relying on fixed control identifiers or coordinates are highly susceptible to failure, leading to operational interruptions or accidental triggering. The operations and maintenance team must repeatedly invest manpower in script correction and rewriting, significantly increasing long-term maintenance costs and weakening the reliability and business continuity of automated systems. In multi-platform collaborative operation scenarios, differences in interface design specifications further amplify the fragmentation problem of these technical solutions.

[0005] In a globalized operating environment, multilingual interface support has become the norm. The same function can appear significantly different in text labels, icon styles, and even layout logic across different language versions. Traditional solutions often rely on single-language keyword matching or static image templates for element location, but their accuracy drops drastically when faced with language switching, mixed text, or regionally customized interfaces. Interface text in non-preset languages ​​is difficult to parse effectively, and strategies that isolate text and icons fail to establish a semantic connection, resulting in a misunderstanding between high-level user commands and low-level interface elements.

[0006] Some attempts have attempted to incorporate basic optical character recognition (OCR) or simple template matching techniques, but these still face various challenges in complex interface scenarios. Low-quality screenshots, artistic fonts, and background interference can easily lead to text recognition errors; icon recognition is limited by template generalization capabilities, making it difficult to adapt to style evolution and dynamic rendering. Furthermore, existing technologies lack deep modeling capabilities for the semantics of social interactions, failing to intelligently map user intentions such as likes and shares to multimodal interface features, and struggling to activate context-aware fault-tolerance mechanisms when element location is obstructed.

[0007] In summary, existing technologies exhibit systemic flaws in complex scenarios involving dynamic interface evolution and multilingual environments, such as strong structural dependence, shallow semantic understanding, and weak cross-language adaptability. The industry urgently needs a new approach to intelligent interaction that transcends the constraints of underlying controls, deeply integrates multimodal interface information, and possesses semantic reasoning and adaptive capabilities, in order to build a more robust, efficient, and scalable foundation for automated social operations. Summary of the Invention

[0008] The purpose of this invention is to address the problems of high maintenance costs, poor robustness, and weak cross-language adaptability in existing automated interaction technologies for social platforms, which rely on proprietary interfaces or fixed UI control structures, especially in environments with frequent interface updates and multilingual environments. This invention proposes a cross-language intelligent social interaction method and system based on multimodal perception. Leveraging the powerful semantic understanding and analysis capabilities of large language models, it reduces the automated operation and maintenance costs of social platforms and improves their cross-language adaptability through an automated "parsing-planning-execution" process.

[0009] The present invention employs the following technical solutions to achieve its objective: A cross-language social intelligent interaction method based on multimodal perception includes the following steps: S1. Obtain the screen image of the current social application interface, and simultaneously perform text recognition and icon recognition processing on the screen image. Extract the text elements and their spatial location information, and the icon elements and their spatial location information in the interface respectively, and fuse them to generate a structured interface element description set containing element type, content and coordinate information. S2. Receive the natural language task instruction input by the user, combine the structured interface element description set, the screen image and the preset social semantic mapping library, call the large language model to perform semantic reasoning, and parse the natural language task instruction into an ordered sequence of atomic operations. S3. Based on the current atomic operation in the atomic operation sequence, and based on the structured interface element description set, locate the corresponding target interactive element, and generate a device operation instruction matching the target interactive element for execution; S4. After the device operation command is executed, return to step S1 to obtain the updated social application interface status, and continue to execute steps S1 to S3 until the interactive task corresponding to the natural language task command is completed.

[0010] Preferably, in step S1, the text recognition is achieved by a multilingual optical character recognition module that operates in parallel. The multilingual optical character recognition module is configured to synchronously process the text content in multiple languages ​​contained in the interface of the screen image. The icon recognition is achieved through an open set icon detection module, which is built based on the Grounding DINO object detection model and is specifically optimized and trained using a typical icon sample dataset from social platforms. The process of fusing and generating the structured interface element description set includes: aligning and integrating the recognition results of text elements and icon elements according to spatial coordinates, and labeling each element with an element type identifier, content text or visual feature description, and bounding box coordinate information.

[0011] Preferably, in step S2, the social semantic mapping library is a pre-built static knowledge base, which stores the mapping relationship between social platform functional semantics and multimodal interface features; the mapping relationship includes functional semantic fields, multilingual text keyword fields associated with the functional semantics, and corresponding visual icon feature fields; the visual icon feature fields include icon category labels and icon visual description information, and the multilingual text keyword fields include the text expression of the same function in different language version interfaces.

[0012] Preferably, in step S3, the location of the corresponding target interactive element adopts a progressive location strategy. First, based on the social semantic mapping library, semantic matching is performed on the elements in the structured interface element description set to determine the target interactive element. If semantic matching fails to locate the element, a sliding window geometric spatial neighborhood search is performed within the interface area covered by the structured interface element description set. If the sliding window geometric spatial neighborhood search still fails to locate the target interactive element, cross-validation is performed by combining the spatial position relationship between the icon element and its neighboring text elements and the contextual semantics to finally determine the coordinate position of the target interactive element.

[0013] Specifically, in steps S2 and S3, the atomic operations in the atomic operation sequence include click operation, swipe operation, text input operation, return operation, and wait operation; The click operation corresponds to the generation of a touch click instruction that acts on the coordinate position of the target interactive element; The sliding operation corresponds to the generation of a trajectory sliding command that defines the start and end coordinates; The text input operation corresponds to generating an input instruction to fill a specified input box with preset text content; The return operation corresponds to the generation of a system return instruction that triggers the interface-level rollback; The waiting operation corresponds to the generation of an execution pause instruction for a set duration.

[0014] This invention also provides a cross-language social intelligent interaction system based on multimodal perception that implements the aforementioned method, comprising the following functional modules: The visual perception unit is configured to capture the screen image of the current social application interface in real time, and perform text recognition processing and icon recognition processing on the screen image in parallel, respectively extracting the content and spatial coordinate information of each text element in the interface, and the visual feature identifier and spatial coordinate information of each icon element, and fusing the extraction results to generate a structured interface element description set. The task planning unit is communicatively connected to the visual perception unit and is configured to receive natural language task instructions input by the user, call a large language model, combine the structured interface element description set, the screen image and the preset social semantic mapping library, perform multimodal semantic reasoning, and output an ordered atomic operation sequence corresponding to the natural language task instructions. The action execution unit is communicatively connected to the task planning unit and the visual perception unit. It is configured to parse the atomic operation to be executed in the atomic operation sequence, locate the coordinate position of the target interactive element according to the structured interface element description set, generate a device operation instruction that matches the atomic operation type and the target interactive element, and control the target terminal device to execute the device operation instruction. In this process, after the action execution unit completes the execution of the current device operation command, it sends an interface update trigger signal to the visual perception unit, driving the visual perception unit to recapture the updated social application interface image, thereby repeating the process of capturing the screen image and executing the device operation command in a loop.

[0015] Preferably, the visual perception unit includes a multilingual optical character recognition module and an open set icon detection module; The multilingual optical character recognition module is configured to simultaneously perform recognition processing of multiple language texts on the screen image, and output the content text, language type identifier and bounding box coordinates of each text element; The open set icon detection module is built on the Grounding DINO object detection architecture and uses a typical icon sample dataset from social platforms for parameter optimization training. It is configured to output the category label, confidence score, and bounding box coordinates of each icon element. The visual perception unit is configured to align and integrate the two types of recognition results, text and icons, based on spatial coordinates to generate a structured set of interface element descriptions that includes element type, content description, visual feature identifier, and precise coordinates.

[0016] Preferably, the social semantic mapping library is stored in a storage module that is communicatively connected to the task planning unit; the social semantic mapping library contains multiple mapping entries, each mapping entry having a functional semantic field, a multilingual text keyword field, and a visual icon feature field; The multilingual text keyword field records the set of text keywords corresponding to the same functional semantics in different language version interfaces; The visual icon feature field records the icon category label, icon visual description text, and typical icon sample feature vector that are associated with the semantic meaning of the function.

[0017] Preferably, the action execution unit is configured with an element localization submodule, which is used to execute a progressive localization process: semantic matching of elements in the structured interface element description set based on the social semantic mapping library to determine the target interactive element; if the semantic matching fails, a sliding window mechanism is initiated within the interface area covered by the structured interface element description set to perform geometric spatial neighborhood retrieval according to a preset step size and window size; if the neighborhood retrieval still fails to locate the element, the context verification module is called to perform cross-verification by combining the spatial relative position, arrangement order, and semantic relevance of the icon element and its neighboring text elements, and output the final coordinate position of the target interactive element.

[0018] Specifically, the action execution unit is further configured with an instruction generation submodule, which is used to generate corresponding device operation instructions according to the atomic operation type, as follows: When the atomic operation is a click operation, a touch click instruction containing the target coordinate position is generated; When the atomic operation is a sliding operation, a trajectory sliding instruction containing the start coordinates and the end coordinates is generated; When the atomic operation is a text input operation, a text fill instruction containing the target input box identifier and the text to be input is generated; When the atomic operation is a return operation, a system-level interface rollback instruction is generated; When the atomic operation is a waiting operation, an execution pause instruction containing a preset delay duration parameter is generated.

[0019] In summary, due to the adoption of this technical solution, the beneficial effects of this invention are as follows: This invention significantly enhances the adaptability and execution stability of automated interactions on social platforms in complex environments through the synergy of multimodal perception and semantic reasoning. The system can simultaneously parse multilingual text content and diverse icon visual features presented in the interface, generating a unified description of interface elements. This effectively addresses the recognition challenges caused by language and regional differences, interface localization customization, or version iterations in global operations, ensuring accurate understanding and reliable execution of interactive commands in multilingual scenarios.

[0020] This invention integrates a pre-built social semantic mapping library with the deep reasoning capabilities of a large language model, accurately converting user natural language commands into atomic operation sequences that conform to interface logic. This achieves semantic alignment from abstract user intent to specific interface functions, significantly improving the accuracy and contextual adaptability of operation planning.

[0021] In this invention, the system essentially adopts a purely vision-driven interaction architecture, completely avoiding dependence on platform-specific interfaces or underlying control structures. It possesses excellent cross-platform versatility and robustness in interface evolution, and can easily cope with the technical adaptation pressure brought about by the high-frequency updates of social applications, significantly reducing long-term operational complexity and script maintenance costs.

[0022] In terms of element localization on the screen interface, this invention implements a progressive strategy, combining semantic matching, geometric spatial neighborhood retrieval, and icon-text context cross-validation in sequence to build a multi-level fault-tolerant mechanism. Even when interface elements change dynamically or recognition confidence fluctuates, it can still accurately lock the target, effectively reducing misoperation and redundant exploration behavior.

[0023] The cyclic execution process of this invention enables the system to perceive the evolution of the interface state in real time and dynamically adjust the subsequent operation sequence, ensuring the smooth progress and high-quality completion of multi-step interactive tasks, and providing technical support for social automated operation that combines intelligence and engineering practicality. Attached Figure Description

[0024] The present invention is described in detail with reference to the following figures, which include three figures as follows: Figure 1 This is a schematic diagram illustrating the overall process of the cross-language social intelligent interaction method of the present invention; Figure 2 This is a schematic diagram illustrating the architecture and workflow of the cross-language social intelligent interaction system of the present invention; Figure 3 This is a schematic diagram of the parallel processing architecture of the visual perception unit in the system of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The parts of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0027] Example 1 A cross-language social intelligent interaction method based on multimodal perception. Figure 1 The overall process of this method is briefly described below and can be viewed concurrently; the key steps of this method can be summarized as follows: S1. Obtain the screen image of the current social application interface, and simultaneously perform text recognition and icon recognition processing on the screen image. Extract the text elements and their spatial location information, and the icon elements and their spatial location information in the interface respectively, and fuse them to generate a structured interface element description set containing element type, content and coordinate information. S2. Receive the natural language task instructions input by the user, combine the structured interface element description set, screen images and the pre-built social semantic mapping library, call the large language model to perform semantic reasoning, and parse the natural language task instructions into an ordered sequence of atomic operations. S3. Based on the current atomic operation in the atomic operation sequence, locate the corresponding target interactive element based on the structured interface element description set, and generate a device operation command that matches the target interactive element for execution. S4. After the device operation command is executed, return to step S1 to obtain the updated social application interface status, and continue to execute steps S1 to S3 until the interaction task corresponding to the natural language task command is completed.

[0028] This embodiment will describe in detail the details and preferred methods for each step according to the above method steps.

[0029] In step S1, text recognition is achieved through a multilingual optical character recognition module that runs in parallel. The multilingual optical character recognition module is configured to synchronously process the text content in multiple languages ​​contained in the screen image. Icon recognition is achieved through the Open Set Icon Detection Module, which is built on the GroundingDINO object detection model and is specifically optimized and trained using a typical icon sample dataset from social platforms. The process of merging and generating a set of structured interface element descriptions includes: aligning and integrating the recognition results of text elements and icon elements according to spatial coordinates, and labeling each element with an element type identifier, content text or visual feature description, and bounding box coordinate information.

[0030] In this embodiment, the complete screen image of the current social application interface can be captured in real time through the screen capture interface of the terminal device. This image completely records the visual presentation state of all visible elements in the interface, serving as the raw input data source for multimodal parsing. The screen capture operation can be triggered by the internal state machine of the execution device to ensure that the image content is strictly synchronized with the interface state at the moment of user interaction.

[0031] Text recognition processing is performed independently by the multilingual optical character recognition module. This module adopts a parallel pipeline architecture and incorporates a deep neural network model trained on a multilingual corpus, enabling it to simultaneously detect and recognize characters in mixed multilingual text regions within an image. The recognition process includes text line localization, character segmentation, and language discrimination, ultimately outputting the recognized content, language attribute identifier, and precise bounding box position information for each text element in the screen coordinate system.

[0032] Icon recognition processing is performed synchronously by the Open Set Icon Detection module. This module is built on the Grounding DINO object detection framework, which integrates visual feature extraction and textual prompts to locate image targets based on semantic descriptions. Building upon the basic model, this module fine-tunes parameters using typical icon samples (likes, comments, shares, searches, etc.) and multiple variants collected from mainstream social media platforms, enabling the model to accurately recognize common icon visual forms in social media scenarios. The detection results include the category semantic label, confidence score, and bounding box coordinates for each icon element.

[0033] The fusion process aligns and integrates text recognition results with icon detection results within a unified spatial coordinate system. A preferred fusion method involves establishing a two-dimensional coordinate reference system with the top-left corner of the screen as the origin, spatially associating elements based on their geometric positions within the bounding boxes, and generating standardized description entries for each element. Each entry clearly identifies the element type, includes specific content information such as text content or icon semantic tags, and the coordinate parameters of the four corners of the bounding box. All entries are logically organized into a structured set of interface element descriptions, which is output in a parsable data structure format, providing a foundation for interface semantic representation in subsequent task planning.

[0034] In step S2, the social semantic mapping library is a pre-built static knowledge base that stores the mapping relationship between the functional semantics of the social platform and the features of the multimodal interface. The mapping relationship includes functional semantic fields, multilingual text keyword fields associated with the functional semantics, and corresponding visual icon feature fields. The visual icon feature fields include icon category labels and icon visual description information, and the multilingual text keyword fields include the text expression of the same function in different language versions of the interface.

[0035] In this embodiment, the system receives natural language task instructions input by the user through the terminal interface. These instructions, in free text form, fully express the user's desired social interaction goals, including specific actions such as searching for contacts, posting content, or forwarding information. The instruction content is captured by the system input interface and used as the basis for semantic parsing before entering the processing flow.

[0036] As a preferred embodiment, the social semantic mapping library resides in the system storage module as a pre-built static knowledge base. The mapping entries are organized in the form of a structured data table, as shown in Table 1 below.

[0037] Table 1. Schematic diagram of structured data in the social semantic mapping library

[0038] Each entry contains: a functional semantic field, used to identify standard interactive functions in the social platform; a multilingual text keyword field, which includes all textual tags that may appear in different language versions of the interface for this function; and a visual icon feature field, which records the icon category identifier associated with this function and the natural language description text of that icon. During the parsing process, matching entry information in the mapping library is dynamically retrieved based on the element content in the structured interface element description set.

[0039] In this embodiment, the user command text, the type identifiers, content descriptions and spatial coordinate information of each element in the structured interface element description set, and the content of the retrieved mapping library entries are semantically fused to construct a structured prompt text, which is then input into the large language model. In actual configurations that support multimodal input, the current screen image is also preferably used as a supplementary visual context input to help the model understand the overall layout of the interface and the spatial relationships of elements. The large language model performs deep semantic parsing based on the prompt content. Through an internal attention mechanism and logical reasoning module, it associates and matches the action intent and functional semantics in the user command, and combines the spatial distribution and semantic attributes of interface elements to deduce the minimum operation unit required to complete the task and its logical order.

[0040] The model ultimately outputs a strictly sequential sequence of atomic operations. Each atomic operation in the sequence is clearly labeled with its operation type identifier, the reference basis of the target element, and the execution parameter requirements. The whole sequence is encapsulated and transmitted using a standardized data structure, providing a complete instruction basis for the implementation of interactive actions.

[0041] In step S3, as a preferred embodiment, the corresponding target interactive element is located using a progressive positioning strategy. First, based on the social semantic mapping library, semantic matching is performed on the elements in the structured interface element description set to determine the target interactive element. If semantic matching fails to locate the element, a sliding window geometric spatial neighborhood search is performed within the interface area covered by the structured interface element description set. If the sliding window geometric spatial neighborhood search still fails to locate the target interactive element, cross-validation is performed by combining the spatial positional relationship between the icon element and its neighboring text elements and the contextual semantics to finally determine the coordinate position of the target interactive element.

[0042] In this embodiment, the atomic operation to be executed in the atomic operation sequence is analyzed to clarify its operation type and the functional semantics of the required interaction. Using a structured set of interface element descriptions as the baseline data source, the process of locating the target interactive element is initiated. Preferably, this process employs a multi-level progressive strategy to ensure accurate element identification and coordinate locking even in complex interface states.

[0043] The first stage of the localization process involves semantic matching. Based on the functional semantics implicit in atomic operations, it retrieves the corresponding multilingual text keyword set and visual icon feature description information from the social semantic mapping library. The retrieval results are then compared with the semantic similarity and feature comparison of the content text and icon category tags of each element in the structured interface element description set. When the description information of an element meets the preset matching conditions with an entry in the mapping library, the element is determined to be the target interactive element, and its bounding box coordinates are extracted as the operation reference.

[0044] If no valid match is found in the semantic matching stage, the process automatically transitions to the geometric space neighborhood retrieval stage. In this stage, a sliding window mechanism is initiated to systematically scan the interface space represented by the structured interface element description set. The window moves gradually in the horizontal and vertical directions at set steps, analyzing in real time the type distribution, content characteristics, and spatial correlation of elements within the window's coverage area with the requirements of atomic operations. By dynamically evaluating the confidence level of candidate elements, target regions that conform to the operational logic are selected.

[0045] When neighborhood retrieval still cannot uniquely identify the target, the process activates a contextual cross-validation mechanism. This mechanism considers the spatial topological relationship between the icon element and its surrounding text elements, comprehensively analyzing their relative positions, distance distribution, and semantic logical connections within the interface. For example, for the sharing function, it verifies whether the send icon forms a reasonable spatial combination with adjacent text such as share and forward, and ultimately confirms the accurate coordinates of the target interactive element through multi-source information fusion.

[0046] After the coordinates of the target interactive element are confirmed, standardized device operation instructions are generated based on the atomic operation type and coordinate parameters. The instruction content strictly corresponds to the operation semantics. The generated instructions are transmitted to the input driver layer of the terminal device through the system execution interface. The operating system parses and triggers physical interaction actions to complete the execution of the current atomic operation.

[0047] In this embodiment, the atomic operation sequences in steps S2 and S3 specifically include click operations, swipe operations, text input operations, return operations, and wait operations, as follows: The click operation generates a touch click command that acts on the coordinate position of the target interactive element; The sliding operation generates a trajectory sliding command that defines the start and end coordinates. The text input operation generates an input instruction to fill the specified input box with preset text content; The return operation generates a system return command that triggers a page-level rollback. The waiting operation generates an execution pause command for a set duration.

[0048] Examples of each operation can be found in Table 2 below.

[0049] Table 2. Schematic diagram of atomic operation types

[0050] Finally, in step S4 of this embodiment, after the device operation command is executed, the interface state update detection mechanism is automatically triggered to recapture the latest screen image of the current social application interface, and the processing flow is seamlessly connected to the multimodal interface parsing step in S1. This constructs a cyclical control structure that continuously executes the three core processing stages of multimodal interface parsing, semantic task planning, and precise action execution, dynamically advancing the sequential implementation of each operation in the atomic operation sequence.

[0051] The task progress can be monitored, and the execution pointer of the atomic operation sequence can be compared with the end position of the sequence in real time. When it is confirmed that all atomic operations in the sequence have been executed in sequence and there are no remaining instructions to be processed, it is determined that the interactive task corresponding to the natural language task instruction input by the user has been completed, and then the iterative loop process is terminated and a task end mark is generated.

[0052] Example 2 Building upon Example 1, this example provides a cross-language social intelligent interaction system based on multimodal perception, which implements the method in Example 1. The overall architecture and workflow of this system can be viewed concurrently. Figure 2 The illustration shows that it specifically includes the following functional modules: The visual perception unit is configured to capture the screen image of the current social application interface in real time, and perform text recognition processing and icon recognition processing on the screen image in parallel. It extracts the content and spatial coordinate information of each text element in the interface, and the visual feature identifier and spatial coordinate information of each icon element. The extraction results are then fused to generate a set of structured interface element descriptions. The task planning unit communicates with the visual perception unit and is configured to receive natural language task instructions input by the user, call a large language model, combine a structured interface element description set, screen images and a pre-built social semantic mapping library to perform multimodal semantic reasoning, and output an ordered atomic operation sequence corresponding to the natural language task instructions. The action execution unit communicates with the task planning unit and the visual perception unit. It is configured to parse the atomic operation to be executed in the atomic operation sequence, locate the coordinate position of the target interactive element based on the structured interface element description set, generate device operation instructions that match the atomic operation type and the target interactive element, and control the target terminal device to execute the device operation instructions.

[0053] In this embodiment, after the action execution unit completes the execution of the current device operation command, it sends an interface update trigger signal to the visual perception unit, driving the visual perception unit to recapture the updated social application interface image, thereby repeating the process of screen image capture to device operation command execution.

[0054] In a preferred embodiment, the visual perception unit includes a multilingual optical character recognition module and an open set icon detection module. This unit performs parallel processing of text and icons through these two sub-modules. The processing architecture can be found in [reference needed]. Figure 3 The illustration.

[0055] The multilingual optical character recognition module is configured to simultaneously perform recognition processing of multiple language texts on screen images, and output the content text, language type identifier, and bounding box coordinates of each text element.

[0056] The Open Set Icon Detection Module is built on the Grounding DINO object detection architecture and uses typical icon sample datasets from social platforms for parameter optimization training. It is configured to output the category label, confidence score, and bounding box coordinates of each icon element.

[0057] The visual perception unit is configured to align and integrate the recognition results of text and icons based on spatial coordinates, generating a structured set of interface element descriptions that includes element type, content description, visual feature identifier, and precise coordinates.

[0058] As a preferred embodiment, the social semantic mapping library is stored in a storage module that is communicatively connected to the task planning unit; the social semantic mapping library contains multiple mapping entries, and each mapping entry has a functional semantic field, a multilingual text keyword field, and a visual icon feature field.

[0059] The multilingual text keyword field records the set of text keywords corresponding to the same functional semantics in different language versions of the interface.

[0060] The visual icon feature field records the icon category label, icon visual description text, and feature vector of typical icon samples that are associated with the functional semantics.

[0061] In a preferred embodiment, the action execution unit is configured with an element localization submodule, which is used to execute a progressive localization process: semantic matching of elements in the structured interface element description set based on the social semantic mapping library to determine the target interactive element; if the semantic matching fails, a sliding window mechanism is initiated within the interface area covered by the structured interface element description set to perform geometric spatial neighborhood retrieval according to a preset step size and window size; if the neighborhood retrieval still fails to locate the element, the context verification module is called to perform cross-verification by combining the spatial relative position, arrangement order, and semantic relevance of the icon element and its neighboring text elements, and output the final coordinate position of the target interactive element.

[0062] In this embodiment, the action execution unit is also configured with an instruction generation submodule, which is used to generate corresponding device operation instructions according to the atomic operation type, as follows: When the atomic operation is a click operation, a touch click instruction containing the target coordinates is generated; When the atomic operation is a sliding operation, a trajectory sliding instruction containing the start and end coordinates is generated; When the atomic operation is a text input operation, a text fill instruction is generated that includes the target input box identifier and the text to be entered. When an atomic operation is a return operation, a system-level interface rollback instruction is generated; When an atomic operation is a wait operation, an execution pause instruction containing a preset delay duration parameter is generated.

[0063] This embodiment will now use a more specific scenario to illustrate in detail how the system works in actual application.

[0064] The system receives the user's natural language task instruction "Search for a friend and share a post with him," and then initiates the interaction process. The visual perception unit captures the image of the current homepage interface of the social platform application, the multilingual optical character recognition module simultaneously parses out text elements such as the English word "Search" and the Chinese word "Share," and the open set icon detection module identifies the magnifying glass icon and the send icon in the shape of a paper airplane. After all elements are aligned by spatial coordinates, a structured set of interface element descriptions is generated.

[0065] The task planning unit inputs user commands, a set of structured interface element descriptions, and screen images into the large language model, and calls a social semantic mapping library for collaborative reasoning. In the mapping library, the "search" function is associated with the semantic tag of a magnifying glass icon and multilingual keywords, while the "share" function is associated with the send icon features and textual expressions such as "Share". The model, combined with the distribution of interface elements, generates a sequence of atomic operations including clicking the search area, entering a friend's name, selecting a search result, locating the target post's share button, specifying the recipient, and confirming sending.

[0066] When the action execution unit performs the first operation, it matches the coordinates of the magnifying glass icon in the structured interface element description set based on the semantic features of the "search" function in the mapping library, generates a touch click command, and triggers terminal execution. After the interface jumps to the search page, the system automatically recaptures the image, completes the input of the friend's name, and waits for loading. When processing the sharing task on the friend's homepage interface, due to the existence of multiple similar interactive icons, the system initiates a progressive positioning strategy: first, it attempts to filter candidate elements through semantic matching; if this fails, it performs a sliding window-style spatial neighborhood search in the interface area; finally, it performs cross-validation by combining the spatial topological relationship between the icon and the text of adjacent posts to lock the coordinates of the share button corresponding to the target post.

[0067] In the subsequent process, the system identifies the recipient's input area and confirmation control in the sharing dialog box interface, and executes the text filling and confirmation commands sequentially according to the atomic operation sequence. After each operation is completed, the visual perception unit captures the updated interface state in real time, the task planning unit dynamically verifies the execution progress, and the action execution unit continues to advance the next operation in the sequence. When all atomic operations are executed in sequence and the system determines that the interactive task objective has been achieved, the process naturally terminates, fully realizing the automated interactive execution of complex social tasks under cross-language interfaces.

Claims

1. A cross-language social intelligent interaction method based on multimodal perception, characterized in that, Includes the following steps: S1. Obtain the screen image of the current social application interface, and simultaneously perform text recognition and icon recognition processing on the screen image. Extract the text elements and their spatial location information, and the icon elements and their spatial location information in the interface respectively, and fuse them to generate a structured interface element description set containing element type, content and coordinate information. S2. Receive the natural language task instruction input by the user, combine the structured interface element description set, the screen image and the preset social semantic mapping library, call the large language model to perform semantic reasoning, and parse the natural language task instruction into an ordered sequence of atomic operations. S3. Based on the current atomic operation in the atomic operation sequence, and based on the structured interface element description set, locate the corresponding target interactive element, and generate a device operation instruction matching the target interactive element for execution; S4. After the device operation command is executed, return to step S1 to obtain the updated social application interface status, and continue to execute steps S1 to S3 until the interactive task corresponding to the natural language task command is completed.

2. The cross-language social intelligent interaction method according to claim 1, characterized in that: In step S1, the text recognition is achieved by a multilingual optical character recognition module that runs in parallel. The multilingual optical character recognition module is configured to synchronously process the text content in multiple languages ​​contained in the interface of the screen image. The icon recognition is achieved through an open set icon detection module, which is built based on the Grounding DINO object detection model and is specifically optimized and trained using a typical icon sample dataset from social platforms. The process of fusing and generating the structured interface element description set includes: aligning and integrating the recognition results of text elements and icon elements according to spatial coordinates, and labeling each element with an element type identifier, content text or visual feature description, and bounding box coordinate information.

3. The cross-language social intelligent interaction method according to claim 1, characterized in that: In step S2, the social semantic mapping library is a pre-built static knowledge base that stores the mapping relationship between social platform functional semantics and multimodal interface features. The mapping relationship includes functional semantic fields, multilingual text keyword fields associated with the functional semantics, and corresponding visual icon feature fields. The visual icon feature fields include icon category labels and icon visual description information, and the multilingual text keyword fields include the text expression of the same function in different language version interfaces.

4. The cross-language social intelligent interaction method according to claim 1, characterized in that: In step S3, the location of the target interactive element adopts a progressive location strategy. First, based on the social semantic mapping library, semantic matching is performed on the elements in the structured interface element description set to determine the target interactive element. If semantic matching fails to locate the element, a sliding window geometric spatial neighborhood search is performed within the interface area covered by the structured interface element description set. If the sliding window geometric spatial neighborhood search still fails to locate the target interactive element, cross-validation is performed by combining the spatial positional relationship between the icon element and its neighboring text elements and the contextual semantics to finally determine the coordinate position of the target interactive element.

5. The cross-language social intelligent interaction method according to claim 1, characterized in that: In steps S2 and S3, the atomic operations in the atomic operation sequence include click operation, swipe operation, text input operation, return operation, and wait operation; The click operation corresponds to the generation of a touch click instruction that acts on the coordinate position of the target interactive element; The sliding operation corresponds to the generation of a trajectory sliding command that defines the start and end coordinates; The text input operation corresponds to generating an input instruction to fill a specified input box with preset text content; The return operation corresponds to the generation of a system return instruction that triggers the interface-level rollback; The waiting operation corresponds to the generation of an execution pause instruction for a set duration.

6. A cross-language social intelligent interaction system based on multimodal perception that implements the method of claim 1, characterized in that, Includes the following functional modules: The visual perception unit is configured to capture the screen image of the current social application interface in real time, and perform text recognition processing and icon recognition processing on the screen image in parallel, respectively extracting the content and spatial coordinate information of each text element in the interface, and the visual feature identifier and spatial coordinate information of each icon element, and fusing the extraction results to generate a structured interface element description set. The task planning unit is communicatively connected to the visual perception unit and is configured to receive natural language task instructions input by the user, call a large language model, combine the structured interface element description set, the screen image and the preset social semantic mapping library, perform multimodal semantic reasoning, and output an ordered atomic operation sequence corresponding to the natural language task instructions. The action execution unit is communicatively connected to the task planning unit and the visual perception unit. It is configured to parse the atomic operation to be executed in the atomic operation sequence, locate the coordinate position of the target interactive element according to the structured interface element description set, generate a device operation instruction that matches the atomic operation type and the target interactive element, and control the target terminal device to execute the device operation instruction. In this process, after the action execution unit completes the execution of the current device operation command, it sends an interface update trigger signal to the visual perception unit, driving the visual perception unit to recapture the updated social application interface image, thereby repeating the process of capturing the screen image and executing the device operation command in a loop.

7. The cross-language social intelligent interaction system according to claim 6, characterized in that: The visual perception unit includes a multilingual optical character recognition module and an open set icon detection module; The multilingual optical character recognition module is configured to simultaneously perform recognition processing of multiple language texts on the screen image, and output the content text, language type identifier and bounding box coordinates of each text element; The open set icon detection module is built on the Grounding DINO object detection architecture and uses a typical icon sample dataset from social platforms for parameter optimization training. It is configured to output the category label, confidence score, and bounding box coordinates of each icon element. The visual perception unit is configured to align and integrate the two types of recognition results, text and icons, based on spatial coordinates to generate a structured set of interface element descriptions that includes element type, content description, visual feature identifier, and precise coordinates.

8. The cross-language social intelligent interaction system according to claim 6, characterized in that: The social semantic mapping library is stored in a storage module that is communicatively connected to the task planning unit; The social semantic mapping library contains multiple mapping entries, each of which has a functional semantic field, a multilingual text keyword field, and a visual icon feature field; The multilingual text keyword field records the set of text keywords corresponding to the same functional semantics in different language version interfaces; The visual icon feature field records the icon category label, icon visual description text, and typical icon sample feature vector that are associated with the semantic meaning of the function.

9. The cross-language social intelligent interaction system according to claim 6, characterized in that: The action execution unit is configured with an element positioning submodule, which is used to execute a progressive positioning process: semantically matching the elements in the structured interface element description set according to the social semantic mapping library to determine the target interactive element; If the semantic match fails, a sliding window mechanism is initiated within the interface area covered by the structured interface element description set to perform a geometric spatial neighborhood search with a preset step size and window size. If the neighborhood search still fails to locate the target element, the context verification module is invoked to perform cross-verification by combining the spatial relative position, arrangement order, and semantic relevance of the icon element and its neighboring text elements, and outputs the final coordinate position of the target interactive element.

10. The cross-language social intelligent interaction system according to claim 9, characterized in that: The action execution unit is also configured with an instruction generation submodule, which is used to generate corresponding device operation instructions according to the atomic operation type, as follows: When the atomic operation is a click operation, a touch click instruction containing the target coordinate position is generated; When the atomic operation is a sliding operation, a trajectory sliding instruction containing the start coordinates and the end coordinates is generated; When the atomic operation is a text input operation, a text fill instruction containing the target input box identifier and the text to be input is generated; When the atomic operation is a return operation, a system-level interface rollback instruction is generated; When the atomic operation is a waiting operation, an execution pause instruction containing a preset delay duration parameter is generated.

Images