Information processing method, neural network model, instruction execution method and electronic equipment

Abstract

The invention discloses an information processing method, a neural network model, an instruction execution method and electronic equipment, and the information processing method comprises the steps: obtaining a to-be-processed image, and enabling the to-be-processed image to be matched with a user instruction inputted to an intelligent agent by a user; performing multi-scale visual feature extraction on the to-be-processed image to obtain multi-scale visual features; performing fusion processing on the visual features of the plurality of scales to obtain visual lexical elements; and determining positioning information of a target element corresponding to the user instruction in the to-be-processed image based on the visual lexical elements and the user instruction.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and more specifically to an information processing method, a neural network model, an instruction execution method, and an electronic device. Background Technology

[0002] Visual multimodal large language models (MLLMs) play an important role in intelligent agents. When using MLLMs to locate target elements, the image to be processed is converted into a large number of visual tiles, which increases the processing power and affects the target element localization process, reducing the efficiency and accuracy of element localization. Summary of the Invention

[0003] In view of the above, this application provides the following technical solution:

[0004] An information processing method, comprising:

[0005] Acquire an image to be processed, which matches the user command input to the intelligent agent;

[0006] Multi-scale visual feature extraction is performed on the image to be processed to obtain visual features at multiple scales;

[0007] Visual features at multiple scales are fused to obtain visual lexical units;

[0008] Based on the visual lexical units and the user instruction, the location information of the target element in the image to be processed corresponding to the user instruction is determined.

[0009] Optionally, the fusion processing of visual features at multiple scales to obtain visual lexical units includes:

[0010] The visual features of at least one first scale among the plurality of scales are adjusted such that the spatial size of the adjusted visual features matches that of a visual feature of another second scale.

[0011] The adjusted visual features corresponding to the first scale are fused with the visual features of the second scale to obtain multiple first fused features corresponding to different scales before fusion.

[0012] The visual lexical units are obtained by fusing multiple of the first fusion features.

[0013] Optionally, fusing multiple first fusion features to obtain the visual lexical units includes:

[0014] At least one high-scale feature among multiple first fusion features is fused with at least one low-scale feature to obtain a second fusion feature; the second number of feature tensors of the second fusion feature is less than the first number of feature tensors of the first fusion feature.

[0015] The second fusion feature is then spliced ​​together to obtain the third fusion feature;

[0016] The visual lexical units are determined based on the third fusion feature.

[0017] Optionally, determining the location information of the target element in the image to be processed corresponding to the user instruction based on the visual lexical units and the user instruction includes:

[0018] The user instruction is converted into text lexical units, wherein the feature dimensions of the text lexical units match the feature dimensions of the visual lexical units;

[0019] In the case of cross-modal feature alignment between the text lexical units and the visual lexical units, visual context information matching the text lexical units is extracted from the visual lexical units.

[0020] Based on the visual context information, the location information of the target element in the image to be processed is determined.

[0021] Optionally, it also includes:

[0022] Based on the location information of the target element and the user instruction, a target operation instruction is generated so that the intelligent agent executes the target operation instruction to perform operation on the target element in the interactive interface.

[0023] A neural network model, comprising:

[0024] The feature extraction module is used to extract visual features at multiple scales from the image to be processed, thereby obtaining visual features at multiple scales.

[0025] The fusion module is used to fuse visual features from multiple scales to obtain visual lexical units;

[0026] The positioning module is used to determine the positioning information of the target element in the image to be processed that corresponds to the user instruction based on the user instruction and the visual lexical units.

[0027] Optionally, the fusion module includes:

[0028] The first fusion submodule is used to adjust the visual features of at least one first scale among the multiple scales so that the spatial size of the adjusted visual features matches the visual features of another second scale; and to fuse the adjusted visual features corresponding to the first scale with the visual features of the second scale to obtain multiple first fusion features corresponding to different scales before fusion.

[0029] The second fusion submodule is used to fuse multiple first fusion features to obtain visual lexical units.

[0030] Optionally, the second fusion submodule includes:

[0031] A fusion unit is used to fuse at least one high-scale feature with at least one low-scale feature among a plurality of first fusion features to obtain a second fusion feature; the second number of feature tensors of the second fusion feature is less than the first number of feature tensors of the first fusion feature;

[0032] The splicing unit is used to splice the second fusion feature to obtain the third fusion feature;

[0033] The determining unit is used to determine the visual lexical based on the third fusion feature.

[0034] An instruction execution method for an intelligent agent, comprising:

[0035] Receive user instructions and obtain the image to be processed corresponding to the interactive interface based on the user instructions;

[0036] The neural network model is invoked to perform the following processing: multi-scale visual feature extraction is performed on the image to be processed to obtain visual features at multiple scales; the visual features at each scale are fused to obtain visual words; based on the visual words and the user instruction, the location information of the target element in the image to be processed corresponding to the user instruction is determined;

[0037] Obtain the positioning information output by the neural network model, and perform a target operation corresponding to the user command on the target element in the interactive interface.

[0038] An electronic device, comprising:

[0039] Memory, used to store computer programs;

[0040] A processor, configured to execute the computer program to achieve:

[0041] Acquire an image to be processed, which matches the user command input to the intelligent agent;

[0042] Multi-scale visual feature extraction is performed on the image to be processed to obtain visual features at multiple scales; the visual features at multiple scales are fused to obtain visual lexical units.

[0043] Based on the visual lexical units and the user instruction, the location information of the target element in the image to be processed corresponding to the user instruction is determined. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 A flowchart illustrating an information processing method provided in an embodiment of this application;

[0046] Figure 2 A schematic diagram of the structure of a neural network model provided in an embodiment of this application.

[0047] Figure 3 A schematic diagram of an MLLM architecture suitable for element localization in an intelligent agent is provided for embodiments of this application;

[0048] Figure 4 A schematic diagram of the optimized hypernetic network structure provided in the embodiments of this application;

[0049] Figure 5 This is a schematic diagram illustrating the construction and training of a network based on search results, as provided in an embodiment of this application.

[0050] Figure 6 A schematic diagram of an inference architecture for agent element localization provided in an embodiment of this application;

[0051] Figure 7 This is a flowchart illustrating an instruction execution method for an intelligent agent, provided as an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0053] The terms "first" and "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units may include steps or units not listed, but may include steps or units not listed.

[0054] This application provides an information processing method, a neural network model, an instruction execution method, and an electronic device, which can be applied to various intelligent agent element location and interaction control scenarios, such as terminal device interaction, industrial inspection, autonomous driving, smart home, and smart office applications. For example, in automated testing, the intelligent agent needs to accurately locate and operate the target according to the user's instruction to click the login button. This application can acquire the image to be processed that matches the user's instruction, and generate visual lexical units that can comprehensively represent the details and semantics of the image by performing multi-scale visual feature extraction and fusion on the image. Then, based on the visual lexical units and the user's instruction, the location information of the target element can be determined. This can effectively compress the amount of visual feature data, improve the processing efficiency and response speed of the large language model, and enhance the location accuracy and instruction understanding ability of small targets and interactive elements in complex scenes through multi-scale feature fusion and cross-modal semantic alignment.

[0055] See Figure 1 This is a flowchart illustrating an information processing method provided in an embodiment of this application. The method may include the following steps:

[0056] S101. Obtain the image to be processed.

[0057] S102. Perform multi-scale visual feature extraction on the image to be processed to obtain visual features at multiple scales.

[0058] S103. The visual features at multiple scales are fused to obtain visual lexical units.

[0059] S104. Based on visual lexical units and user instructions, determine the location information of the target element in the image to be processed that corresponds to the user instructions.

[0060] In this embodiment, the image to be processed in step S101 is matched with the user command input to the agent, which can represent a screenshot of the visual information corresponding to the agent's current interactive interface, such as the potential interactive element pointed to by the user command. The agent represents a software or hardware entity that can perceive the environment, make decisions, and autonomously perform operations to complete a specific goal, such as an automated program or robot system that operates graphical user interface (GUI) elements by simulating user clicks, inputs, and other interactive methods.

[0061] For example, after receiving a user command, the intelligent agent acquires image data of the interactive interface through an image acquisition module (such as the screenshot function module of the terminal device, camera, etc.). In remote interaction scenarios, real-time images of the interactive interface can be acquired via network transmission. This process needs to ensure that the image resolution meets the element localization requirements and matches the scene of the user command (e.g., if the user command is "send message," the image to be processed can be the input interface of a chat application). This allows for accurate acquisition of visual data matching the user command, providing reliable input for subsequent feature extraction and localization, and avoiding localization failures due to image-compatibility with the command.

[0062] In step S102, multi-scale visual feature extraction is performed on the image to be processed to obtain visual features at multiple scales. Multi-scale refers to multiple scales, where scale can represent the magnification of the observed image or the size of the region described by the image features. Multi-scale visual features refer to a set of features describing the same image at different levels of abstraction and resolution. Specifically, multi-scale visual features refer to feature data of different spatial resolutions extracted from the image to be processed, including low-scale (global context features), mid-scale (local structural features), and high-scale (detail texture features), thus covering target elements of different sizes (especially small targets). This step can be implemented using a deep neural network. During forward propagation, the different depths of the network naturally generate feature maps with different receptive fields and levels of abstraction, thus obtaining multi-scale visual features. This can be achieved using a pre-trained Convolutional Neural Network (CNN) backbone, such as ResNet (which enables the network to learn depths of hundreds or even thousands of layers by introducing residual blocks of shortcut connections) or VGG (which improves performance by simply stacking deeper convolutional layers, but its training difficulty increases with depth).

[0063] After obtaining visual features at multiple scales, step S103 is performed to fuse these features to obtain visual lexical units. Visual lexical units are feature representations obtained by fusing multi-scale visual features, capable of being understood by MLLM (Multimodal Large Scale Language Model), and retain key localization information. For example, visual features at different scales can be adapted and adjusted first, and then multi-scale information can be integrated through feature fusion processing methods such as feature concatenation and weighted summation. Furthermore, a searchable projection module can be combined to map the fused features into visual lexical units with uniform dimensions. For instance, the output channels of the projection module should not exceed one-third of the number of channels in the fused features, thus compressing the number of visual lexical units.

[0064] In step S104, based on visual lexical units and user commands, the location information of the target element in the image to be processed corresponding to the user command is determined. This location information refers to the coordinate position (such as pixel coordinates, relative coordinates), size information, etc., of the target element in the image to be processed, which can uniquely identify the spatial location of the target element. Then, the user command can be converted into text lexical units, and the visual lexical units and text lexical units are aligned across modal features. Based on the aligned features, visual context information that semantically matches the user command is extracted from the visual lexical units. Combining the spatial association between the visual context information and multi-scale features, the location information of the target element is calculated and output to the agent.

[0065] For example, taking the application launch command of a mobile phone smart assistant as an example, when a user inputs a voice command to the smart assistant, such as "open the image processing application," the smart assistant will perform the following operations: using the phone's screen capture function, it will capture a real-time image of the phone's desktop, which contains icons of various applications. Then, the smart assistant will call the corresponding network model to extract visual features at three scales from the desktop image: low-scale (e.g., global desktop layout), mid-scale (e.g., application icon area), and high-scale (e.g., image processing application detail texture). Then, it will fuse these three scale features, such as adjusting the size of the high-scale feature to match the mid-scale feature, and generate corresponding visual lexical units. The "open the image processing application" command will be converted into text lexical units, cross-modal alignment will be performed, the visual context information corresponding to the image processing application icon will be extracted, the pixel coordinates of the image processing image in the desktop image will be calculated, and the location information will be output so that the smart assistant can open the image processing application based on the location information.

[0066] This application provides an information processing method that acquires an image to be processed that matches the user's input to an intelligent agent. Then, it performs multi-scale visual feature extraction on the image to be processed, obtaining visual features at multiple scales. These multi-scale visual features are then fused to obtain visual lexical units. Through multi-scale visual feature extraction and fusion, comprehensive visual information containing details, structure, and semantics is efficiently compressed into low-dimensional visual lexical units. Based on cross-modal understanding with user commands, localization information is determined. Front-end feature optimization reduces the amount of data input to a large language model, thereby significantly improving the inference efficiency and response speed of the intelligent agent. Simultaneously, multi-scale features ensure fine-grained perception of targets, while user commands enhance semantic understanding. Together, they achieve accurate and robust localization of multi-scale targets, especially small targets, in the interactive interface, effectively solving the problems of large localization errors and high computational requirements.

[0067] The relevant technical features of the information processing method in the embodiments of this application will be described below in conjunction with the corresponding application scenarios.

[0068] In this embodiment, the problem of information loss due to mismatch in the spatial dimensions of features at different scales can be solved by adjusting the scale and performing step-by-step fusion, thereby further improving the quality of visual lexical units. Specifically, the process of fusing visual features at multiple scales to obtain visual lexical units may include: adjusting at least one visual feature at a first scale so that the spatial dimensions of the adjusted visual feature match those of another visual feature at a second scale; fusing the adjusted visual feature at the first scale with the visual feature at the second scale to obtain multiple first fused features corresponding to different scales before fusion; and fusing the multiple first fused features to obtain visual lexical units.

[0069] The first and second scales refer to any two different scales among multiple scales. For example, the first scale might be a high-scale (e.g., high-resolution) feature, and the second scale a medium-scale (e.g., medium-resolution) feature, or the first scale might be a low-scale feature, and the second scale a medium-scale feature. Spatial size matching of visual features means that the width, height, and pixel dimensions of the features are consistent, ensuring information compatibility during feature fusion. For example, interpolation methods (e.g., bilinear interpolation) or convolutional upsampling / downsampling techniques can be used to adjust the size of visual features at the first scale. If the spatial size of the first scale is larger than the second scale, the size can be reduced by convolutional downsampling (e.g., a 3×3 convolution kernel with a stride of 2); if the spatial size of the first scale is smaller than the second scale, the size can be increased by bilinear interpolation upsampling, preserving key feature information during the adjustment process. For example, suppose the feature spatial sizes of the three scales are: low scale (64×64), medium scale (128×128), and high scale (256×256), and the medium scale is selected as the second scale, with the first scale being both low and high scales. Low-scale features are upsampled using bilinear interpolation to a size of 128×128; high-scale features are downsampled using convolution to a size of 128×128, ensuring that both match the second-scale dimensions. This solves the fusion problem caused by inconsistent spatial dimensions of features at different scales and avoids the loss of key information during the fusion process.

[0070] Then, the adjusted visual features corresponding to the first scale are fused with the visual features of the second scale to obtain multiple first fused features. That is, the first fused feature is the feature resulting from the fusion of a single first-scale feature and a second-scale feature, and each first fused feature retains the core information of the corresponding original scale (i.e., the first scale) and the correlation information of the second scale. For example, a weighted summation fusion method can be used, assigning different weights to the adjusted first-scale visual features and the second-scale visual features. These weights can be learned through the supernet training process, and the weighted summation yields the first fused feature. If multiple first scales exist (such as low-scale and high-scale), they are fused with the second scale separately to generate multiple first fused features. For example, the adjusted low-scale features (128×128) and mid-scale features (128×128) are weighted and summed with weights of 0.3 and 0.7 to obtain the corresponding low-scale first fusion feature F1; the adjusted high-scale features (128×128) and mid-scale features (128×128) are weighted and summed with weights of 0.6 and 0.4 to obtain the corresponding high-scale first fusion feature F2, for a total of two first fusion features. This achieves preliminary fusion of features at different scales, preserves the correlation information between each scale and related scales, enhances the representational ability of the features, and provides high-quality input for subsequent deep fusion. Then, the multiple first fusion features are concatenated by channel dimension, for example, the number of channels of the concatenated feature is the number of channels of a single first fusion feature multiplied by the number of first fusion features; the concatenated features are input into the corresponding projection module and mapped to visual units with uniform dimensions. This embodiment solves the problem of information loss when fusing features at different scales by adjusting and fusing them step by step; and improves the information density of visual words by using a combination of weighted summation and channel splicing fusion strategy, providing richer feature support for subsequent localization.

[0071] Correspondingly, in this embodiment of the application, the process of fusing multiple first fusion features to obtain visual lexical units may include: fusing at least one high-scale feature with at least one low-scale feature among the multiple first fusion features to obtain a second fusion feature; concatenating the second fusion feature to obtain a third fusion feature; and determining visual lexical units based on the third fusion feature. Wherein, the second number of feature tensors of the second fusion feature is less than the first number of feature tensors of the first fusion feature.

[0072] The high-scale first fusion feature corresponds to the original high-scale visual features, preserving the detailed information of the image to be processed. The low-scale first fusion feature corresponds to the original low-scale visual features, preserving the global information of the image to be processed. The number of feature tensors refers to the dimensionality of the feature data. The second fusion feature is obtained through the first fusion feature, achieving feature compression. For example, the first fusion feature includes a high-scale fusion feature F2 (tensor count 1024) and a low-scale fusion feature F1 (tensor count 1024). The scratch detail information in F2 is extracted and injected into the global part information of F1. Through 1×1 convolution dimensionality reduction, the second fusion feature F3 is obtained, with a tensor count of 1536 (less than the total tensor count of F1 and F2, 2048). This achieves deep fusion of global and detailed information. By compressing the tensor count through feature dimensionality reduction, the computational power requirement can be reduced. The third fusion feature is the integrated feature after concatenating the second fusion features by channel dimension. Its channel count can be the sum of the channel counts of all second fusion features, which can concentrate all the fused cross-scale information. For example, multiple second fusion features are concatenated along the channel dimension, maintaining the spatial size of the features while increasing the number of channel dimensions. If multiple second fusion features exist (e.g., fusion features for different regions), they are sequentially concatenated into a unified feature tensor. The third fusion feature can then be input into the projector. The number of output channels for each fully connected (FC) layer of the projector is a flexible searchable variable, determined through the supernet training process to achieve the optimal number of channels. The number of output channels for the last FC layer is set to no more than 1 / 3 of the number of channels in the third fusion feature, ensuring visual word compression. The projector weights are optimized using an once-for-all weight-sharing search algorithm. A balance between the number of visual words and localization accuracy can be achieved through a searchable projector module.

[0073] In some embodiments of this application, a process of text lexical conversion, cross-modal feature alignment, and visual context extraction can be used to achieve accurate semantic matching between user commands and visual features, further improving the accuracy of localization. Correspondingly, the process of determining the localization information of the target element in the image to be processed, based on visual lexical units and user commands, may include: converting the user command into text lexical units; extracting visual context information matching the text lexical units from the visual lexical units after cross-modal feature alignment between the text lexical units and visual lexical units; and determining the localization information of the target element in the image to be processed based on the visual context information.

[0074] In this process, the feature dimensions of text words are matched with those of visual words. Text words are vector representations of user commands, and feature dimensions refer to the dimensions of the vector, such as 80-dimensional or 128-dimensional. Dimensional matching between text and visual words enables cross-modal computation. A text encoding module (such as the encoding layers of BERT or GPT) within a multimodal large language model architecture can be used to convert user commands (text or speech-to-text) into text vectors. A linear mapping layer adjusts the dimensions of the text vectors to match the feature dimensions of the visual words; for example, if the visual words are 80-dimensional, the text words are also adjusted to 80-dimensional. The weights of the linear mapping layer are optimized using training data to ensure that semantic information is not lost during the dimensionality conversion. Then, cross-modal feature alignment is performed between text and visual words. Cross-modal alignment refers to training to ensure that text and visual words have semantic relevance in the same feature space. Then, after cross-modal alignment, the corresponding visual context information is extracted, that is, visual feature association information that matches the semantics of the user command is obtained. For example, if the user command is "locate speed limit sign", the corresponding visual context information can be the association data of features such as the shape, color, and text of the speed limit sign. Then, the specific location information of the target element is inferred by using the spatial association between the visual context information and the original multi-scale features. For example, a spatial mapping relationship between visual words and original multi-scale features can be established (such as recording the correspondence between words and feature pixels during training). Based on the word weights in the visual context information, the original feature regions corresponding to key visual words are determined. Combining the spatial size and coordinate mapping of multi-scale features, the pixel coordinates and size information of the target element in the image to be processed are calculated, and then the positioning information is obtained. When outputting the positioning information, it can be converted into relative coordinates according to the needs of the agent, such as the proportion of the target element relative to the upper left corner of the image.

[0075] In this embodiment, the information processing method further includes: generating a target operation instruction based on the location information of the target element and the user instruction, so that the intelligent agent executes the target operation instruction to perform operation on the target element in the interactive interface. The target operation instruction is a specific operation command that the intelligent agent can execute, including information such as operation type (e.g., click, swipe, input), operation object (target element), and operation parameters (e.g., swipe distance, input content), and is semantically consistent with the user instruction. The intelligent agent executing the operation means that the intelligent agent performs a preset operation on the target element in the interactive interface through a hardware or software interface, and the operation result is consistent with the user instruction expectation. This enables automatic conversion from location information to operation instructions, adapting to the intelligent agent's execution system.

[0076] See Figure 2This application also provides a schematic diagram of a neural network model, which can be applied to the element location processing in the above embodiments. The neural network model includes:

[0077] The feature extraction module 201 is used to extract multi-scale visual features from the image to be processed, thereby obtaining visual features at multiple scales.

[0078] The fusion module 202 is used to fuse visual features at multiple scales to obtain visual lexical units.

[0079] The positioning module 203 is used to determine the positioning information of the target element in the image to be processed that corresponds to the user command based on the user command and visual lexical units.

[0080] In some embodiments of this application, the fusion module in the neural network model includes:

[0081] The first fusion submodule is used to adjust the visual features of at least one first scale among multiple scales so that the spatial size of the adjusted visual features matches the visual features of another second scale; and to fuse the adjusted visual features corresponding to the first scale with the visual features of the second scale to obtain multiple first fusion features corresponding to different scales before fusion. The second fusion submodule is used to fuse the multiple first fusion features to obtain visual lexical units.

[0082] Furthermore, the second fusion submodule includes a fusion unit, a splicing unit, and a determination unit. The fusion unit fuses at least one high-scale feature with at least one low-scale feature from a plurality of first fusion features to obtain a second fusion feature; the second number of feature tensors in the second fusion feature is less than the first number of feature tensors in the first fusion feature. The splicing unit splices the second fusion feature to obtain a third fusion feature. The determination unit determines visual lexical units based on the third fusion feature.

[0083] The following explanation uses this neural network model as an example of an MLLM model architecture. This MLLM architecture represents a multimodal large language model architecture, which integrates a visual encoder, a multi-scale feature fusion module, and a projector that works in conjunction with the large language model (such as...). Figure 3The MLLM model architecture shown includes a visual lexical generation module. The visual encoder extracts multi-level features from the image, using different depths of a convolutional neural network to extract high-resolution detail features, mid-scale structural features, and low-resolution semantic features, providing a comprehensive visual representation foundation for subsequent multi-scale fusion. The multi-scale feature fusion module integrates visual features at different scales. Through upsampling and feature addition operations in a feature pyramid network, it propagates and enhances high-level semantic information to low-level features, resulting in output features with both rich detail and strong semantic understanding capabilities. The projector compresses and maps the fused high-dimensional visual features into low-dimensional semantic representations. A searchable multilayer perceptron converts the visual information into visual lexical units understandable by a large language model, significantly reducing data dimensionality while preserving key localization information and improving the efficiency of subsequent processing.

[0084] See Figure 3 This illustrates a schematic diagram of an MLLM architecture suitable for element localization in an intelligent agent, provided by an embodiment of this application. Figure 3 The image to be processed is input into the multi-scale feature extraction module consisting of Block 1 to Block 8 (i.e., the corresponding...). Figure 2 (feature extraction module in the middle), where, Figure 3 The Block shown is a feature extraction unit in a neural network model, serving as the fundamental structure for multi-scale visual feature extraction. Each Block contains several convolutional layers and activation functions, forming a multi-level feature representation from local details to global semantics through progressive feature transformations. Specifically, shallow Blocks primarily extract basic visual features such as edges and textures, mid-level Blocks are responsible for combining basic features to form local structural information, and deep Blocks capture high-level semantic features through a larger receptive field. Multiple Blocks are concatenated to form a complete forward propagation path, and by progressively decreasing the feature map size and gradually increasing the number of channels, a multi-scale feature representation system with rich semantic layers is constructed.

[0085] For example, in Figure 3After the image to be processed is processed by each block in sequence, three different scales of visual features are obtained: "H / 8, W / 8" (meaning the spatial size of the feature map is 1 / 8 of the height and width of the original image, that is, the resolution of the feature map is reduced by 8 times compared to the input image to be processed), "H / 16, W / 16" (meaning the spatial size of the feature map is 1 / 16 of the height and width of the original image, that is, the resolution of the feature map is reduced by 16 times compared to the input image to be processed), and "H / 32, W / 32" (meaning the spatial size of the feature map is 1 / 32 of the height and width of the original image, that is, the resolution of the feature map is reduced by 32 times compared to the input image to be processed). Among them, "H / 8, W / 8" represents high-resolution scale features, such as preserving the detailed texture features of the icon; "H / 16, W / 16" represents medium-resolution scale features, such as characterizing the local structure of the icon; and "H / 32, W / 32" represents low-resolution scale features, such as the global layout features of the entire interface. These multi-scale visual features are then input into a super network (referred to as the super network, corresponding to...). Figure 2 The fusion module performs multi-scale feature fusion through operations such as 4x upsampling and addition, 2x upsampling and addition, 2x downsampling and addition, and 4x downsampling and addition. The fused features are input into the visual word generation module to obtain visual words. Finally, the visual words and the text words converted from user commands are input into the LLM (intermodal localization) module to output the location information of the target element.

[0086] In the embodiments of this application, the multi-scale feature fusion module (i.e. Figure 3The super network shown is implemented based on an improved PANet (Path Aggregation Network) architecture, which introduces a bidirectional path aggregation mechanism of bottom-up and top-down. Specifically, in the bottom-up path, the network fuses high-resolution features from the lower levels with high-level semantic features through progressive upsampling, preserving rich spatial detail information; in the top-down path, high-level semantic information is passed to lower-level features through downsampling operations, enhancing their feature representation capabilities. This bidirectional fusion mechanism ensures that each level in the feature pyramid contains sufficient semantic information and spatial detail. The LLM (Landing-Modal Localization) module in this MLLM architecture is built based on a large language model and optimized using parameter-efficient fine-tuning methods (such as LoRA), specifically designed to achieve deep semantic alignment between visual and language modalities. This module uses a cross-modal attention mechanism, treating text terms as queries and visual terms as keys and values, calculating semantic association weights to filter out the visual features most relevant to user commands. Based on the aligned multimodal features, this module uses the reasoning ability of a large language model to understand the user's intent and generates precise positioning information of the target element (such as bounding box coordinates or location markers) through a dedicated positioning output head, ultimately achieving accurate visual positioning under command guidance.

[0087] The MLLM architecture described above, applicable to element localization in intelligent agents, can be obtained through a systematic training process, which may include two main stages. The first stage involves training the multi-scale feature extraction module and the visual lexical generation module. The parameters of the feature extraction module can be fixed initially, and the fusion module and visual lexical generation module can be jointly trained using an element localization task dataset. During training, the visual lexical generation module acts as a resiliently searchable supernetwork, requiring sampling of subnetworks for training in each training epoch to evaluate the performance of different projector structures. Through this stage of training, the model can effectively fuse multi-scale features and generate high-quality visual lexical units, providing a foundation for subsequent accurate element localization. After completing the first stage of training, a second stage of training is performed to efficiently fine-tune the parameters of the large language model. The parameters of the feature fusion module and visual lexical generation module trained in the first stage can be fixed, and then efficient parameter fine-tuning methods can be used. For example, a LoRA (Low-Rank Adaptation) structure can be added to the large language model, and instruction alignment can be achieved by updating only the parameters of the LoRA part. This stage uses the same element localization task dataset, focusing on optimizing the large language model's ability to understand visual lexical units and respond to localization commands, so that the large language model can specialize in element localization tasks while maintaining its original language understanding capabilities.

[0088] Throughout the training process of the neural network model, neural network architecture search technology was employed to optimize the model structure. Specifically, a supernetwork containing multiple candidate projector structures was constructed, and an architecture search algorithm based on weight sharing was used for architecture search. The supernetwork was trained on an element localization task dataset. Based on the architecture parameters obtained after training, the candidate operation with the highest value for each layer of the visual word generation module was selected from the supernetwork to form the final target structure. In this way, the system can automatically obtain a model architecture that achieves an optimal balance between computational efficiency and localization accuracy.

[0089] See Figure 4 It shows a schematic diagram of the optimized hypernetwork structure provided in the embodiments of this application, that is, for Figure 3 The optimized structure of the supernetwork is derived from the optimal subnetwork extracted from the supernetwork through a neural network search architecture. It includes optimized multi-scale feature fusion paths and... Figure 3 Compared to the basic architecture of hypernets in the original system, the optimized hypernet can reduce computational complexity while maintaining positioning accuracy.

[0090] See Figure 5 The illustration shows a schematic diagram of a network built and trained based on search results provided in this application embodiment. The image to be processed sequentially passes through a multi-scale feature extraction module, an optimized super network, a multi-scale word block packer, a searchable projector, and an LLM cross-modal localization module, and finally outputs the localization information of the target element. At the same time, the generation efficiency of words is optimized by the elastic number of multi-layer perception in the projector. First, the image to be processed (such as a screenshot of a software interface) is sequentially processed through feature extraction from Block 1 to Block 8, outputting visual features at three scales. For example, Block 6 outputs scale features of "H / 8, W / 8" (meaning the spatial size of the feature map is 1 / 8 of the original image's height and width, i.e., the resolution of the feature map is reduced by 8 times compared to the input image to be processed, which can represent high-resolution visual features, such as preserving the details of target elements), Block 7 outputs scale features of "H / 16, W / 16" (meaning the spatial size of the feature map is 1 / 16 of the original image's height and width, i.e., the resolution of the feature map is reduced by 16 times compared to the input image to be processed, which can represent medium-resolution visual features, such as characterizing local regions), and Block 8 outputs scale features of "H / 32, W / 32" (meaning the spatial size of the feature map is 1 / 32 of the original image's height and width, i.e., the resolution of the feature map is reduced by 32 times compared to the input image to be processed, which can represent low-resolution visual features, such as features that cover the global layout). These three scale features are then input into the optimized hypernetwork, and through a pre-defined feature fusion operation, the adaptation and fusion of multi-scale features are achieved. Figure 5The optimized hypernetwork shown is the optimal subnetwork extracted from the hypernetwork through a neural network search architecture. It contains optimized multi-scale feature fusion paths, and its structure can be found in [reference needed]. Figure 4 The structure shown.

[0091] The three fused feature sets are then input into a multi-scale lexical packer, which is the core module for efficient compression and transformation of visual features. This multi-scale lexical packer uses a systematic processing flow to transform multi-scale visual features into semantic representations suitable for large language models. The multi-scale lexical packer includes a dimension reshaping module for converting feature maps of different scales into a unified serialization format; an information injection module that enhances the complementarity of detailed and semantic information through cross-scale feature interaction; a channel stitching module that integrates features while preserving the unique information of each scale; and a searchable projector that performs intelligent mapping from high-dimensional features to low-dimensional semantic space through an adjustable multilayer perception system. Specifically, the dimension reshaping module in the multi-scale lexical packer performs dimension reshaping on the three fused feature sets output by the hypernetwork, unifying the dimensional format of each feature. Then, the information injection module injects complementary information into features of different scales, such as injecting high-resolution detailed information into low-resolution global information, to enhance the feature localization and representation capabilities. Then, the injected multi-scale features are stitched together along the channel dimension using the channel stitching module, integrating all effective information to obtain high-dimensional features after channel stitching. Then, by using the flexible number of channels in the multilayer perception of the projector, i.e., the number of channels in each layer of perception can be dynamically adjusted, the high-dimensional features are compressed into visual units with uniform dimensions. Figure 5 The black modules in the middle represent different layers of multi-layer perception (the number of channels flexibly adapts to localization requirements), and then output visual lexical units, which are input into the LLM (cross-modal localization) module. This LLM module matches the semantic information of text lexical units and visual lexical units through cross-modal feature alignment (combined with low-rank adaptation technology), extracts the visual context corresponding to the target element, and finally outputs the localization information of the target element.

[0092] In the model training process described above, the element localization task dataset used for training includes sample images and corresponding target element localization annotations. Each training sample contains the interface element to be localized and its precise location information, ensuring that the model can learn accurate element localization capabilities. Through the above systematic training process, a MLLM architecture specifically optimized for agent element localization tasks was finally obtained. This training method ensures that the model has both powerful multi-scale feature processing capabilities and can efficiently generate visual lexical units suitable for understanding large language models, ultimately achieving accurate and efficient element localization and effectively solving the problems of inference latency and high computational requirements caused by an excessive number of visual lexical units.

[0093] See Figure 6 , Figure 6This diagram illustrates a reasoning architecture for agent element localization provided in an embodiment of this application. The processing steps are as follows: The image to be processed is first input into a multi-scale feature extraction module, generating low, medium, and high-scale visual features; the optimized multi-scale fusion module integrates the features, and then compresses them into visual words through an improved word packer and a searchable projector; simultaneously, user commands are converted into text words, which, along with the visual words, are input into a multimodal large-scale language model (MLLM) equipped with a low-rank adaptation (LoRA) structure; the model aligns cross-modal features, matches semantically related visual context information, and finally outputs the localization information of the target element, supporting the agent in performing corresponding operations. For details, please refer to [link to relevant documentation]. Figure 5 The corresponding description process will not be detailed here.

[0094] An embodiment of this application also provides an instruction execution method applied to an intelligent agent, see [link to relevant documentation]. Figure 7 The method may include the following steps:

[0095] S301. Receive user instructions and obtain the image to be processed corresponding to the interactive interface based on the user instructions.

[0096] S302, Call the neural network model to perform target processing.

[0097] Among them, the neural network model is Figure 2 The neural network model shown includes a feature extraction module, a fusion module, and a localization module. The target processing that can be performed by calling this neural network model includes: multi-scale visual feature extraction of the image to be processed, obtaining visual features at multiple scales; fusion processing of the visual features at each scale to obtain visual words; and determining the localization information of the target element in the image to be processed corresponding to the user command based on the visual words and the user command. It should be noted that the specific architecture and functional module description of this neural network model can be found in the aforementioned description. Figure 2 , Figure 3 , Figure 4 and Figure 5 The relevant descriptions will not be elaborated here.

[0098] S303. Obtain the positioning information output by the neural network model, and perform target operations on the target elements in the interactive interface corresponding to the user's instructions.

[0099] In this instruction execution method, user instructions are operation requests issued by the user to the intelligent agent (such as voice, text, or gesture commands), the interactive interface is the carrier for interaction between the intelligent agent and the user (such as a software interface or hardware control panel), and the image to be processed is the visual data of the interactive interface (such as a screenshot of the current interactive interface), matched with the user instruction scenario. The intelligent agent receives user instructions through input modules (microphone, keyboard, touchscreen), and voice instructions are converted into text instructions through speech recognition technology. The semantics of the user instructions are analyzed to determine the corresponding interactive interface (e.g., "open chat application" corresponds to the mobile phone desktop interface, "print document" corresponds to the printer operation panel). Real-time images of the interactive interface are acquired through image acquisition modules (screenshot, camera, industrial camera) to ensure that the image resolution and clarity meet the positioning requirements; if the interactive interface is a remote device (such as a cloud printer), real-time images are acquired through network transmission. The intelligent agent's processing module calls the deployed neural network model through the API interface, inputting the image to be processed and the user instructions. Then, the neural network model sequentially performs multi-scale feature extraction, feature fusion, and cross-modal alignment and localization processes, finally outputting the positioning information of the target element. The intelligent agent generates target operation instructions based on location information and user commands, and then selects the execution method according to the type of interactive interface. For example, the software interface executes the operation through the system API (such as the UI automation interface), while the hardware interface executes the operation by controlling the actuator (such as the touch module or motor) through the drive module.

[0100] Taking the execution of instructions in an intelligent office system as an example, when a user inputs the text instruction "Print current document" into the intelligent office system (i.e., the intelligent agent), the intelligent office system implements the following process: It receives the instruction, analyzes the scenario as a printer operation interface, and obtains a real-time image of the printer control panel (e.g., resolution 1024×768) via the network; it calls a neural network model, inputting the image and the instruction; the model extracts features at three scales, such as fusing them to generate 80 visual units, locates the "Print Start" button, and outputs coordinates (400, 300) to (450, 330); the intelligent office system's instruction generation module generates the target operation instruction: {Operation type: Click; Target coordinates: {"x1":400, "y1":300, "x2":450, "y2":330}; Parameters, etc.}; the intelligent office system sends the instruction to the printer via the network, controlling the printer's touch actuator to perform the click operation at the target coordinates; after the printer starts printing, it provides feedback with "Printing started" information, and the intelligent office system sends a text notification to the user.

[0101] It should be noted that the specific implementation of the execution process of the neural network model in the instruction execution method in this embodiment can be found in the corresponding content above, and will not be described in detail here.

[0102] In another embodiment of this application, a readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the information processing method and instruction execution method as described above.

[0103] In another embodiment of this application, an electronic device is also provided, the electronic device comprising:

[0104] Memory, used to store computer programs;

[0105] A processor, configured to execute the computer program to achieve:

[0106] Acquire an image to be processed, which matches the user command input to the intelligent agent;

[0107] Multi-scale visual feature extraction is performed on the image to be processed to obtain visual features at multiple scales; the visual features at multiple scales are fused to obtain visual lexical units.

[0108] Based on the visual lexical units and the user instruction, the location information of the target element in the image to be processed corresponding to the user instruction is determined.

[0109] It should be noted that the specific implementation of the processor in this embodiment can be referred to the corresponding content above, and will not be described in detail here.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0111] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0113] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An information processing method, comprising: Acquire an image to be processed, which matches the user command input to the intelligent agent; Multi-scale visual feature extraction is performed on the image to be processed to obtain visual features at multiple scales; Visual features at multiple scales are fused to obtain visual lexical units; Based on the visual lexical units and the user instruction, the location information of the target element in the image to be processed corresponding to the user instruction is determined.

2. The method according to claim 1, wherein fusing visual features at multiple scales to obtain visual lexical units includes: The visual features of at least one first scale among the plurality of scales are adjusted such that the spatial size of the adjusted visual features matches that of a visual feature of another second scale. The adjusted visual features corresponding to the first scale are fused with the visual features of the second scale to obtain multiple first fused features corresponding to different scales before fusion. The visual lexical units are obtained by fusing multiple of the first fusion features.

3. The method according to claim 2, wherein fusing multiple first fusion features to obtain the visual lexical units includes: At least one high-scale feature among multiple first fusion features is fused with at least one low-scale feature to obtain a second fusion feature; the second number of feature tensors of the second fusion feature is less than the first number of feature tensors of the first fusion feature. The second fusion feature is then spliced ​​together to obtain the third fusion feature; The visual lexical units are determined based on the third fusion feature.

4. The method according to claim 1, wherein determining the location information of the target element corresponding to the user instruction in the image to be processed based on the visual lexical and the user instruction includes: The user instruction is converted into text lexical units, wherein the feature dimensions of the text lexical units match the feature dimensions of the visual lexical units; In the case of cross-modal feature alignment between the text lexical units and the visual lexical units, visual context information matching the text lexical units is extracted from the visual lexical units. Based on the visual context information, the location information of the target element in the image to be processed is determined.

5. The method according to claim 1, further comprising: Based on the location information of the target element and the user instruction, a target operation instruction is generated so that the intelligent agent executes the target operation instruction to perform operation on the target element in the interactive interface.

6. A neural network model, comprising: The feature extraction module is used to extract multi-scale visual features from the image to be processed, and obtain visual features at multiple scales. The fusion module is used to fuse visual features from multiple scales to obtain visual lexical units; The positioning module is used to determine the positioning information of the target element in the image to be processed that corresponds to the user instruction based on the user instruction and the visual lexical units.

7. The neural network model according to claim 6, wherein the fusion module comprises: The first fusion submodule is used to adjust the visual features of at least one first scale among the multiple scales so that the spatial size of the adjusted visual features matches the visual features of another second scale; and to fuse the adjusted visual features corresponding to the first scale with the visual features of the second scale to obtain multiple first fusion features corresponding to different scales before fusion. The second fusion submodule is used to fuse multiple first fusion features to obtain visual lexical units.

8. The neural network model according to claim 7, wherein the second fusion submodule comprises: A fusion unit is used to fuse at least one high-scale feature with at least one low-scale feature among a plurality of first fusion features to obtain a second fusion feature; the second number of feature tensors of the second fusion feature is less than the first number of feature tensors of the first fusion feature; The splicing unit is used to splice the second fusion feature to obtain the third fusion feature; The determining unit is used to determine the visual lexical based on the third fusion feature.

9. An instruction execution method for an intelligent agent, comprising: Receive user instructions and obtain the image to be processed corresponding to the interactive interface based on the user instructions; The neural network model is invoked to perform the following processing: multi-scale visual feature extraction is performed on the image to be processed to obtain visual features at multiple scales; Visual features at each scale are fused to obtain visual lexical units; based on the visual lexical units and the user command, the location information of the target element in the image to be processed corresponding to the user command is determined; Obtain the positioning information output by the neural network model, and perform a target operation corresponding to the user command on the target element in the interactive interface.

10. An electronic device, comprising: Memory, used to store computer programs; A processor, configured to execute the computer program to achieve: Acquire an image to be processed, which matches the user command input to the intelligent agent; Multi-scale visual feature extraction is performed on the image to be processed to obtain visual features at multiple scales; Visual features at multiple scales are fused to obtain visual lexical units; Based on the visual lexical units and the user instruction, the location information of the target element in the image to be processed corresponding to the user instruction is determined.

Images