Data processing method and apparatus

Abstract

Embodiments of the present specification provide a data processing method and device, the data processing method comprising: determining a target image; recognizing a reference text object in the target image and obtaining an identified bounding box of the reference text object; inputting the target image, reference text corresponding to the reference text object and prompt text of a to-be-positioned object into an image processing model to obtain a reference bounding box of the reference text object and a predicted bounding box of the to-be-positioned object; obtaining a comparison result of the identified bounding box and the reference bounding box, and in a case where the comparison result meets a preset threshold, determining a target coordinate corresponding to the to-be-positioned object according to the predicted bounding box; and providing a reliable judgment basis for output of the image processing model by comparing the reference bounding box and the identified bounding box of the reference text object, so as to determine the target coordinate of the to-be-positioned object according to the reliable predicted bounding box, effectively suppress hallucination interference, and finally ensure accuracy and reliability of the target coordinate.

Description

Technical Field

[0001] This specification relates to the field of image object positioning technology, and in particular to a data processing method. One or more embodiments of this specification also relate to a data processing device, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Existing multimodal large models have powerful image and text understanding capabilities. When an image is uploaded to a multimodal large model, it can detect objects in the image and obtain their corresponding bounding box coordinates.

[0003] However, due to the illusion problem of multimodal large models, it cannot be guaranteed that the same result will be returned each time when multiple requests are made for objects in the same image. As a result, multimodal large models are not compatible and stable enough, and their effectiveness is greatly reduced when used for automated target localization of page images, resulting in inaccurate localization results. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising: Identify the target image; Identify reference text objects in the target image and obtain the bounding box of the reference text objects; The target image, the reference text corresponding to the reference text object, and the prompt text of the object to be located are input into the image processing model to obtain the reference bounding box of the reference text object and the predicted bounding box of the object to be located. Obtain the comparison result between the identified bounding box and the reference bounding box. If the comparison result meets a preset threshold, determine the target coordinates corresponding to the object to be located based on the predicted bounding box.

[0006] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising: The image determination module is configured to determine the target image; The reference recognition module is configured to recognize reference text objects in the target image and obtain the recognition bounding box of the reference text objects; The model prediction module is configured to input the target image, the reference text corresponding to the reference text object, and the prompt text of the object to be located into the image processing model to obtain the reference bounding box of the reference text object and the predicted bounding box of the object to be located. The coordinate determination module is configured to obtain the comparison result between the recognition bounding box and the reference bounding box, and, if the comparison result meets a preset threshold, determine the target coordinates corresponding to the object to be located based on the predicted bounding box.

[0007] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0010] This specification provides a data processing method in one embodiment, which determines a target image, identifies a reference text object, and obtains the corresponding bounding box of the reference text object to provide a reliable reference for the subsequent localization of the object to be located. The target image, the reference text corresponding to the reference text object, and the prompt text of the object to be located are input into an image processing model to obtain the reference bounding box of the reference text object and the predicted bounding box of the object to be located, output by the image processing model. By comparing the reference bounding boxes and the identified bounding boxes of the reference text object obtained from different sources, the differences between the two are determined. If the comparison result meets a preset threshold, the output result of the image processing model is determined to be reliable. The predicted bounding box of the object to be located output by the image processing model is then adopted. In other words, by comparing the reference bounding box of the reference text object and the identified bounding box, a reliable judgment basis is provided for the output of the image processing model. Based on the reliable predicted bounding box, the target coordinates of the object to be located are determined, effectively suppressing hallucination interference and ultimately ensuring the accuracy and reliability of the target coordinates. Attached Figure Description

[0011] Figure 1 This is a schematic diagram illustrating a data processing method provided in one embodiment of this specification. Figure 2 This is a flowchart illustrating a data processing method provided in one embodiment of this specification; Figure 3 This is a schematic diagram of a target image in a data processing method provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the processing procedure of a data processing method provided in one embodiment of this specification; Figure 5 This is a schematic diagram of a data processing method applied to an automated interface interaction scenario, provided by one embodiment of this specification. Figure 6 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification; Figure 7 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0012] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0013] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0015] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0016] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0017] Multimodal large models: Deep learning models that can simultaneously process multiple input modalities (such as text, images, audio, video, etc.) and output text.

[0018] OCR: Optical Character Recognition, can convert text information in an image into editable text.

[0019] Bounding box: A rectangular area used to enclose objects in an image. It is usually represented by the coordinates of the top left and bottom right corners (or the coordinates of the center point and the width and height).

[0020] Bounding box regression is a technique in computer vision and deep learning, primarily used for object detection tasks. In object detection, the model not only needs to identify the categories of objects in an image but also determine the locations of these objects within the image. Bounding box regression is a technique used to precisely adjust the predicted bounding box positions to more accurately enclose the target objects.

[0021] IoU: Intersection over Union, is commonly used in computer vision tasks such as object detection and image segmentation to measure the degree of overlap between the predicted results and the ground truth annotations.

[0022] Multimodal large models possess powerful image and text understanding capabilities. By uploading a webpage image, a multimodal large model can detect objects within the image and obtain their bounding box coordinates. However, multimodal large models typically pre-assume the size of the uploaded webpage image. This results in the returned coordinates being based on a visual estimate of the uploaded image, while the actual coordinates need to be adjusted according to the dimensions of the specific target device.

[0023] Furthermore, due to the illusion of a large multimodal model, repeatedly requesting the same object in an image does not guarantee the same result each time. These factors can cause the seemingly powerful large multimodal model capabilities to suffer from insufficient compatibility and stability, significantly diminishing their effectiveness in scenarios involving automated UI localization.

[0024] The data processing method provided in the embodiments of this specification can solve the problems of insufficient stability and compatibility of the results returned by multimodal large models, and obtain the real and accurate target coordinates of the object to be located on the target device, thereby achieving accurate target positioning.

[0025] This specification provides a data processing method, and also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0026] See Figure 1 , Figure 1 A schematic diagram of a data processing method according to an embodiment of this specification is shown. Specifically, this data processing method is applied to a data processing system, which includes a terminal device 102 and a server 104. The terminal device 102 is used to send a prompt text for the object to be located to the server 104, such as "Click the back button". In practical applications, users can input the prompt text into the terminal device 102 via text or voice. If voice is used, the terminal device 102 will also include a corresponding voice processing component, such as a voice parsing, voice-to-text, and voice synthesis module, to convert the content input by the user via voice into prompt text. This specification does not limit this.

[0027] An image processing model is trained in server 104. When server 104 determines the target image corresponding to the object to be located based on the prompt text, it identifies a reference text object in the target image and obtains the bounding box of the reference text object. The target image, the reference text corresponding to the reference text object, and the prompt text of the object to be located are input into the image processing model to obtain the reference bounding box of the reference text object and the predicted bounding box of the object to be located. A comparison result between the identified bounding box and the reference bounding box is obtained. If the comparison result meets a preset threshold, the target coordinates corresponding to the object to be located are determined based on the predicted bounding box. Afterwards, a page tool can be invoked to perform interactive operations on the target coordinates, such as sending an interaction command to the terminal device 102 to achieve interaction with the object to be located.

[0028] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on a software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, data processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0029] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0030] It is worth noting that the data processing method provided in the embodiments of this specification can be executed by the server 104. In other embodiments of this specification, the data processing model can be deployed in the end-side device 102, so that the end-side device 102 can also have similar functions to the server 104, thereby executing the data processing method provided in the embodiments of this specification. In other embodiments, the data processing method provided in the embodiments of this specification can also be jointly executed by the end-side device 102 and the server 104.

[0031] See Figure 2 , Figure 2A flowchart of a data processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0032] Step 202: Determine the target image.

[0033] In this context, the target image can be understood as an image containing both the object to be located and a text object. The object to be located is usually a visual element in the target image, such as a button, icon, or picture. This object is the interactive object that the user wants to click or view further. The text object can be understood as a text element in the target image. For example, the target image could be a search page image. The search box in this search page image contains a "camera" icon and the text element "search". In this case, the "camera" icon can be identified as the object to be located, and the "search" text element can be identified as the text object.

[0034] Taking a shopping scenario as an example, when a user searches for a product and is instructed to "click to take a picture and search" via natural language description, the image of the search page of the e-commerce website is identified as the target image, the "camera" icon in the search box of the search page image is identified as the object to be located, and the "search" text control in the search box, or the "recommend" text control, "follow" text control, etc. in the search page image are identified as text objects.

[0035] Taking a document processing scenario as an example, when a user is processing a document and instructs the user to "copy the format of the selected text" through natural language description, the target image is determined based on the document page with the "Home" tab selected (the "Format Painter" icon is included in the toolbar under the "Home" tab). The "Format Painter" icon (a picture button shaped like a brush) on the document page is determined as the object to be located, and the "Insert" tab or other text controls on the document page are determined as text objects.

[0036] In one or more embodiments of this specification, when it is determined that a user has an interaction need with the object to be located based on the prompt text corresponding to the object to be located, an object page containing the object to be located is determined. By taking a screenshot of the object page, an initial image containing the object to be located can be obtained. Then, through image preprocessing, a target image that is easier to locate the object can be obtained. Specific implementation methods are as follows: Determine the target image, including: Obtain the prompt text of the object to be located, wherein the prompt text is text used for interactive operations on the object to be located; Use the page tool to take a screenshot of the page containing the object to be located, and obtain an initial image; The initial image is preprocessed to obtain the target image.

[0037] The prompt text can be understood as the text content associated with the object to be located, used to describe or locate the object. For example, if an icon A corresponds to the "back" function, the prompt text "click the back button" can be used to determine that icon A is the object to be located, and then the location of the object can be achieved subsequently. Page tools can be understood as tools that can automate the processing of the page. Page tools can perform functions including but not limited to opening a specific page, taking screenshots of the page, and interacting with objects on the page.

[0038] The initial image can be understood as a screenshot of a page containing the object to be located; image preprocessing can be understood as a series of optimization operations performed on the image, including but not limited to grayscale conversion, noise removal, binarization, edge detection, etc., to improve the accuracy of subsequent image recognition.

[0039] Specifically, obtain the prompt text related to the object to be located. If the prompt text is text that allows interaction with the object, it can be determined that the user wants to interact with the object. Therefore, it is necessary to determine the specific location of the object, so that interaction with the object can be achieved by performing interaction operations on the determined specific location.

[0040] Once the object to be located is identified, an automated tool (i.e., a page tool with screenshot functionality) is used to take a screenshot of the page containing the object to be located, obtaining an initial image. This initial image is then preprocessed, such as by grayscale conversion (to remove color information), binarization (to separate the foreground from the background), and Canny edge detection (to generate clear and continuous object edge lines), to avoid interference from colored backgrounds. This makes the text, icons, etc., in the initial image clearly separated from the background, thus obtaining the final target image that can be used for precise location. This helps improve the accuracy of subsequent image processing model recognition or OCR (Optical Character Recognition) recognition.

[0041] In practical applications, when the prompt text is "Click the back button in the upper left corner," the icon corresponding to the "back button" is identified as the object to be located. Typically, the target page for this object is the current page; therefore, an automated tool can be used to take a screenshot of the current page to obtain the initial image. Alternatively, if the target object is not on the current page, an automated tool can be used to open the target page and then take a screenshot; this is not a limitation. Image preprocessing of the initial image yields the target image for subsequent image recognition.

[0042] Taking the prompt text "Search by taking a picture through application a" as an example, by obtaining and parsing the prompt text, it can be determined that the object to be located is "the camera icon on the search page of application a". By checking whether the current page is the search page of application a through the page tool, if it is determined that the current page is the system main interface, application a is launched first and the search page of application a is entered. At this time, when the page tool confirms that the current page is the object page containing the object to be located, a screenshot of the object page is taken so that the camera icon to be located is included in the obtained initial image.

[0043] The data processing method provided in the embodiments of this specification provides a basis for determining the object to be located by obtaining the prompt text of the object to be located. By calling the page tool object to take a screenshot, an initial image containing the object to be located can be accurately obtained. Furthermore, when the target image is obtained through image preprocessing, it helps to improve the accuracy of subsequent image recognition.

[0044] Step 204: Identify the reference text object in the target image and obtain the identification bounding box of the reference text object.

[0045] The reference text object can be understood as a text element with clear semantics determined from the target image. The reference text object is used to provide a reference for locating other objects. The bounding box can be understood as a rectangular area, which is a smaller rectangle that surrounds the text area.

[0046] Specifically, OCR technology is used to detect and recognize text in target images. Based on OCR technology, it is possible not only to identify text blocks in target images and extract the text content in the text blocks, but also to obtain the position information of each text block in the target image, which is the coordinate of the bounding box. In specific implementation, one text block is determined as a reference text object from the multiple identified text blocks, and the recognition bounding box corresponding to the reference text object is obtained.

[0047] In practical applications, taking a screenshot of the shopping main interface as the target image as an example, when performing text recognition on the target image, multiple text blocks such as "Home", "Recommendations", and "Messages" can be identified. These text blocks are text objects in the target image. Therefore, from the multiple identified text blocks, one text block (such as "Home") can be determined as the reference text object, and the recognition bounding box corresponding to the reference text object can be obtained. The recognition bounding box is represented by coordinates.

[0048] In one or more embodiments of this specification, when performing text object recognition on a target image, multiple initial text objects and their corresponding recognition confidence scores can be obtained, and a reference text object can be determined from the multiple initial text objects based on the recognition confidence scores. Specific implementation methods are described below: Identifying a reference text object in the target image and obtaining the bounding box of the reference text object includes: Text detection and recognition technology is used to perform text object recognition on the target image to obtain an initial text object, the initial text of the initial text object, an initial bounding box, and a recognition confidence score. Based on the recognition confidence of the initial text object, a reference text image is determined from the initial text object, and the reference text and recognition bounding box of the reference text object are obtained.

[0049] Text detection and recognition technology can be understood as a technology that combines computer vision and natural language processing. Text detection is responsible for locating the regions where text appears in an image, while text recognition is responsible for converting the text regions into editable strings (i.e., extracting the text content). Recognition confidence can be understood as a probability assessment of the correctness of the recognition result. The higher the recognition confidence, the more reliable the recognition result. The reference text image can be understood as the initial text object that has been screened by recognition confidence and is determined to be reliable (i.e., with a high recognition confidence).

[0050] Specifically, OCR technology is used to perform text object recognition on the target image, identifying multiple text objects (i.e., initial text objects) that may be contained in the target image. For each initial text object, its text content (i.e., the initial text corresponding to the initial text object), position coordinates (i.e., the coordinates of the initial bounding box corresponding to the initial text object), and recognition confidence are output.

[0051] Due to factors such as lighting and occlusion, errors or uncertainties may exist among multiple initial text objects. Therefore, the text object with the most reliable recognition result (e.g., a recognition confidence level of 1) is selected as the reference text object. See also Figure 3 , Figure 3 A schematic diagram of a target image in a data processing method is shown. Figure 3 The target image in the image is a "purchasing medicine" image. By performing text object recognition on this target image, the following information can be obtained: Text content: 'Prescription for medicine', left border: 429, top border: 173, width: 220, height: 57, center point coordinates (539, 201), recognition confidence: 1.0; Text content: 'My Consultation', left border: 873, top border: 175, width: 171, height: 53, center point coordinates (958, 201), recognition confidence: 1.0.

[0052] In the embodiments of this specification, the bounding box is represented by coordinates, and the left border, top border, width, height, and center point coordinates in the above information are the coordinate information corresponding to the identified bounding box.

[0053] In practice, if there are multiple initial text objects with a recognition confidence of 1, any initial text object with a recognition confidence of 1 can be determined as the reference text object. Alternatively, multiple initial text objects can be sorted in descending order based on their recognition confidence, and the initial text object that is first in the sorted order (with a higher recognition confidence) can be determined as the reference text object.

[0054] The data processing method provided in the embodiments of this specification can filter multiple initial text objects by identifying confidence levels, and select the initial text object with higher identification reliability as the reference text object. When the reference text object is used to provide a reference basis for the subsequent positioning of the object to be located, the accuracy of subsequent positioning of the object to be located can be greatly improved by increasing the reliability of the reference basis.

[0055] Step 206: Input the target image, the reference text corresponding to the reference text object, and the prompt text of the object to be located into the image processing model to obtain the reference bounding box of the reference text object and the predicted bounding box of the object to be located.

[0056] The image processing model can be understood as a multimodal large model capable of processing images and text. In the embodiments of this specification, the image processing model can accurately locate visual elements in an image that are semantically related to the given prompt text and output their corresponding bounding boxes. The reference bounding box of the reference text object is the bounding box corresponding to the reference text object output by the image processing model. The predicted bounding box of the object to be located is the bounding box corresponding to the object to be located output by the image processing model.

[0057] Specifically, the reference text corresponding to the reference text object (i.e., the text content corresponding to the reference text object) is used to guide the image processing model to locate the reference text object in the target image and output its corresponding reference bounding box. This reference bounding box is used to verify the recognition accuracy of the image processing model. The prompt text of the object to be located is used to guide the image processing model to locate the object to be located in the target image and output its corresponding predicted bounding box.

[0058] In practice, the target image, the reference text corresponding to the reference text object (which can be understood as the prompt text corresponding to the reference text object, used to guide the image processing model to locate the reference text object in the target image), and the prompt text of the object to be located are used to output the position coordinates of the object to be located in the target image (predicted bounding box) and the position coordinates of the reference text object in the target image (i.e., reference bounding box). That is, the image processing model will identify the reference text object used for reference calibration and the actual interactive target object to be located.

[0059] Step 208: Obtain the comparison result between the identified bounding box and the reference bounding box. If the comparison result meets a preset threshold, determine the target coordinates corresponding to the object to be located based on the predicted bounding box.

[0060] The comparison result can be understood as a quantitative index (including but not limited to intersection-union ratio, Jaccard similarity coefficient, Dice similarity coefficient, etc.) obtained after geometric calculation of the recognition bounding box corresponding to the reference text object and the reference bounding box. It is used to objectively measure the consistency between the recognition result corresponding to the text detection and recognition technology and the prediction result corresponding to the image processing model. The preset threshold can be understood as a pre-set critical standard. If the comparison result is greater than or equal to the preset threshold, the prediction result corresponding to the image processing model is considered reliable.

[0061] By verifying the reliability of the prediction results output by the image processing model, the accuracy of the localization of the object to be located is ensured. In specific implementation, the recognized bounding box of the reference text object obtained based on OCR technology is geometrically compared with the reference bounding box predicted by the image processing model for the same reference text object. For example, the intersection-union ratio (IUR) of the recognized bounding box of the reference text object and the reference bounding box can be calculated. If the IUR meets the preset threshold (e.g., IUR ≥ 0.9), the prediction result of the image processing model is determined to be reliable. Therefore, the prediction result of the image processing model for the object to be located (i.e., the predicted bounding box corresponding to the object to be located) can be adopted, and the target coordinates corresponding to the object to be located can be determined based on the predicted bounding box.

[0062] By introducing an objective and quantitative comparison result, it is possible to effectively prevent inaccurate recognition caused by illusions or misidentification in image processing models.

[0063] In one or more embodiments of this specification, the comparison result is determined by calculating the intersection and union regions of the identified bounding box and the reference bounding box. Specific implementation methods are described below: Obtaining the comparison result between the recognized bounding box and the reference bounding box includes: Calculate the intersection and union regions of the identified bounding box and the reference bounding box, and determine the comparison result based on the intersection and union regions.

[0064] The intersection region can be understood as the area where the two rectangular regions, the identification bounding box and the reference bounding box, overlap and intersect. The size of the intersection region reflects the degree of overlap between the positions of the identification bounding box and the reference bounding box. The union region can be understood as the total area occupied by the identification bounding box and the reference bounding box, that is, the total area covered after the identification bounding box and the reference bounding box are merged together. The comparison result can be understood as a quantitative index obtained by calculating the ratio of the area of ​​the intersection region to the area of ​​the union region (i.e., IoU, Intersection over Union), which is used to accurately evaluate the similarity between the identification bounding box and the reference bounding box in spatial position.

[0065] Specifically, based on the coordinates of the identified bounding box and the reference bounding box, the rectangular area of ​​the intersection of the two bounding boxes (the area of ​​the intersection region) and the total area covered by their merger (the area of ​​the union region) are calculated through mathematical operations. The area of ​​the intersection region is divided by the area of ​​the union region to calculate the IoU value. This IoU value is the comparison result, which is a value between 0 and 1. The closer the value is to 1, the more completely the same the positions of the two bounding boxes are, and the higher the similarity. The closer the value is to 0, the farther apart the two bounding boxes are, and the lower the similarity.

[0066] In practical applications, taking the reference bounding box coordinates as the top left corner (389, 189) and the bottom right corner (576, 240), and the recognition bounding box coordinates as the top left corner (429, 173) and the bottom right corner (649, 230) as an example, the intersection region of the two bounding boxes is calculated as follows: the top left corner coordinates of the intersection region are determined as (max(389, 429), max(189, 173)) = (429, 189), and the bottom right corner coordinates are determined as (min(576, 649), min(240, 230)) = (576, 230). Therefore, the width of the intersection region is calculated to be 576 - 429 = 147, the height is calculated to be 230 - 189 = 41, and the area of ​​the intersection region is 147 * 41 = 6027.

[0067] Calculate the union region of the two bounding boxes: Determine the area of ​​the reference bounding box (576-389)*(240-189)=187*51=9537, and the area of ​​the identified bounding box (649-429)*(230-173)=220*57=12540; The area of ​​the union region is the area of ​​the reference bounding box + the area of ​​the identified bounding box - the area of ​​the intersection region, that is, 9537+12540-6027=16050; IoU=area of ​​intersection region / area of ​​union region=6027 / 16050≈0.3755.

[0068] The data processing method provided in the embodiments of this specification offers a standardized, quantifiable, and geometrically meaningful similarity evaluation criterion. By calculating the intersection-union ratio (IoU), it can more comprehensively evaluate the comparison results between the identified bounding box and the reference bounding box.

[0069] In one or more embodiments of this specification, when a predicted bounding box is determined, the object enclosed by the predicted bounding box is verified. That is, by verifying the visual content within the predicted bounding box itself, the situation where the image processing model incorrectly identifies a similar object or background as the object to be located is effectively avoided. Specific implementation methods are described below: Determining the target coordinates corresponding to the object to be located based on the predicted bounding box includes: Confirm the enclosing object corresponding to the predicted bounding box; When the enclosing object is the object to be located, the target coordinates corresponding to the object to be located are determined based on the predicted bounding box.

[0070] The enclosing object can be understood as the actual visual content within the area defined by the predicted bounding box.

[0071] Specifically, the region enclosed by the predicted bounding box is analyzed to identify the enclosing objects within that region. In practical applications, bounding box regression can be used to identify the enclosing objects within the predicted bounding box. When it is determined that the enclosing object matches the expected object to be located, the coordinates of the predicted bounding box are adopted, and the target coordinates are finally determined based on the predicted bounding box.

[0072] The data processing method provided in the embodiments of this specification effectively avoids possible erroneous predictions in image processing models by recognizing and verifying the content of the bounding box. That is, the coordinates of the predicted bounding box are only adopted when the bounding object within the predicted bounding box is the expected object to be located, and the final target coordinates are determined, which greatly improves the accuracy and reliability of the final target coordinates.

[0073] In one or more embodiments of this specification, since both the reference bounding box and the predicted bounding box output by the image processing model are calculated based on their preset page size, and when comparing the recognized bounding box and the reference bounding box, it must be ensured that both are under the same page size standard. Therefore, coordinate transformation is required for the reference bounding box output by the image processing model. Specific implementation methods are described below: Obtaining the comparison result between the recognized bounding box and the reference bounding box includes: Perform coordinate transformation on the reference bounding box to obtain the target reference bounding box; The identified bounding box is compared with the target reference bounding box to obtain the comparison result.

[0074] Coordinate transformation can be understood as mapping the bounding box coordinates output by the image processing model based on the predicted size of the target image to the actual size of the target image proportionally, in order to solve the coordinate deviation caused by inconsistent image resolution; the target reference bounding box can be understood as a bounding box that matches the actual size of the target image after coordinate transformation and can be used for direct comparison.

[0075] Specifically, the coordinate information of the reference bounding box output by the image processing model is the coordinate information of the reference bounding box at a preset size of the target image. However, the size of the target image predicted by the image processing model (i.e., the predicted size) usually differs from the actual size of the target image. When the bounding box is directly identified and located from the target image, the coordinate information of the identified bounding box is determined based on the actual size of the target image, while the coordinate information of the reference bounding box is determined based on the predicted size of the target image.

[0076] Directly comparing a reference bounding box determined by a preset size with a recognition bounding box determined by the actual size will result in incorrect comparison results due to the different page size references. Therefore, it is necessary to perform coordinate transformation on the reference bounding box output by the image processing model so that its transformed coordinate values ​​are adapted to the actual size of the target image, thus obtaining a transformed target reference bounding box. This ensures that the target reference bounding box and the recognition bounding box are under the same page size standard, which is the actual size of the target image, thereby obtaining an accurate comparison result.

[0077] The data processing method provided in the embodiments of this specification eliminates coordinate deviations caused by differences in image resolution through coordinate transformation, ensuring that coordinate data from different sources are placed under the same scale standard for accurate comparison. Furthermore, through coordinate transformation, this data processing method can flexibly adapt to device screens with various resolutions.

[0078] In one or more embodiments of this specification, when performing coordinate transformation on a reference bounding box, the coordinate transformation ratio is determined based on the actual size and the predicted size of the target image. The coordinate information of the reference bounding box is then transformed according to the calculated coordinate transformation ratio to obtain a target reference bounding box based on its actual size. Specific implementation methods are described below: Performing coordinate transformation on the reference bounding box to obtain the target reference bounding box includes: Determine the image size of the target image and obtain the predicted size of the target image output by the image processing model; Calculate the first coordinate transformation ratio based on the image size and the predicted size; The reference bounding box is transformed according to the first coordinate transformation ratio to obtain the target reference bounding box.

[0079] Here, the image size can be understood as the actual size of the target image in the above embodiments, which is the actual width and height of the target image (unit: pixels). This image size is the final target scale for coordinate mapping. The predicted size can be understood as the width and height of the target image predicted by the image processing model. The bounding box coordinates output by the image processing model are calculated based on this predicted size. The first coordinate transformation ratio can be understood as a set of scaling parameters required to map the bounding box coordinates output by the image processing model from the preset size to the image size. It usually includes two independent ratio values ​​in the horizontal and vertical directions (i.e., width and height).

[0080] Specifically, the actual size (actual width and height) of the target image is determined, and the predicted size (assumed width and height) of the target image output by the image processing model is obtained. In fact, when the image processing model receives the target image, the reference text corresponding to the reference text object, and the prompt text of the object to be located, it will not only output the reference bounding box corresponding to the reference text object and the predicted bounding box of the object to be located, but also output the predicted size of the target image.

[0081] By dividing the actual size by the predicted size, the transformation ratios in the width and height directions (i.e., the first coordinate transformation ratios) are calculated. Using the calculated transformation ratios, the coordinate values ​​of the reference bounding box output by the image processing model are scaled to obtain a target reference bounding box that matches the actual size of the target image and can be used for accurate positioning and comparison.

[0082] For example, taking a screenshot of a mobile phone with an actual resolution of 1080x2400 as the target image, when the image processing model performs object localization on the target image, the output reference bounding box coordinates are (x1, y1, x2, y2), and the predicted size of the target image is 750×1400. That is, the coordinates of the above reference bounding box are determined based on the predicted size of 750×1400.

[0083] At this point, directly comparing the reference bounding box with the recognized bounding box identified using OCR technology on the 1080x2400 image is ineffective. Specifically, it's necessary to calculate the width scaling ratio (1080 / 750) and the height scaling ratio (2400 / 1400), and then scale the bounding box coordinates output by the image processing model using these scaling ratios to obtain the corresponding target reference bounding box on the 1080x2400 image. Only by comparing this transformed target reference bounding box with the recognized bounding box can an accurate comparison result be obtained, thus correctly determining the accuracy of the image processing model's prediction.

[0084] The data processing method provided in the embodiments of this specification accurately obtains the first coordinate transformation ratio by using the image size of the target image and the predicted size determined by the image processing model. This eliminates the deviation between the preset size and the actual size through precise coordinate mapping, providing an accurate coordinate basis for subsequent comparison, verification and final operation positioning. It also ensures the universality and reliability of the method on devices with different resolutions.

[0085] In one or more embodiments of this specification, when acquiring a target image through a screenshot, there may be a discrepancy between the physical screen size of the target device and the screenshot size of the acquired target image. However, given that the embodiments of this specification aim to achieve accurate interactive operations with the object to be located, these operations need to be performed based on the actual coordinates on the target device. Therefore, the physical screen size of the target device should be used as the final true size standard. Specific implementation methods are described below: Determining the image size of the target image includes: When the target image is obtained by taking a screenshot of the target device, the physical screen size of the target device and the screenshot size of the target image are determined. When the physical screen size is the same as the screenshot size, the physical screen size and / or the screenshot size are determined as the image size of the target image; If the physical screen size is inconsistent with the screenshot size, the physical screen size shall be determined as the image size of the target image.

[0086] The physical screen size can be understood as the inherent, pixel-based original resolution of the target device's screen (e.g., a physical screen size of 1080x2400); the screenshot size can be understood as the actual width and height of the image obtained through a screenshot operation. This screenshot size may be the same as the physical screen size of the target device, or it may differ due to scaling caused by the screenshot tool or settings.

[0087] Specifically, when obtaining a target image by taking a screenshot of the target device, the screenshot size may or may not match the physical screen size of the target device. If the screenshot size matches the physical screen size, it means the screenshot is a direct capture of the target device's screen as is, and either the screenshot size or the physical screen size can be used as a reliable image size. However, if the screenshot size differs from the physical screen size due to scaling, cropping, or adding outlines by the screenshot tool, the physical screen size is used as the baseline and is taken as the image size of the target image.

[0088] In practical applications, since the goal of the embodiments in this specification is to achieve real interactive operations with the object to be located, it is necessary to ensure that the target coordinates of the object to be located are real and reliable. When the object to be located is a visual element displayed on the target device page, its target coordinates need to be determined based on the physical screen size of the target device. Therefore, the physical screen size of the target device should be used as a unified size reference for the recognition bounding box, the reference bounding box, and the predicted bounding box.

[0089] In practice, since the bounding box is identified on the target image obtained from a screenshot, its coordinates are determined based on the screenshot size. When the physical screen size differs from the screenshot size, the bounding box determined based on the screenshot size also needs coordinate transformation so that both the transformed target bounding box and the target reference bounding box are determined based on the physical screen size. The specific implementation method is as follows: The comparison of the identified bounding box with the target reference bounding box to obtain the comparison result includes: If the physical screen size and the screenshot size are inconsistent, calculate the second coordinate transformation ratio based on the physical screen size and the screenshot size; The target recognition bounding box is obtained by performing coordinate transformation on the recognition bounding box according to the second coordinate transformation ratio; The target recognition bounding box is compared with the target reference bounding box to obtain the comparison result.

[0090] The second coordinate transformation ratio can be understood as a set of scaling parameters required to map the coordinates from the screenshot size back to the physical screen size, so that the coordinates return to the actual device screen reference; the target recognition bounding box can be understood as the bounding box that matches the physical screen size after being adjusted by the second coordinate transformation ratio.

[0091] Specifically, when the physical screen size and the screenshot size are inconsistent, and the recognition bounding box is recognized from the target image based on the screenshot size, the coordinates of the recognition bounding box cannot be directly compared with the target reference bounding box converted based on the physical screen size (image size). The coordinates of the recognition bounding box obtained by OCR technology need to be corrected.

[0092] In practice, a second coordinate transformation ratio (physical screen size / screenshot size) is calculated based on the actual physical screen size and the screenshot size. This second coordinate transformation ratio also includes two coordinate transformation ratios: height and width. The coordinates of the recognition bounding box are scaled using the second coordinate transformation ratio and transformed into a coordinate space based on the physical screen size, thereby obtaining an accurate target recognition bounding box. By geometrically comparing the target recognition bounding box under the same size reference (i.e., physical screen size) with the target reference bounding box, a real and effective comparison result is obtained.

[0093] In practical applications, if the coordinate transformation ratio is calculated using the screenshot size and the predicted size, and the reference bounding box is transformed based on the calculated coordinate transformation ratio to obtain the target reference bounding box, both the target reference bounding box and the recognition bounding box are determined based on the screenshot size. Since comparing the target reference bounding box and the recognition bounding box is used to determine the prediction accuracy of the image processing model, the comparison can be achieved as long as the target reference bounding box and the recognition bounding box are under a unified size reference. Therefore, no additional coordinate transformation is required for the recognition bounding box. The specific processing method can be flexibly determined according to actual needs and is not limited here.

[0094] The data processing method provided in the embodiments of this specification recalibrates the OCR recognition coordinates, which are inconsistent with the physical screen size, to the physical screen coordinate system of the target device through coordinate transformation. This ensures that when the obtained target recognition bounding box is compared with the target reference bounding box, the size reference is consistent and accurate, providing an accurate reference for the final realization of high-precision positioning.

[0095] In one or more embodiments of this specification, if the comparison result meets a preset threshold, it indicates that the prediction result of the image processing model is reliable. Therefore, the predicted bounding box of the object to be located output by the image processing model can be adopted. However, if the predicted bounding box is also determined based on the predicted size of the target image, it is necessary to use the first coordinate transformation ratio to perform coordinate transformation on the predicted bounding box to obtain the target predicted bounding box determined based on the physical screen size. The specific implementation is as follows: Determining the target coordinates corresponding to the object to be located based on the predicted bounding box includes: The predicted bounding box is transformed according to the first coordinate transformation ratio to obtain the target predicted bounding box; The target coordinates of the object to be located are determined based on the target prediction bounding box.

[0096] The target predicted bounding box can be understood as the predicted bounding box output by the large image processing model based on the predicted size, which is then scaled using the first coordinate transformation ratio to obtain a bounding box that matches the physical screen size of the target device. The target coordinates can be understood as the coordinate values ​​extracted from the finally confirmed target predicted bounding box and used to drive the page tools to interact with it. Usually, the target coordinates are the coordinates of the center point of the target predicted bounding box.

[0097] Specifically, the image processing model generates both the predicted bounding box of the object to be located and the reference bounding box of the reference text object at the same predicted size. Therefore, the first coordinate transformation ratio calculated above for correcting the reference bounding box can be applied to the predicted bounding box. The predicted bounding box is transformed by the first coordinate transformation ratio and mapped to the image size of the target image (i.e., the physical screen size of the target device) to obtain the target predicted bounding box. Based on the transformed target predicted bounding box, the final interactive operation coordinates (i.e., target coordinates, usually the center point coordinates of the target predicted bounding box) are calculated.

[0098] The data processing method provided in the embodiments of this specification ensures the consistency of the correction of the prediction results output by the image processing model by reusing the first coordinate transformation ratio calculated for the reference bounding box, and the final output target coordinates correspond to the actual screen pixel positions of the target device, so that the automated operation based on vision recognition can be executed accurately and reliably.

[0099] In one or more embodiments of this specification, when the prompt text for the object to be located is text used for interactive operations on the object, after obtaining the target coordinates corresponding to the object, the actual interactive operation on the target coordinates is performed by calling the page tools to realize the interaction with the object to be located. Specific implementation methods are as follows: After determining the target coordinates corresponding to the object to be located based on the predicted bounding box, the method further includes: The page tool is invoked to perform interactive operations on the target coordinates in order to achieve interaction with the object to be located.

[0100] Specifically, by invoking page tools, various user interactions with the graphical interface can be simulated, such as clicking and long-pressing. After obtaining the target coordinates of the object to be located, interactive commands can be executed on those coordinates using page tools. For example, a click event can be sent to the target coordinates to simulate the user's finger touching the screen or the mouse clicking a button, thereby successfully triggering the application function associated with the object to be located. This transforms the calculated target coordinates into a physical operation that can be executed and trigger an application response.

[0101] For example, if the prompt text for the object to be located is "Click the back button," then the object to be located is the back button. The ultimate goal is to automatically click the back button on the page, thereby returning to the previous page. The steps described above calculate the target coordinates of the back button on the screen, and then the page tool sends a click command to these coordinates, simulating the user's manual click behavior. This precisely triggers the back button's function, achieving the goal of automatically returning to the previous page.

[0102] In practical applications, the target coordinates are usually taken as the center point of the bounding box. Therefore, even if there is a certain deviation in the target coordinates, it can still be ensured that the interactive operation occurs within the bounding box, thereby reliably triggering the expected operation on the object to be located.

[0103] The data processing method provided in the embodiments of this specification transforms the intelligently identified and located target coordinates into actual, automated capabilities applicable to specific scenarios by calling page tools, greatly improving operational efficiency and accuracy.

[0104] The data processing method provided in the embodiments of this specification uses OCR recognition results to supervise the prediction results of the image processing model, effectively solving the illusion problem of image understanding in the image processing model and ensuring the stability of the output results. By using natural language prompts to describe the page target (i.e. the object to be located) and combining them with visual recognition, the target localization capability of the image processing model is greatly enhanced. Furthermore, through automatic coordinate transformation, it can adapt to the screen resolution of different devices, generate coordinates that are truly interactive with the corresponding devices, solve the compatibility problem of multiple devices, and achieve accurate target localization across platforms and technology stacks.

[0105] See Figure 4 , Figure 4 This specification illustrates a schematic diagram of the processing procedure of a data processing method provided in one embodiment.

[0106] Taking the interactive operation of target A (i.e. the object to be located) on a mobile page as an example, this data processing method will be explained in detail.

[0107] Specifically, based on the natural language description of the input target A, "Click the back button in the upper left corner" (i.e., the prompt text), a UI (user interface) automation tool (i.e., a page tool) is used to open the mobile page and take a screenshot, obtaining information such as... Figure 3 The image shown undergoes grayscale conversion, binarization, and Canny edge detection through an image preprocessing module to refine the obtained screenshot (i.e., the initial image), thereby suppressing interference from colored backgrounds, enhancing the separation between text and background, and improving the accuracy of subsequent OCR and image processing model recognition.

[0108] In the OCR recognition module, the locally deployed OCR service is called to recognize the pre-processed page screenshot (i.e., the target image), extract the text (i.e., the initial text object in the above embodiment) and the corresponding coordinates and confidence scores of the text in the target image. The initial text object "prescription and medicine purchase" with a confidence score of 1 is selected from the OCR recognition results as the observation target B (i.e., the reference text object), and the bounding box coordinates of target B (i.e., the coordinates of the recognition bounding box in the above embodiment) are determined based on the OCR recognition results for subsequent verification with the results output by the image processing model.

[0109] The prompt text "top left corner return button" for target A, the text content "prescription and medicine purchase" for target B, and the preprocessed screenshot of the page are used as input parameters of the large model and input into the large model recognition module. The large model recognition module obtains the bounding box coordinates of target A (i.e., the predicted bounding box coordinates in the above embodiment) and the bounding box coordinates of target B (i.e., the reference bounding box coordinates in the above embodiment) predicted by the multimodal large model (i.e., the image processing model in the above embodiment) and the prompt text (including the text content of target B and the prompt text of target A) through the multimodal large model, and obtains the width and height of the page screenshot assumed by the multimodal large model (i.e., the predicted size in the above embodiment).

[0110] The coordinate transformation module converts the large model recognition results (boundary box coordinates of target A and target B) and the OCR recognition results (boundary box coordinates of target B) into the coordinates of the corresponding mobile page using a coordinate transformation tool. Specifically, based on the proportional relationship between the predicted size, the screenshot size, and the actual screen size of the mobile device (i.e., the physical screen size of the target device in the above embodiment), the three bounding box coordinates obtained above are uniformly transformed into the mobile page coordinate system. It should be noted that the bounding box coordinates of target B recognized by OCR (including coordinate information such as left border, top border, width, and height) need to be converted into a standard bounding box coordinate format, that is, represented using the coordinates of the top left corner and the bottom right corner, before performing coordinate transformation. If the screenshot size of the page is consistent with the actual screen size of the mobile device, there is no need to scale the coordinates recognized by OCR.

[0111] In practice, based on the multimodal large model assumptions regarding the width and height of the page screenshot (750x1400) and the actual width and height of the mobile device (1080x2400), the actual bounding box coordinates a1 of target A on the mobile page are calculated. For example, before the conversion, the predicted bounding box coordinates of target A are (x1, y1): (30, 110) at the top left corner and (x2, y2): (60, 140) at the bottom right corner. After the conversion, the predicted bounding box coordinates of target A (i.e., the actual bounding box coordinates a1) are calculated. The top left corner is (new_x1, new_y1):(43,189), and the bottom right corner is (new_x2, new_y2):(86,240). The specific calculation method is as follows: new_x1=(x1*1080) / 750=43, new_x2=(x2*1080) / 750=86, new_y1=(y1*2400) / 1400=189, new_y2=(y2*2400) / 1400=240.

[0112] Based on the multimodal large model assumptions, the width and height of the page screenshot and the actual width and height of the mobile device are used to convert the bounding box coordinates of target B identified by the multimodal large model into the actual bounding box coordinates b1 of target B on the mobile page. For example, before the conversion, the reference bounding box coordinates of target B are (x1, y1): (270, 110) at the top left corner and (x2, y2): (400, 140) at the bottom right corner. After the conversion, the target reference bounding box coordinates (i.e., the actual bounding box coordinates b1) of target B are (x1, y1): (270, 110) at the top left corner and (x2, y2): (400, 140) at the bottom right corner. (new_x1,new_y1):(389,189), bottom right corner (new_x2,new_y2):(576,240), the specific calculation method is new_x1=(x1*1080) / 750=389, new_x2=(x2*1080) / 750=576, new_y1=(y1*2400) / 1400=189, new_y2=(y2*2400) / 1400=240.

[0113] If the width and height (1080x2400) of the page screenshot are the same as the width and height (1080x2400) of the mobile device, then the bounding box coordinates of the target B recognized by OCR will be determined as the actual bounding box coordinates b2 of the target B "prescription for medicine" on the mobile page. However, the coordinates need to be converted into a different format. For example, the bounding box coordinates in OCR recognition are represented as follows: text content: 'prescription for medicine', left border: 429, top border: 173, width: 220, height: 57, center point coordinates (539, 201), recognition confidence: 1.0; after conversion, the bounding box coordinates are represented using the coordinate format of the top left corner (x1, y1): (429, 173) and the bottom right corner (x2, y2): (649, 230). The specific calculation method is x1=left (left border), x2=left+width (left border + width), y1=top (top border), y2=top+height (top border + height).

[0114] The bounding box coordinate comparison module calculates the degree of overlap between the transformed actual bounding box coordinates b1 and b2 based on the IoU (Intersection over Union) ratio. If the IoU value is greater than or equal to 90%, the prediction result of the multimodal large model is considered reliable. The bounding box regression is then used to confirm whether the object enclosed by the predicted bounding box is consistent with the target A. If so, the large model recognition result of target A (i.e., the actual bounding box coordinates a1) is adopted. If the IoU is less than 90%, the large model recognition module is used again for prediction until the output that meets the consistency condition is obtained, that is, until the IoU value is greater than or equal to 90%.

[0115] Based on the actual bounding box coordinates a1 corresponding to target A, the center point coordinates of the "top left corner back button" of target A (i.e., target coordinates) are calculated, and then the click operation of the "top left corner back button" of target A is completed through UI automation tools.

[0116] See Figure 5 , Figure 5 This diagram illustrates a data processing method for automated user interface interaction scenarios according to an embodiment of this specification; specifically, upon receiving the prompt text "Click the upper left corner back button", the data processing method can... Figure 5 (a) is identified as the target image, and the center coordinates of the return button are obtained through the above steps. An automated tool is then used to enable navigation to the desired page after clicking the return button. Figure 5 Page (b) in the document refers to the page that displays a back button pop-up when the back button is clicked.

[0117] In practical applications, the prompt text can be "Recognize the shopping cart icon," where the "shopping cart" icon is the object to be located. The page tool automatically clicks the target coordinates corresponding to the "shopping cart" icon, thereby jumping to the shopping cart interface. In the scenario of recognizing the search results list, the prompt text can also be "Click the first product name in the search results list," thereby jumping to the details page of the first product in the search results list through the page tool. There is no limitation here.

[0118] The data processing method provided in the embodiments of this specification utilizes the high accuracy of OCR text recognition, uses the OCR recognition result as the real labeled data, and the multimodal large model recognition result as the prediction result. By measuring the degree of overlap (IoU value) between the prediction result and the real labeled data, the real usability of the multimodal large model return result is initially confirmed. Then, the bounding box regression algorithm of deep learning is applied to further confirm that the object enclosed by the bounding box is consistent with the expected target. This can effectively solve the illusion problem of multimodal large models and ensure the stability of the output result.

[0119] This data processing method leverages the language and image understanding capabilities of a multimodal large model. In addition to recognizing common front-end page icons, it can also identify page targets with specific semantics. This not only greatly improves the intelligence level of target localization but also broadens the scope of application scenarios.

[0120] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 6 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 6 As shown, the device includes: Image determination module 602 is configured to determine a target image; The reference recognition module 604 is configured to recognize a reference text object in the target image and obtain the recognition bounding box of the reference text object; The model prediction module 606 is configured to input the target image, the reference text corresponding to the reference text object, and the prompt text of the object to be located into the image processing model to obtain the reference bounding box of the reference text object and the predicted bounding box of the object to be located. The coordinate determination module 608 is configured to obtain the comparison result between the recognition bounding box and the reference bounding box, and, if the comparison result meets a preset threshold, determine the target coordinates corresponding to the object to be located based on the predicted bounding box.

[0121] Optionally, the reference identification module 604 is further configured to: Text detection and recognition technology is used to perform text object recognition on the target image to obtain an initial text object, the initial text of the initial text object, an initial bounding box, and a recognition confidence score. Based on the recognition confidence of the initial text object, a reference text image is determined from the initial text object, and the reference text and recognition bounding box of the reference text object are obtained.

[0122] Optionally, the coordinate determination module 608 is further configured to: Perform coordinate transformation on the reference bounding box to obtain the target reference bounding box; The identified bounding box is compared with the target reference bounding box to obtain the comparison result.

[0123] Optionally, the coordinate determination module 608 is further configured to: Determine the image size of the target image and obtain the preset size of the target image output by the image processing model; Calculate the first coordinate transformation ratio based on the image size and the predicted size; The reference bounding box is transformed according to the first coordinate transformation ratio to obtain the target reference bounding box.

[0124] Optionally, the coordinate determination module 608 is further configured to: When the target image is obtained by taking a screenshot of the target device, the physical screen size of the target device and the screenshot size of the target image are determined. When the physical screen size is the same as the screenshot size, the physical screen size and / or the screenshot size are determined as the image size of the target image; If the physical screen size is inconsistent with the screenshot size, the physical screen size shall be determined as the image size of the target image.

[0125] Optionally, the coordinate determination module 608 is further configured to: If the physical screen size and the screenshot size are inconsistent, calculate the second coordinate transformation ratio based on the physical screen size and the screenshot size; The target recognition bounding box is obtained by performing coordinate transformation on the recognition bounding box according to the second coordinate transformation ratio; The target recognition bounding box is compared with the target reference bounding box to obtain the comparison result.

[0126] Optionally, the coordinate determination module 608 is further configured to: The predicted bounding box is transformed according to the first coordinate transformation ratio to obtain the target predicted bounding box; The target coordinates of the object to be located are determined based on the target prediction bounding box.

[0127] Optionally, the coordinate determination module 608 is further configured to: Calculate the intersection and union regions of the identified bounding box and the reference bounding box, and determine the comparison result based on the intersection and union regions.

[0128] Optionally, the coordinate determination module 608 is further configured to: Confirm the enclosing object corresponding to the predicted bounding box; When the enclosing object is the object to be located, the target coordinates corresponding to the object to be located are determined based on the predicted bounding box.

[0129] Optionally, the image determination module 602 is further configured to: Obtain the prompt text of the object to be located, wherein the prompt text is text used for interactive operations on the object to be located; Use the page tool to take a screenshot of the page containing the object to be located, and obtain an initial image; The initial image is preprocessed to obtain the target image.

[0130] The device further includes: The calling module is configured to invoke the page tool to perform interactive operations on the target coordinates in order to achieve interaction with the object to be located.

[0131] The data processing apparatus provided in this specification determines a target image, identifies a reference text object, and obtains the corresponding bounding box of the reference text object, providing a reliable reference basis for the subsequent localization of the object to be located. The target image, the reference text corresponding to the reference text object, and the prompt text of the object to be located are input into an image processing model to obtain the reference bounding box of the reference text object and the predicted bounding box of the object to be located, output by the image processing model. By comparing the reference bounding boxes and the identified bounding boxes of the reference text object obtained from different sources, the differences between the two are determined. When the comparison result meets a preset threshold, the result output by the image processing model is determined to be reliable. The predicted bounding box of the object to be located output by the image processing model is then adopted. In other words, by comparing the reference bounding box and the identified bounding box of the reference text object, a reliable judgment basis is provided for the output of the image processing model. Based on the reliable predicted bounding box, the target coordinates of the object to be located are determined, effectively suppressing hallucination interference and ultimately ensuring the accuracy and reliability of the target coordinates.

[0132] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0133] Figure 7 A structural block diagram of a computing device 700 according to one embodiment of this specification is shown. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0134] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0135] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0136] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0137] The processor 720 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0138] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0139] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0140] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0141] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0142] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0143] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0144] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0145] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0146] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0147] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Identify the target image; The reference text object in the target image is identified based on OCR technology, and the bounding box of the reference text object is obtained. The target image, the reference text corresponding to the reference text object, and the prompt text of the object to be located are input into the image processing model to obtain the reference bounding box of the reference text object and the predicted bounding box of the object to be located. The comparison result between the identified bounding box and the reference bounding box is obtained. If the comparison result meets a preset threshold, the target coordinates corresponding to the object to be located are determined according to the predicted bounding box. The comparison result includes the intersection-union ratio of the identified bounding box and the reference bounding box.

2. The method as described in claim 1, wherein the reference text object in the target image is identified based on OCR technology, and the recognition bounding box of the reference text object is obtained, comprising: Text detection and recognition technology is used to perform text object recognition on the target image to obtain an initial text object, the initial text of the initial text object, an initial bounding box, and a recognition confidence score. Based on the recognition confidence of the initial text object, a reference text image is determined from the initial text object, and the reference text and recognition bounding box of the reference text object are obtained.

3. The method as described in claim 1, wherein obtaining the comparison result between the recognized bounding box and the reference bounding box includes: Perform coordinate transformation on the reference bounding box to obtain the target reference bounding box; The identified bounding box is compared with the target reference bounding box to obtain the comparison result.

4. The method as described in claim 3, wherein coordinate transformation is performed on the reference bounding box to obtain the target reference bounding box, comprising: Determine the image size of the target image and obtain the preset size of the target image output by the image processing model; Calculate the first coordinate transformation ratio based on the image size and the predicted size; The reference bounding box is transformed according to the first coordinate transformation ratio to obtain the target reference bounding box.

5. The method of claim 4, wherein determining the image size of the target image comprises: When the target image is obtained by taking a screenshot of the target device, the physical screen size of the target device and the screenshot size of the target image are determined. When the physical screen size is the same as the screenshot size, the physical screen size and / or the screenshot size are determined as the image size of the target image; If the physical screen size is inconsistent with the screenshot size, the physical screen size shall be determined as the image size of the target image.

6. The method of claim 5, wherein comparing the identified bounding box with the target reference bounding box to obtain the comparison result includes: If the physical screen size and the screenshot size are inconsistent, calculate the second coordinate transformation ratio based on the physical screen size and the screenshot size; The target recognition bounding box is obtained by performing coordinate transformation on the recognition bounding box according to the second coordinate transformation ratio; The target recognition bounding box is compared with the target reference bounding box to obtain the comparison result.

7. The method of claim 4, wherein determining the target coordinates corresponding to the object to be located based on the predicted bounding box includes: The predicted bounding box is transformed according to the first coordinate transformation ratio to obtain the target predicted bounding box; The target coordinates of the object to be located are determined based on the target prediction bounding box.

8. The method according to any one of claims 1-7, wherein determining the target image comprises: Obtain the prompt text of the object to be located, wherein the prompt text is text used for interactive operations on the object to be located; Use the page tool to take a screenshot of the page containing the object to be located, and obtain an initial image; The initial image is preprocessed to obtain the target image; After determining the target coordinates corresponding to the object to be located based on the predicted bounding box, the method further includes: The page tool is invoked to perform interactive operations on the target coordinates in order to achieve interaction with the object to be located.

9. A data processing apparatus, comprising: The image determination module is configured to determine the target image; The reference recognition module is configured to recognize reference text objects in the target image based on OCR technology and obtain the recognition bounding box of the reference text objects; The model prediction module is configured to input the target image, the reference text corresponding to the reference text object, and the prompt text of the object to be located into the image processing model to obtain the reference bounding box of the reference text object and the predicted bounding box of the object to be located. The coordinate determination module is configured to obtain the comparison result between the identified bounding box and the reference bounding box, and, if the comparison result meets a preset threshold, determine the target coordinates corresponding to the object to be located based on the predicted bounding box. The comparison result includes the intersection-union ratio of the identified bounding box and the reference bounding box.

10. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1-8.

11. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1-8.

12. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1-8.

Images