Region determination method

Abstract

The invention provides a region determination method. The region determination method comprises the following steps: acquiring a first target information set carried by a target display interface; obtaining target input data input by a user; determining second target information related to the target input data in the first target information set; based on an area occupied by the second target information in the target display interface, determining a target area meeting an area condition in the target display interface; wherein the target area meeting the meeting area condition is an effective area capable of representing the target input data in the target display interface.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for determining a region. Background Technology

[0002] When extracting information of interest from the display interface of an electronic device, it is often interfered with by irrelevant information in other areas of the display interface, resulting in the inability to accurately extract the information of interest in the effective area. Summary of the Invention

[0003] The purpose of this application is to provide a method for determining a region.

[0004] The embodiments of this application adopt the following technical solution: a region determination method, comprising:

[0005] Obtain the first set of target information displayed on the target interface;

[0006] Obtain the target input data from the user;

[0007] Determine second target information related to the target input data from the first target information set;

[0008] Based on the area occupied by the second target information in the target display interface, a target area that meets the area conditions is determined in the target display interface.

[0009] The target region that satisfies the specified regional conditions is a valid region in the target display interface that can represent the target input data.

[0010] In some embodiments, determining the second target information related to the target input data in the first target information set includes:

[0011] Based on the target input data, determine the type of target information related to the target input data;

[0012] In the first set of target information, determine the second target information whose information type matches the target information type.

[0013] In some embodiments, determining a target area that meets the area conditions in the target display interface based on the area occupied by the second target information includes:

[0014] Based on the area block occupied by each of the second target information in the target display interface, third target information that satisfies the conditions of the first area block is determined from each of the second target information;

[0015] In the target display interface, a target region is determined that contains the corresponding area blocks of each of the third target information; wherein, the target region does not contain the area blocks corresponding to the fourth target information in each of the second target information that do not meet the conditions of the first area block;

[0016] or

[0017] The distribution information of each region block corresponding to each second target information in each first candidate region is obtained respectively;

[0018] In each of the first candidate regions, a target region whose distribution information satisfies the distribution conditions is determined.

[0019] In some embodiments, determining third target information that satisfies the first region block condition from among the second target information based on the region block occupied by each of the second target information includes:

[0020] Obtain the distance information set corresponding to each of the second target information;

[0021] Based on the distance information set corresponding to each of the second target information, third target information that meets the regional block distance condition is determined from each of the second target information;

[0022] The distance information set includes distance information between the region block corresponding to the second target information and the region blocks corresponding to other second target information; the region block corresponding to the second target information that does not meet the region block distance condition is an outlier region block.

[0023] In some embodiments, based on the area block occupied by each of the second target information pieces in the target display interface, determining third target information that satisfies the first area block condition from among the second target information pieces includes:

[0024] Mark the area blocks occupied by each of the second target information in the target display interface;

[0025] Delete the region block that meets the outlier condition in the target display interface to obtain the region block corresponding to the third target information in the target display interface; or

[0026] In a comparison interface corresponding to the target display interface, mark the area block occupied by each of the second target information;

[0027] Delete the region block that meets the outlier condition in the comparison interface to obtain the region block corresponding to the third target information in the comparison interface.

[0028] In some embodiments, determining the target region containing the corresponding region blocks of each of the third target information in the target display interface includes:

[0029] Based on the position of each region block corresponding to the third target information in the target display interface, determine the region boundary parameters of the region occupied by each region block corresponding to the third target information in the target display interface.

[0030] The target region is obtained based on the region boundary parameters.

[0031] In some embodiments, it also includes:

[0032] If the first target information carried by the target display interface is obtained as image information without text, the first text information used to describe the image information will be used as the first target information.

[0033] If the first target information carried by the target display interface is image information including text, the second text information contained in the image information is taken as the first target information.

[0034] In some embodiments, obtaining the target input data from the user includes:

[0035] Get the data that the user inputs into the target input box, and obtain the target input data; or

[0036] Obtain the data corresponding to the target control touched by the user to obtain the target input data.

[0037] In some embodiments, determining a target area that meets the area conditions in the target display interface based on the area occupied by the second target information includes:

[0038] For each of the aforementioned information types, a fifth target information that satisfies the conditions of the second region block is determined from each of the second target information corresponding to the information type;

[0039] In the target display interface, a target area containing the area blocks corresponding to each of the fifth target information is determined;

[0040] The target region does not include the region block corresponding to the sixth target information that does not meet the conditions of the second region block.

[0041] In some embodiments, the second region block condition is selected from one of the following combinations:

[0042] Among the second target information corresponding to the information type, the second target information whose distance from the target edge of the target display interface is among the top N; wherein, N is not greater than the number of second target information contained in the information type;

[0043] Among the second target information corresponding to the information type, the second target information whose distance from the target region block is in the first M positions; wherein, M is not greater than the number of second target information contained in the information type; the target region block is the region block corresponding to the second target information that is most correlated with the target input data, or, the target region block is the region block corresponding to the second target information that is most correlated with the target input data among the second target information contained in the information type. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating the method for determining the region of this application;

[0046] Figure 2 This is a format block diagram of one embodiment of the target display interface of this application;

[0047] Figure 3 This is a schematic diagram showing the distribution of outlier and non-outlier regions in this application;

[0048] Figure 4 Determine the system's structural block diagram for the region covered in this application;

[0049] Figure 5 This is a structural block diagram of the electronic device of this application. Detailed Implementation

[0050] Various embodiments and features of this application are described herein with reference to the accompanying drawings.

[0051] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.

[0052] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0053] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0054] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.

[0055] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0056] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.

[0057] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.

[0058] To address the problems in the background technology, one solution is to train a visual model to partition the screenshot according to the application's UI (User Interface) design, such as rectangular areas for background color blocks and background images. Then, from top to bottom, a rectangular area large enough to hold the information of interest is selected from these rectangular areas. This area, along with all areas above it (mainly the title bar), is considered the valid area of ​​the information of interest on the current interface. This method is highly dependent on the UI design style of the application interface; the UI design style can affect the acquisition of the information of interest and the determination of the valid area. For example, for interfaces without background color blocks, it is impossible to determine the valid area; for interfaces with too many background color blocks, it is easy to cause excessive information exclusion; and for background images with cluttered content, the visual model is prone to calculation errors, leading to inaccurate determination of the valid area.

[0059] Unlike the existing solutions mentioned above, the area determination method provided in this application is not affected by the design style of the application interface, and can more accurately calculate the effective area of ​​the information that the user is interested in on the electronic device interface.

[0060] Combination Figure 1 This application provides a method for determining a region, which includes the following steps:

[0061] S10, Obtain the first target information set carried by the target display interface.

[0062] For example, the target display interface can carry a lot of information, which may include text information and / or image information, etc. Some or all of the target information can be obtained from the information carried by the target display interface. This target information can form a first target information set, which is a collection of multiple first target information sets.

[0063] Obtaining the first target information set carried by the target display interface can be achieved by obtaining the first text information and / or image information carried by the display interface.

[0064] If the information carried by the target display interface includes text information, then the first text information carried by the display interface can be directly used as the first target information set.

[0065] Each piece of information in the first target information set corresponds to a piece of text / image information in the target display interface.

[0066] In one embodiment, text / image information contained in the target display interface can be extracted by taking a screenshot of the target display interface and using OCR (Optical Character Recognition), while also extracting the boundary of the display area occupied by each piece of text information on the target display interface.

[0067] Currently, other methods can also be used to extract the first target information set carried by the target display interface, which will not be listed in this application.

[0068] In one embodiment, the hotel booking details page of a travel app is used as the target display page (as shown in the attached image). Figure 2 The following example (showing a virtual structure diagram) will be used for illustration. This interface is used to display the core information of the booked hotel to the user. The overall layout includes a basic order information area, a hotel information area, a room type information area, and an operation area, carrying two main categories of information: text and images.

[0069] The extraction process of the first target information set may include: extracting all information from the hotel order details interface to form the first target information set. The first target information set may include the following first target information: order number: "ORD20231008001" (text information); check-in time: "after 14:00 on 2023-10-15" (text information); check-out time: "2023-10-17". Before 12:00 (text information); Hotel Name: "XX International Hotel (City Center Branch)" (text information); Hotel Address: "No. 88, XX Road, XX District, XX City" (text information); Hotel Main Image: Panoramic view of the hotel exterior (image information); Room Type Name: "Deluxe King Room (Includes Breakfast for Two)" (text information); Room Layout Diagram: Interior layout of the room (image information); Order Status: "Confirmed" (text information); Operation Button: Icon and text for "View Details" (combined text and image information, which in this embodiment can be separated into independent text information "View Details" and image information "Details Icon"); Bath Center Name: "XX Bath Center" (text information); Bath Center Image: Interior view of the bath center (image information).

[0070] Each piece of text information is represented as an independent text block, with the text within the block revolving around a single information dimension and containing no cross-topic content. Similarly, each piece of image information is represented as an independent image. (See attached image.) Figure 2 As shown in "Area 1", the order number "ORD20231008001" is a separate text block containing only the unique order identifier, with no other redundant content; it is a single text message. Individual image information is represented as a separate image block, with the image content focusing on a single visual theme, without any mixed themes. (See attached image.) Figure 2 As shown in "Area 6", the hotel's main image is an independent image block, which only presents a panoramic view of the hotel's exterior and does not include other information such as room types and surrounding environment. It is a single image.

[0071] The content aggregation of a single piece of information is highly concentrated, meaning that content elements are closely related to the core theme, with no scattered information. Taking text information as an example... Figure 2 In "Area 9," the order status is "Confirmed," and the text only indicates the current processing status of the order, without any other additional information, indicating a very high degree of clustering. The clustering of different information depends on the degree of thematic relevance; for example, combining... Figure 2 Both "Area 4" (hotel name) and "Area 5" (hotel address) point to "basic hotel information," indicating a close thematic connection and high clustering. "Area 2" (check-in time) and "Area 6" (hotel main image) have weaker thematic connection and lower clustering, with the former pointing to booking time information and the latter to the hotel exterior.

[0072] S20, Obtain the target input data input by the user.

[0073] For example, in the process of recognizing a target display interface, it is generally necessary to obtain the user's intent. The target display interface is then recognized based on this intent to obtain a recognition result that meets the user's needs. Here, the target input data is the data that reflects the user's intent, which can be understood as the user's attention signal. Alternatively, the target input data can be understood as the user's indication of the recognition direction for the target display interface.

[0074] The target input data can be input by voice, input through a target input box, or selected from a fixed set of options.

[0075] The first target information set obtained in step S10 can also be input into the large language model. The large language model can obtain the first target information set input by the user and the target input data. Continuing with the embodiment of step S10, the target input data can be a hotel order.

[0076] S30, determine second target information related to the target input data in the first target information set.

[0077] For example, in conjunction with the embodiment of step S20, after the large language model obtains the target input data, it can evaluate each piece of first target information in the first target information set one by one, and evaluate the correlation between each piece of first target information in the first target information set and the target input data. If the first target information set includes one or more pieces of first target information related to the target input data, then the first target information related to the target input data can be determined as the second target information.

[0078] For example, if the target input data is a hotel order, the correlation between all the first target information in the first target information set and the hotel order can be evaluated. The name and picture of the bath center in the first target information set are not related to the hotel order, while other first target information is related to the hotel order. The first target information related to the hotel order is determined as the second target information.

[0079] S40, based on the area occupied by the second target information in the target display interface, determine a target area that meets the area conditions in the target display interface.

[0080] The target region that satisfies the specified regional conditions is a valid region in the target display interface that can represent the target input data.

[0081] For example, each piece of second target information has a defined area on the target display interface. These areas can be independent, overlapping, or partially overlapping. Once the second target information is determined, the area it occupies on the target display interface is also determined. Target areas that meet the set area conditions can be determined on the target display interface.

[0082] In determining the second target information, this embodiment does not rely on the UI design style of the target display interface, thus enabling more accurate determination of the second target information and consequently, more accurate determination of the target area. The target area is the effective area in the target display interface that can represent the target input data. The information displayed within this effective area will not be interfered with by irrelevant information, improving the accuracy of extracting the information of interest.

[0083] In one feasible embodiment, step S30 may include:

[0084] S301, Based on the target input data, determine the target information type related to the target input data.

[0085] For example, if the target input data is "hotel order", then the target information types related to "hotel order" can be determined as: 1. Hotel name, 2. Hotel address, 3. Check-in time, 4. Check-out time, 5. Price, 6. Room type, 7. Breakfast, etc.

[0086] The target information type of the target input data can be provided by the user to the large language model, determined by the large language model based on the user's input information, or determined by the large language model after the user provides the information type, which is then analyzed by the target input data and the user's input information type.

[0087] For example, a large language model can have a location or prompt for inputting target data, such as an input box labeled "What the user wants," which can accept either verbal or text input. At this prompt location, the user can enter "Record shopping list," and the model will provide the prompt "Please consider the types of information you might need." The large language model can then output the target information types, such as item, quantity, unit price, and total price. Alternatively, if the user selects a recommended action, the large language model first selects potentially relevant actions from the developer's pre-defined action set based on the first set of target information on the target display interface. For instance, if the large language model determines that the current display interface may contain hotel booking information, it might select "Add hotel information to schedule" from the pre-defined action set. The parameters of this pre-defined action item can include the prompt: "Please consider what information the hotel booking includes." The large language model can then output the target information types, such as hotel name, hotel address, room type, check-in time, and check-out time.

[0088] The type of target information determined based on the target input data can help to accurately determine the second target information in the future.

[0089] S302, determine second target information whose information type matches the target information type in the first target information set.

[0090] For example, each piece of first target information in the first target information set is evaluated one by one to determine which of the target information types it may contain, or which one, or no target information type is included.

[0091] For example, if the first target information includes the Holiday Inn X, located at No. X, Street X, City X, Province X, then the information type of this first target information matches the hotel name and hotel address in the target information type, and this first target information can be identified as the second target information. In this way, the second target information identified from the first target information set is more accurate.

[0092] In one possible implementation, step S40 may include:

[0093] S401, based on the area block occupied by each of the second target information in the target display interface, determine the third target information that satisfies the first area block condition from each of the second target information.

[0094] S402, in the target display interface, a target region containing the corresponding area blocks of each of the third target information is determined; wherein, the target region does not contain the area blocks corresponding to the fourth target information in each of the second target information that do not meet the conditions of the first area block.

[0095] or

[0096] S403, respectively obtain the distribution information of the region blocks corresponding to each of the second target information in each first candidate region.

[0097] S404, determine the target region in each of the first candidate regions whose distribution information meets the distribution conditions.

[0098] For example, when the second target information consists of multiple pieces of information, the second target information can be a set of third target information and fourth target information. The third target information consists of the second target information that satisfies the conditions of the first region block, and the fourth target information consists of the second target information that does not satisfy the conditions of the first region block.

[0099] The third target information may be one or more. The target area is defined in the target display interface as the region enclosed by the blocks corresponding to all the third target information. The target area does not include the block occupied by the fourth target information on the target display interface.

[0100] Each second target information occupies a corresponding area block on the target display interface. The area block occupied by each second target information on the target display interface is compared with the first area block condition. If any second target information occupies an area block that satisfies the first area block condition, then the second target information that satisfies the first area block condition is determined as the third target information; otherwise, the second target information that does not satisfy the first area block condition is determined as the fourth target information.

[0101] Among them, the third target information that meets the conditions of the first region block indicates that the region block corresponding to the third target information is a non-outlier region block, and the third target information that does not meet the conditions of the first region block indicates that the region block corresponding to the third target information is an outlier region block.

[0102] In practical applications, the criteria for determining whether a region is an outlier vary depending on the algorithm used. For example, the deviation between the center coordinates and the average center coordinates of a region block is calculated. If the distance between the center coordinates and the average center coordinates is greater than a distance threshold, the region block is considered an outlier. Correspondingly, the first region block condition indicates that the distance between the center coordinates and the average center coordinates of the region block corresponding to the third target information is not greater than the distance threshold.

[0103] After determining the third and fourth target information, the target area can be determined based on the areas occupied by the third and fourth target information on the target display interface. For example, the target area can be defined as the area comprised of all the areas occupied by the third target information on the target display interface. The fourth target information does not meet the first area condition, so the target area does not include the area occupied by the fourth target information on the target display interface. This eliminates areas occupied by irrelevant information on the target display interface, improving the accuracy of the target display area determination.

[0104] Alternatively, the distribution information of the region blocks corresponding to each of the second target information in each of the first candidate regions can be obtained respectively.

[0105] The first candidate region can be a region randomly defined on the target display interface according to different boundary sizes, or it can be a region predicted as a potential target display region after the target display interface has been identified. For example, the first candidate region can be the left column (or other area) of the target display interface. The specific information can be the coverage rate and / or number of the region blocks corresponding to each second target information in each first candidate region. Target regions whose distribution information meets the distribution conditions are determined within each first candidate region. The distribution conditions can be a coverage rate exceeding a first set threshold (e.g., coverage rate greater than 80%) and / or the number of region blocks corresponding to the second target information contained in each first candidate region exceeding a second set threshold (greater than 3). Region blocks corresponding to the second target information that do not meet the specific conditions in the first candidate region can be removed. The remaining region blocks corresponding to the second target information that meet the specific conditions in the first candidate region can accurately represent the target region, improving the accuracy of target region determination.

[0106] Of course, in practical applications, the distribution conditions can vary depending on the target display interface. For example, for sparsely distributed applications, the coverage / number will be relatively small; for applications with more concentrated distribution, the coverage / number will be relatively large.

[0107] In one implementable instance, step S401 may include:

[0108] S4011, Obtain the distance information set corresponding to each of the second target information.

[0109] S4012, based on the distance information set corresponding to each of the second target information, determine the third target information that satisfies the regional block distance condition from each of the second target information.

[0110] The distance information set includes distance information between the region block corresponding to the second target information and other region blocks corresponding to the second target information. Combined with... Figure 3The region block corresponding to the second target information representation that satisfies the region block distance condition is a non-outlier region block; the region block corresponding to the second target information representation that does not satisfy the region block distance condition is an outlier region block.

[0111] For example, in the target display interface, the sum of the distances from the center point of each region to the center points of all other regions is determined.

[0112] The sum of all distances is normalized based on the minimum sum of distances. In other words, the minimum sum of distances can be used as the unit 1 to normalize the sum of all distances.

[0113] The region corresponding to the sum of distances exceeding a set threshold after normalization is identified as an outlier region. This means that significant faults exceeding an empirical threshold in the normalized sorting queue can be identified, and black blocks outside these faults are considered outliers. Outlier regions within the area occupied by the second target information on the target display interface are then removed.

[0114] Alternatively, multiple real-world examples can be used to generate manually marked views with red boxes. These views can be white-background black-block images. A visual model can be trained using these multiple views, and this visual model can be used to exclude outlier regions.

[0115] In another implementable instance, step S401 may include:

[0116] S4013, mark the area blocks occupied by each of the second target information in the target display interface.

[0117] For example, the area occupied by the second target information can be directly marked on the target display interface to clearly indicate the area occupied by the second target information. For instance, if there is no red information on the target display interface, a red border or red block can be used to mark the area occupied by the second target information on the target display interface. The border can mark the boundary of the area occupied by the second target information, while the red block can mark the entire area occupied by the second target information.

[0118] S4014, delete the region block that meets the outlier condition in the target display interface to obtain the region block corresponding to the third target information in the target display interface.

[0119] For example, in the target display interface, outlier blocks in the area occupied by the second target information are deleted, and the remaining area blocks on the target display interface are the area blocks corresponding to the third target information.

[0120] or

[0121] S4015, mark the area blocks occupied by each of the second target information in the comparison interface corresponding to the target display interface.

[0122] For example, on a white image with the same width and height as the target display interface (the target display interface is rectangular), the area occupied by the second target information is painted black, or the boundaries of the area occupied by the second target information are marked. The "white background with black blocks" image is roughly as follows: Figure 2 As shown, the area outside the red box is the outlier area that will be excluded in the next step.

[0123] Alternatively, the area occupied by the second target information can be painted black on a white image whose height and width are not set at a 1:1 ratio with the target display interface.

[0124] S4016, delete the region block that meets the outlier condition in the comparison interface to obtain the region block corresponding to the third target information in the comparison interface.

[0125] For example, the outlier regions in the comparison interface are deleted, and the remaining regions in the comparison interface are the regions corresponding to the third target information in the comparison interface.

[0126] If the comparison interface is not set at a 1:1 ratio with the target display interface, the comparison interface can be enlarged (or reduced) according to the scaling ratio. At this time, the remaining area block on the comparison interface is the area block corresponding to the third target information in the comparison interface.

[0127] This embodiment can more clearly identify outlier regions and remove them to accurately obtain non-outlier regions.

[0128] In one possible implementation, step S402 may include:

[0129] S4021, based on the position of each of the third target information corresponding area blocks in the target display interface, determine the area boundary parameters of the area occupied by each of the third target information corresponding area blocks in the target display interface.

[0130] For example, each region block of the third target information has a corresponding position in the target display interface. Based on this position information, the boundary of the total area occupied by all regions corresponding to the third target information in the target display interface can be determined. For example, the coordinates of the region blocks in four directions (top, left, bottom, and right) can be read from the image excluding outlier regions.

[0131] S4022, Based on the region boundary parameters, the target region is obtained.

[0132] Based on the region boundary parameters obtained in step S4021, namely the four coordinate sizes, the target region is determined; that is, the region enclosed by the four coordinates is the target region. Here, the four coordinates can enclose a rectangular region, which can accurately represent the target region.

[0133] In some embodiments, if the first target information carried by the display interface is obtained as image information without text, then the first text information used to describe the image information can be used as the first target information.

[0134] For example, if the target display interface carries image information, and the image information does not include text information, then the acquired image information can be described first, with the first text information used to describe the image information being the first target information. For instance, an existing visual mini-model can be used to extract the image carried by the target display interface from a screenshot of the target display interface (other methods can also be used to obtain image information). Then, the image is input into an AI model, allowing the AI ​​model to provide a textual description of the image. For example, if the input image is a picture of a yellow dress, it can be described as a yellow floral print skirt, with the specific textual description corresponding to the input image.

[0135] Of course, the AI ​​model can also output multiple text descriptions for confirmation and selection, or it can output a single text description while providing corresponding options for further improvement, such as prompts like "change description".

[0136] Finally, a text message used to describe the image information is selected as the first text message, which can be used as the first target information.

[0137] If the first target information carried by the target display interface is image information including text, then the second text information contained in the image information can be used as the first target information.

[0138] For example, if the image information carried by the target display interface includes text information, and the text information can be second text information, then the second text information can be used as the first target information.

[0139] For example, if the image information is a picture of a yellow dress, and the picture includes text such as "yellow floral print skirt", then "yellow floral print skirt" can be considered as the second text information used to describe the image information, and the second text information can be used as the first target information.

[0140] The target display interface can be captured by taking a screenshot, and then combined with optical character recognition technology to obtain the first set of target information carried by the target display interface. Here, the target display interface can be the entire display interface of the electronic device screen, or a part of the entire display interface.

[0141] The method of taking a screenshot varies depending on the electronic device and / or the app logged into it. For example, if the electronic device is a smartphone, the target display interface can be captured by using the buttons on the smartphone. If the electronic device is a desktop or laptop computer, the target display interface can be captured by using keyboard shortcuts. Of course, it is understood that other methods can also be used to capture the target display interface; this is merely an example and does not constitute a limitation on the scope of the claims.

[0142] The first target information set obtained using this embodiment can comprehensively acquire the information carried by the target display interface, avoid information omissions, and ensure the accuracy of target area determination.

[0143] In some embodiments, data input by the user into the target input box can be obtained to obtain the target input data; or data corresponding to the target control touched by the user can be obtained to obtain the target input data.

[0144] For example, a large language model can have a target input box where users can directly enter target input data. This input can be done via voice or text. Alternatively, the large oracle model can provide a voice prompt (e.g., "Please enter the target input data") instead of a specific target input box, allowing users to provide their target input data via voice after hearing the prompt.

[0145] After obtaining the initial target information set, the big oracle model can provide a target input box or the aforementioned voice prompts for the input of target data. For blind users, elderly users, and teenagers, voice input can be selected; of course, voice input is also suitable for users of other age groups.

[0146] Alternatively, after recognizing the target display interface, the large language model can recommend some operation items to the user. The user can touch these recommended operation items, that is, the user can select all or some of the operation items recommended by the large language model through touch. The selected operation item can send the corresponding target input data to the large oracle model.

[0147] For example, in the Big Prophecy model, after obtaining the first target information set, the Big Prophecy model can provide the user with a microphone icon and / or a target input box. The user can click the microphone icon and input "Please help me find a 128G Apple 16 phone" via voice, or click the target input box and input "128G Apple 16 phone".

[0148] This embodiment provides multiple input methods for target input data, allowing users to choose according to their needs and improving the user experience.

[0149] In some embodiments, determining a target area that meets the area conditions in the target display interface based on the area occupied by the second target information includes:

[0150] For each of the aforementioned information types, a fifth target information that satisfies the conditions of the second region block is determined from each of the second target information corresponding to the information type.

[0151] In the target display interface, a target region is determined that contains the area blocks corresponding to each of the fifth target information. This target region does not contain the area blocks corresponding to the sixth target information that do not meet the conditions of the second area block.

[0152] The second region block condition is selected from one of the following combinations:

[0153] The first one is: among the second target information corresponding to the information type, the second target information whose distance from the target edge of the target display interface is in the first N positions; where N is not greater than the number of second target information contained in the information type, and N is a positive integer greater than or equal to 1.

[0154] For example, the above "N" can be determined in the following way:

[0155] For the results of the second target information provided by the large language model, calculate the average number of hits for non-zero information types and round down. For example, if there are 5 first target information items containing the hotel name, 2 containing the hotel address, 2 containing the check-in time, and 2 containing the check-out time in the first target information set, and no other information type has a corresponding first target information item in the first target information set, then the average number of hits for information types is N = (5 + 2 + 2 + 2) / 4 = 2.

[0156] When N is 2, among the second target information corresponding to the information type, the two second target information whose distance from the top edge of the target display interface is in the first two positions are identified as the fifth target information, and the second target information whose distance from the top edge of the target display interface is in the third position and beyond is identified as the sixth target information.

[0157] Of course, the target edge of the target display interface can also be the far left of the target display interface. For example, for foldable screen phones with multiple displays, when the foldable phone is in landscape display mode, the effective area is more likely to be located on the left side of the target display interface. In this case, the target edge of the target display interface can be determined as the far left of the target display interface.

[0158] In different application objects or application scenarios, the target edge of the target display interface can also be other edges of the target display interface. This is only used as an example and does not constitute a limitation on the scope of protection of the claims.

[0159] The second is: among the second target information corresponding to the information type, the second target information whose distance from the target region block is in the top M positions; wherein M is not greater than the number of second target information contained in the information type. The target region block is the region block corresponding to the second target information most strongly associated with the target input data, or, the target region block is the region block corresponding to the second target information most strongly associated with the target input data among the second target information contained in the information type.

[0160] For example, the method for determining "M" can be the same as the method for determining "N" in the previous embodiments, or other methods can be used to determine M. It should be noted that M and N can be the same or different. M is not greater than the number of times the information type contains the second target information. For example, if the number of times the information type contains the second target information is 5, then M is a positive integer not greater than 5.

[0161] Within each second target information corresponding to the information type, a target region block is determined. This target region block is the region block among the region blocks corresponding to each second target information that has the strongest correlation with the target input data. For example, if M is 2 and the number of second target information items is 10, then two region blocks with the strongest correlation to the target input data are determined from the ten region blocks corresponding to the second target information items as the target region blocks.

[0162] Alternatively, the target region block is the region block corresponding to the second target information that is most strongly related to the target input data among the second target information contained in the information type.

[0163] For example, if the number of second target information is 10, the information type contains 4 second target information, and M is 2, then the two second target information blocks that are most strongly related to the target input data can be determined as target region blocks from the four second target information blocks.

[0164] This application also provides a region determination system, combined with Figure 4 The system for determining this area includes:

[0165] The acquisition module is configured to acquire the first target information set carried by the target display interface and the target input data input by the user.

[0166] The determination module is configured to determine second target information related to the target input data in the first target information set and the area occupied by the second target information in the target display interface, and to determine a target area that meets the area conditions in the target display interface.

[0167] The target region that satisfies the specified regional conditions is a valid region in the target display interface that can represent the target input data.

[0168] An electronic device, combined with Figure 5 Electronic devices include displays and processors.

[0169] The display is used to carry the first target information set through the target display interface.

[0170] The processor is configured to acquire a first set of target information carried by the target display interface of the display, and to acquire target input data input by the user. The processor is further configured to determine second target information related to the target input data within the first set of target information, and, based on the area occupied by the second target information on the target display interface, determine a target area in the target display interface that satisfies a certain area condition. The target area that satisfies the area condition is a valid area in the target display interface that can characterize the target input data.

[0171] Electronic devices can use the above-mentioned area determination method to determine the effective area on the target display interface.

[0172] Furthermore, one embodiment of this application also provides a storage medium storing a computer program that, when executed by a processor, implements the region determination method described above. It should be understood that the various solutions in this embodiment have the corresponding technical effects in the above method embodiments, and will not be repeated here.

[0173] Furthermore, embodiments of this application also provide a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions that, when executed, cause at least one processor to perform a region determination method such as those described in the embodiments above.

[0174] It should be noted that the computer storage medium in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access storage media (RAM), read-only storage media (ROM), erasable programmable read-only storage media (EPROM or flash memory), optical fibers, portable compact disk read-only storage media (CD-ROM), optical storage media, magnetic storage media, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program configured for use by or in connection with an instruction execution system, system, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, antenna, optical fiber, RF, etc., or any suitable combination thereof.

[0175] Furthermore, those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0176] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0177] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0178] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0179] The foregoing has described in detail several embodiments of this application, but this application is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications based on the concept of this application, and all such variations and modifications should fall within the scope of protection claimed in this application.

Claims

1. A method for determining a region, comprising: Obtain the first set of target information displayed on the target interface; Obtain the target input data from the user; Determine second target information related to the target input data from the first target information set; Based on the area occupied by the second target information on the target display interface, a target area that meets the area conditions is determined in the target display interface. The target region that satisfies the specified regional conditions is a valid region in the target display interface that can represent the target input data.

2. The method according to claim 1, wherein determining the second target information related to the target input data in the first target information set comprises: Based on the target input data, determine the type of target information related to the target input data; In the first set of target information, determine the second target information whose information type matches the target information type.

3. The method according to claim 1, The step of determining a target area that meets the area conditions in the target display interface based on the area occupied by the second target information includes: Based on the area block occupied by each of the second target information in the target display interface, third target information that satisfies the conditions of the first area block is determined from each of the second target information; In the target display interface, a target region is determined that contains the corresponding area blocks of each of the third target information; wherein, the target region does not contain the area blocks corresponding to the fourth target information in each of the second target information that do not meet the conditions of the first area block; or The distribution information of each region block corresponding to each second target information in each first candidate region is obtained respectively; In each of the first candidate regions, a target region whose distribution information satisfies the distribution conditions is determined.

4. The method according to claim 3, wherein determining third target information that satisfies the first region block condition from among the second target information based on the region block occupied by each of the second target information in the target display interface includes: Obtain the distance information set corresponding to each of the second target information; Based on the distance information set corresponding to each of the second target information, third target information that meets the regional block distance condition is determined from each of the second target information; The distance information set includes distance information between the region block corresponding to the second target information and each other region block corresponding to the second target information; The second target information that does not meet the region block distance condition indicates that the corresponding region block is an outlier region block.

5. The method according to claim 3, wherein, based on the area block occupied by each of the second target information in the target display interface, third target information satisfying the first area block condition is determined from each of the second target information, including: Mark the area blocks occupied by each of the second target information in the target display interface; Delete the region block that meets the outlier condition in the target display interface to obtain the region block corresponding to the third target information in the target display interface; or In a comparison interface corresponding to the target display interface, mark the area block occupied by each of the second target information; Delete the region block that meets the outlier condition in the comparison interface to obtain the region block corresponding to the third target information in the comparison interface.

6. The method according to claim 3, Determining the target region containing the corresponding area blocks of each of the third target information in the target display interface includes: Based on the position of each region block corresponding to the third target information in the target display interface, determine the region boundary parameters of the region occupied by each region block corresponding to the third target information in the target display interface. The target region is obtained based on the region boundary parameters.

7. The method according to claim 1, further comprising: If the first target information carried by the target display interface is obtained as image information without text, the first text information used to describe the image information will be used as the first target information. If the first target information carried by the target display interface is image information including text, the second text information contained in the image information is taken as the first target information.

8. The method according to claim 1, wherein obtaining the target input data input by the user comprises: Get the data that the user inputs into the target input box to obtain the target input data; or Obtain the data corresponding to the target control touched by the user to obtain the target input data.

9. The method according to claim 2, wherein determining a target area satisfying the area condition in the target display interface based on the area occupied by the second target information in the target display interface comprises: For each of the aforementioned information types, a fifth target information that satisfies the conditions of the second region block is determined from each of the second target information corresponding to the information type; In the target display interface, a target area containing the area blocks corresponding to each of the fifth target information is determined; The target region does not include the region block corresponding to the sixth target information that does not meet the conditions of the second region block.

10. The method according to claim 9, The second region block condition is selected from one of the following combinations: In each of the second target information corresponding to the information type, the second target information whose distance to the target edge of the target display interface is among the top N; wherein... N is not greater than the number of the second target information contained in the information type; Among the second target information corresponding to the information type, the second target information whose distance from the target region block is in the first M positions; wherein, M is not greater than the number of second target information contained in the information type; the target region block is the region block corresponding to the second target information that is most correlated with the target input data, or, the target region block is the region block corresponding to the second target information that is most correlated with the target input data among the second target information contained in the information type.

Images