An AI agent-oriented data flywheel labeling quality evaluation system

Abstract

The present application relates to the technical field of data labeling evaluation, and more particularly to a data flywheel labeling quality evaluation system for AI agents. Through the cooperative operation of the information acquisition module, the interface segmentation module, the information block determination module, the adhesion analysis module, the adhesion removal module and the flywheel labeling module, the target interface is segmented based on the visual cycle degree, the easily adhering sub-interfaces are further calibrated according to the boundary gap loss characteristics and the block height deviation characteristics, the aggregation extraction or independent extraction is adopted according to the relative position relationship of each easily adhering sub-interface, the adhesion sentences are removed based on the semantic correlation degree between adjacent sentences in the same information block in the extracted text, and finally the text not removed is executed for labeling and input into the data flywheel training set, so as to improve the accuracy of information labeling and the reliability of the agent in the retrieval scene.

Description

Technical Field

[0001] This invention relates to the field of data annotation and evaluation technology, and in particular to a data flywheel annotation quality evaluation system for AI agents. Background Technology

[0002] With the rapid development of artificial intelligence and AI agent technology, the application of AI agents in the fields of information extraction and structured processing of retrieval results is becoming increasingly in-depth. The quality of annotation is not only closely related to the reliability of training samples in the data flywheel, but also closely linked to the accuracy of annotation boundaries and the precision of information extraction. High-quality labeled data and a stable and reliable data flywheel cycle have become indispensable components in ensuring that AI agents can achieve accurate information extraction and continuous optimization capabilities in retrieval scenarios.

[0003] Chinese Patent Publication No. CN119862427A discloses a data annotation method and related apparatus. The method involves extracting defect data from data to be annotated to obtain defect data; determining the type of defect data based on the similarity between the defect data and annotated data in a defect retrieval database; identifying the annotated data in the defect retrieval database as an annotated defect data type; and annotating the data to be annotated based on the type of defect data to obtain annotated data. In this embodiment, even if the defect data extracted from the data to be annotated differs from the annotated defect data, the type of defect data can still be determined based on the similarity between the defect data and the annotated data in the defect retrieval database.

[0004] However, the following problems still exist in the existing technology.

[0005] During the process of the agent annotating the search results page, the visual features of adjacent information blocks are highly similar, which causes the agent to make adhesion errors when annotating the boundaries. This leads to training errors in subsequent flywheel iterations, thereby reducing the accuracy of structured information extraction based on the annotation results and the reliability of the agent's operation in the search scenario. Summary of the Invention

[0006] To address this issue, the present invention provides a data flywheel annotation quality assessment system for AI agents, which overcomes the problem in the prior art where, when the agent annotates the search results page, the visual features of adjacent information blocks are highly similar, causing the agent to produce adhesion errors when annotating the boundaries, thereby reducing the accuracy of structured information extraction based on the annotation results and the reliability of the agent's operation in the retrieval scenario.

[0007] To achieve the above objectives, this invention provides a data flywheel annotation quality assessment system for AI agents, comprising:

[0008] The information acquisition module is used to acquire the target interface retrieved by the intelligent agent, as well as the page visual data of the target interface;

[0009] The interface segmentation module is used to segment the target interface based on visual periodicity to obtain several sub-interfaces. The visual periodicity is determined based on the vertical spacing of the most frequent occurrence of adjacent text density peak lines, and the text density peak lines are determined based on the pixel density of the horizontal strips satisfying the baseline density condition.

[0010] The information block determination module is used to determine several information blocks based on the pixel spacing in the sub-interface;

[0011] The adhesion analysis module is used to identify easily adhered sub-interfaces based on the boundary gap missing feature and the block height deviation feature. The boundary gap missing feature is determined based on the ratio of the average pixel density of the gap region between adjacent information blocks to the average pixel density within the information block. The block height deviation feature is determined based on the variance of the deviation rate between the height of each information block and the visual periodicity within the sub-interface.

[0012] The adhesion removal module is used to determine the extraction method based on the relative positional relationship of each easily adhesive sub-interface, and to determine whether to remove a sentence based on the semantic correlation between adjacent sentences in the extracted text; the extraction method includes aggregate extraction and independent extraction, and the extracted text is determined based on the extraction method.

[0013] The flywheel annotation module is used to annotate the text that has not been removed from the target interface and input the annotation results into the flywheel training set.

[0014] Furthermore, the interface segmentation module includes,

[0015] Interface segmentation units are used to segment the target interface starting from the first text density peak line and using visual periodicity as the segmentation interval to obtain several sub-interfaces.

[0016] Furthermore, the interface segmentation module also includes,

[0017] The vertical spacing determination sub-unit is used to determine the vertical spacing between adjacent lines of text density peaks;

[0018] The visual periodicity determination subunit is used to determine the visual periodicity by identifying the vertical spacing with the highest frequency among the vertical spacing of adjacent text density peak lines.

[0019] Furthermore, the interface segmentation module also includes,

[0020] The horizontal strip division sub-unit is used to divide the target interface into several horizontal strips from top to bottom;

[0021] The pixel density determination subunit is used to determine the pixel density within each horizontal strip;

[0022] The peak row determination sub-unit is used to determine the horizontal strip as the text density peak row when the pixel density of the horizontal strip meets the reference density condition.

[0023] The baseline density condition is that the pixel density of the horizontal strip exceeds the baseline density value, and the difference between the pixel density of the horizontal strip and the pixel density of its adjacent horizontal strip exceeds the baseline difference value.

[0024] Furthermore, the information block determination module includes,

[0025] The information block defines the sub-unit, which is used to group pixels in the sub-interface whose pixel spacing is lower than the reference spacing value into the same information block.

[0026] Furthermore, the adhesion resolution module includes,

[0027] The easy-adhesion calibration sub-unit is used to calibrate sub-interfaces with boundary gap missing features exceeding the reference gap missing value or block height deviation features exceeding the reference deviation value as easy-adhesion sub-interfaces.

[0028] Furthermore, the adhesion resolution module also includes,

[0029] The gap region determination sub-unit is used to determine the gap region between adjacent information blocks;

[0030] The boundary gap missing feature determination subunit is used to determine the ratio of the average pixel density of the gap region to the average pixel density within the information block as the boundary gap missing feature.

[0031] Furthermore, the adhesion resolution module also includes,

[0032] The information block height determination subunit is used to determine the information block height by the absolute value of the difference between the maximum and minimum vertical coordinates of each pixel within the information block.

[0033] The deviation rate determination subunit is used to determine the deviation rate of each information block as the ratio of the absolute value of the difference between the height of each information block and the visual periodicity to the visual periodicity.

[0034] The block height deviation feature determination sub-unit is used to determine the variance of the deviation rate of information blocks within the sub-interface as the block height deviation feature.

[0035] Furthermore, the adhesion removal module includes,

[0036] The average distance determination sub-unit is used to determine the average distance between the center points of each easily sticky sub-interface;

[0037] The extraction method determines the sub-unit, which is used to determine aggregate extraction when the average distance does not exceed the benchmark distance value, and to determine independent extraction when the average distance exceeds the benchmark distance value;

[0038] Among them, aggregate extraction involves extracting the entire region enclosed by the outer boundaries of each easily sticky sub-interface, while independent extraction involves extracting the easily sticky sub-interfaces separately.

[0039] Furthermore, the adhesion removal module includes,

[0040] The extracted text determination subunit is used to determine the extracted text based on the extraction method;

[0041] The semantic association calculation subunit is used to determine the semantic association degree between adjacent sentences in the extracted text;

[0042] The statement removal subunit is used to remove the statement that is less than the baseline relevance value in the semantic relevance of adjacent statement pairs within the same information block.

[0043] Compared with existing technologies, this invention utilizes the coordinated efforts of an information acquisition module, an interface segmentation module, an information block determination module, an adhesion analysis module, an adhesion removal module, and a flywheel annotation module. The interface segmentation module provides the basis for information block boundaries. During the annotation process, the adhesion analysis module judges the annotation quality of each sub-interface based on boundary gap loss features and block height deviation features. The easily adhered sub-interfaces identified are those with abnormal quality. The adhesion removal module extracts these easily adhered sub-interfaces and, within the same information block, locates and removes adhered statements based on the semantic correlation between adjacent statements. The retained text is the text with acceptable annotation quality. The flywheel annotation module performs annotation on the retained text and inputs the annotation results into the flywheel training set for subsequent iterative training of the AI ​​agent. Therefore, this embodiment completes the process judgment and control of annotation quality during the annotation process and forms a closed loop of controllable annotation data through the flywheel training set.

[0044] In particular, this invention segments the target interface based on visual periodicity, resulting in several sub-interfaces. When the AI ​​agent annotates the search results page, the page repeatedly displays multiple similar information blocks in a list format, such as several company cards in the company information query results. The arrangement of each information block on the page is highly regular, and the vertical height of each individual information block is basically the same. However, since the page itself does not use explicit boundary lines to separate the information blocks, but relies on white space, line spacing, and layout rhythm to distinguish adjacent content, the AI ​​agent is prone to causing the annotation boundaries to shift or stick together between adjacent information blocks due to the lack of a stable spatial reference.

[0045] Therefore, this invention segments the target interface based on visual periodicity to obtain several sub-interfaces. The visual periodicity is based on the vertical spacing of adjacent text density peak lines, selecting the vertical spacing with the highest frequency as the visual periodicity. The text density peak lines correspond to the positions with the densest text pixels in each information block, such as the title line or key field line. The vertical spacing of these lines, which repeats in different information blocks, constitutes a certain regularity in the arrangement of page information blocks. In this invention, after dividing the target interface into horizontal strips from top to bottom and determining their pixel density, the horizontal strips, carrying characters, exhibit a higher density than blank areas, forming identifiable text density peak lines. Although the vertical spacing between adjacent peak lines may fluctuate due to individual line breaks, paragraph spacing, etc., most vertical spacing remains concentrated around the same value. Therefore, the visual periodicity reflects the reference height of the repeated arrangement of information blocks in the search results page. By segmenting the target interface based on this visual periodicity, the page can be divided into multiple sub-interfaces according to its internal rhythm. Each sub-interface corresponds to a reference height of an information block, reducing the risk of annotation boundary offset caused by the high similarity of page visual features. This enables the system to adapt to different formats of search result pages, providing a stable and reliable spatial benchmark for subsequent adhesion analysis and annotation quality control, thereby improving the accuracy of information annotation and the reliability of the agent in the retrieval scenario.

[0046] In particular, this invention determines several information blocks based on the pixel spacing within the sub-interface. In actual enterprise information retrieval results pages, text within the same information block is arranged in a continuous and clustered manner. For example, fields such as enterprise name, registered capital, and establishment date are vertically adjacent with only a small line spacing between them, while there are significantly larger gaps between adjacent information blocks. Furthermore, due to differences in specific content, the number of fields, summary length, and line break positions within different information blocks are not entirely consistent, making it impossible for the sub-interface obtained solely by visual periodicity segmentation to perfectly match the true boundaries of each information block. Therefore, this invention, based on sub-interface segmentation, further determines information blocks based on the pixel spacing between text. The pixel spacing objectively reflects the degree of clustering or dispersion of text arrangement; the pixel spacing within the same information block is generally small, while the pixel spacing corresponding to the gaps between information blocks is significantly larger. This invention groups text pixels with a pixel spacing lower than a benchmark spacing value into the same information block, and uses the structural features of the page itself to accurately define the information blocks, thereby avoiding boundary adhesion caused by the high similarity of visual features of adjacent information blocks. It also provides a reliable information block basis for subsequent adhesion analysis and annotation quality control, thereby improving the accuracy of information annotation and the reliability of the agent in the retrieval scenario.

[0047] In particular, this invention identifies easily sticky sub-interfaces based on the characteristics of missing boundary gaps and block height deviations. After obtaining information blocks with reliable internal boundaries for each sub-interface through visual periodicity segmentation and pixel spacing definition, each information block already has an independent boundary. However, this boundary may still have residual adhesion caused by slight boundary crossings of adjacent information blocks. Residual adhesion manifests as one or more lines of text that do not belong to the current information block mixed in near the boundary. Moreover, these texts are highly consistent with the current information block in terms of format and content, making them difficult to detect with conventional semantic checks.

[0048] Therefore, this invention uses boundary gap missing features and block height deviation features to quantitatively verify the boundaries of information blocks at the structural level. The boundary gap missing feature is used to check whether the boundary falls within a blank gap. The gap area between normal information blocks should be essentially free of text pixels, with its average pixel density much lower than that inside the information block. If the boundary shifts, text from adjacent information blocks will intrude into the gap area, increasing the average pixel density of the gap area and its ratio to the average pixel density inside the information block. The block height deviation feature uses visual periodicity as a regularity benchmark for the repeated arrangement of page information blocks to determine the variance of the deviation rate between the height of each information block within the sub-interface and the visual periodicity. The height of normal information blocks should be concentrated near the visual periodicity, with a small variance; when adhesion occurs, the height of incorrectly merged information blocks will significantly deviate from the visual periodicity, leading to an increase in variance. The boundary gap missing feature detects whether the boundary is intruded by text locally, while the block height deviation feature detects whether the information block has undergone height deformation overall. Based on this, when any feature exceeds the corresponding benchmark value, the sub-interface is marked as a sub-interface prone to adhesion. Both capture the abnormal performance of adhesion in different dimensions. Accordingly, this invention can accurately screen out sub-interfaces with boundary adhesion risk before the subsequent adhesion removal process, providing targeted extraction basis for aggregation extraction or independent extraction and semantic association removal, while avoiding over-processing of normal sub-interfaces, thereby improving the accuracy of information labeling and the reliability of the agent in the retrieval scenario.

[0049] In particular, this invention determines the extraction method based on the relative positional relationship of each easily linked sub-interface and determines whether to remove sentences based on the semantic correlation between adjacent sentences within the extracted text. After identifying easily linked sub-interfaces using boundary gap missing features and block height deviation features, the system has determined which areas are at risk of boundary intrusion by adjacent content. However, the distribution of these easily linked sub-interfaces on the page does not follow the same pattern. Some easily linked sub-interfaces appear in clusters due to regional layout defects in the page template itself, with their centers close to each other; others appear isolated due to local content anomalies, with their centers far apart. If these two situations are not distinguished, the possible connections between sub-interfaces will be severed during text extraction, making it impossible to recover the boundary of the area as a whole; or if isolated easily linked sub-interfaces are included in a whole area for extraction, a large number of normal information blocks will be involved in subsequent semantic analysis, causing unnecessary risk of false removal.

[0050] Therefore, this invention calculates the average distance between the center points of each easily sticky sub-interface. Concentrated distributions where the average distance does not exceed a benchmark distance are identified as aggregated extraction, allowing the entire region enclosed by the outer boundaries of each easily sticky sub-interface to be extracted as a whole, thus preserving structural relationships within the region. Dispersed distributions where the average distance exceeds the benchmark distance are identified as independent extraction, allowing for separate extraction of each easily sticky sub-interface, thus avoiding excessive interference with normal regions. After extraction, in the extracted text, because statements within normal information blocks revolve around the same information object, the semantic connection between adjacent statements is natural and the correlation is high. However, sticky statements are content that crosses boundaries from adjacent information blocks; they lack the coherence within the same information block as the preceding statement, resulting in a significant decrease in semantic correlation. This embodiment determines the semantic correlation between adjacent statements within the extracted text. Utilizing the semantic break between sticky statements and normal statements caused by their origin from different information blocks, the latter statement in adjacent statement pairs with semantic correlation lower than the benchmark correlation value is removed, and calculation is performed only when adjacent statement pairs belong to the same information block. Adjacent statements at the boundaries of different information blocks are not included in the semantic relevance comparison, thereby completing the identification of adhesion risks during the annotation stage, improving the accuracy of information annotation and the reliability of the agent in the retrieval scenario. Attached Figure Description

[0051] Figure 1 This is a system block diagram of a data flywheel annotation quality assessment system for AI intelligent agents, as described in an embodiment of the invention.

[0052] Figure 2 This is a logic decision diagram for calibrating easily sticky sub-interfaces in an embodiment of the invention.

[0053] Figure 3 This is a logic block diagram illustrating the determination of the extraction method in an embodiment of the invention.

[0054] Figure 4 This is a logic diagram for determining whether to remove a statement according to an embodiment of the invention. Detailed Implementation

[0055] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0056] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0057] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0058] Please see Figure 1 The diagram shown is a system block diagram of a data flywheel annotation quality assessment system for AI agents according to an embodiment of the invention. The system of the present invention includes:

[0059] The information acquisition module is used to acquire the target interface retrieved by the intelligent agent, as well as the page visual data of the target interface;

[0060] The interface segmentation module is used to segment the target interface according to the visual periodicity to obtain several sub-interfaces;

[0061] The information block determination module is used to determine several information blocks based on the pixel spacing in the sub-interface;

[0062] The adhesion analysis module is used to identify easily adhesive sub-interfaces based on the characteristics of missing boundary gaps and block height deviations.

[0063] The Adhesion Removal Module is used to determine the extraction method based on the relative positional relationship of each easily adhering sub-interface, and to determine whether to remove sentences based on the semantic correlation between adjacent sentences in the extracted text.

[0064] The flywheel annotation module is used to annotate the text that has not been removed from the target interface and input the annotation results into the flywheel training set.

[0065] It is understandable that the target interface is the search results page obtained by the agent during the search process, such as the enterprise information search results page or other similar list pages. It is sufficient to provide information blocks and corresponding page visual data, which will not be elaborated further.

[0066] It is understandable that the visual data of the target interface includes, but is not limited to, the position coordinates of each pixel on the page, as well as information such as pixel density and pixel spacing determined based on pixel position statistics. It is sufficient to reflect the arrangement and boundary of information blocks on the page, which will not be elaborated further.

[0067] It is understandable that the visual data of the target interface can be obtained by performing image analysis or parsing the page rendering data. As long as the basic data required for subsequent interface segmentation and connection analysis can be provided, this will not be elaborated further.

[0068] It should be noted that in this embodiment, the pixels, pixel density, and pixel spacing mentioned in the claims and specification all refer to text pixels, text pixel density, and the spacing between text pixels. Text pixels can be determined through image binarization processing or OCR recognition results, which will not be elaborated further here.

[0069] Specifically, the interface segmentation module includes,

[0070] Interface segmentation units are used to segment the target interface starting from the first text density peak line and using visual periodicity as the segmentation interval to obtain several sub-interfaces.

[0071] Understandably, the interface segmentation unit does not directly use the top edge of the target interface as the segmentation starting point. Instead, it first determines the vertical position of the first text density peak line in the target interface, and then uses this position as the starting alignment point to determine a sub-interface every one visual cycle along the vertical direction until the bottom edge of the target interface is covered. Since the text density peak line corresponds to the title line or key field line with the densest text in each information block, starting with the first peak line reduces the probability of the segmentation boundary falling inside the information block.

[0072] In this embodiment, if there are multiple spacing values ​​with the highest frequency in the vertical spacing of adjacent text density peak lines, the minimum value among the highest spacing values ​​is determined as the visual periodicity to ensure that the segmentation spacing is small enough to avoid erroneously merging smaller information blocks.

[0073] Specifically, the interface segmentation module also includes,

[0074] The vertical spacing determination sub-unit is used to determine the vertical spacing between adjacent lines of text density peaks;

[0075] The visual periodicity determination subunit is used to determine the visual periodicity by identifying the vertical spacing with the highest frequency among the vertical spacing of adjacent text density peak lines.

[0076] It is understandable that the vertical spacing is the distance between adjacent text density peak lines in the vertical direction. It can be determined by the difference in the vertical coordinates of adjacent text density peak lines in the target interface. It is sufficient to quantify the vertical distance between adjacent text density peak lines, so this will not be elaborated further.

[0077] It is understandable that the frequency of each vertical spacing is the proportion of the number of times the same vertical spacing value appears in all adjacent vertical spacings. The system arranges each vertical spacing in descending order of frequency of occurrence and determines the first vertical spacing as the visual periodicity.

[0078] Specifically, the interface segmentation module also includes,

[0079] The horizontal strip division sub-unit is used to divide the target interface into several horizontal strips from top to bottom;

[0080] The pixel density determination subunit is used to determine the pixel density within each horizontal strip;

[0081] The peak row determination sub-unit is used to determine the horizontal strip as the text density peak row when the pixel density of the horizontal strip meets the reference density condition.

[0082] The baseline density condition is that the pixel density of the horizontal strip exceeds the baseline density value, and the difference between the pixel density of the horizontal strip and the pixel density of its adjacent horizontal strip exceeds the baseline difference value.

[0083] In this embodiment, there is no limitation on the way the target interface is divided into several horizontal strips from top to bottom. Each pixel row can be used as a unit, and each pixel row can be regarded as a horizontal strip, or several consecutive pixel rows can be merged into a horizontal strip. As long as the horizontal strips cover the entire vertical range of the target interface and the number of pixels in each horizontal strip can be counted separately, this will not be elaborated further.

[0084] It is understandable that the pixel density within a horizontal strip can be determined by statistically analyzing the ratio of the number of pixels belonging to text within the horizontal strip to the area of ​​the pixel region covered by the horizontal strip. It is sufficient to reflect the density of text pixels within each horizontal strip, which will not be elaborated further.

[0085] In this embodiment, the baseline density value is a reference density value representing the overall density level of the text pixel distribution in the target interface. The baseline density value is predetermined. As an example, those skilled in the art can determine the average value of the pixel density of each horizontal strip in the target interface as the baseline density value. The specific value does not constitute a limitation on the present invention, and will not be elaborated here.

[0086] In this embodiment, the benchmark difference is a reference value used to measure the difference in pixel density between adjacent horizontal stripes. It can reflect the normal fluctuation level of pixel density between adjacent horizontal stripes. The benchmark difference is predetermined. As an example, those skilled in the art can determine the average value of the pixel density difference between each adjacent horizontal strip within the target interface as the benchmark difference. The specific value does not constitute a limitation on the present invention, and will not be elaborated here.

[0087] It is understood that for any given transverse strip, its adjacent transverse strips include transverse strips located above the transverse strip and transverse strips located below the transverse strip.

[0088] In this embodiment, the baseline density condition is that the pixel density of the horizontal strip exceeds a baseline density value, and the difference between the pixel density of the horizontal strip and the pixel density of its adjacent horizontal strips both exceed a baseline difference. Horizontal strips that meet the baseline density condition are identified as text density peak rows. The baseline density condition, by simultaneously limiting absolute density and relative difference, ensures that only the horizontal strip with the most prominent pixel density is identified as a text density peak row, avoiding the repeated identification of consecutive high-density rows as peak rows.

[0089] Specifically, the information block determination module includes,

[0090] The information block defines the sub-unit, which is used to group pixels in the sub-interface whose pixel spacing is lower than the reference spacing value into the same information block.

[0091] In this embodiment, the pixel spacing is the coordinate difference between two adjacent text pixels within the sub-interface. All text pixels within the sub-interface can be sorted by vertical coordinates, and the absolute value of the difference between the vertical coordinates of adjacent pixels can be calculated as the pixel spacing; alternatively, they can be sorted by horizontal coordinates, and the absolute value of the difference between the horizontal coordinates of adjacent pixels can be calculated as the pixel spacing. The spacing only needs to reflect the degree of clustering or dispersion of text pixels, and will not be elaborated further.

[0092] In this embodiment, the reference spacing value is used to distinguish the pixel spacing of text within the same information block from the gap between adjacent information blocks. The reference spacing value is predetermined. As an example, those skilled in the art can determine the spacing value with the highest frequency among all adjacent pixel spacings in the target interface as the reference spacing value. The specific value does not constitute a limitation on the present invention and will not be elaborated further.

[0093] Please see Figure 2 As shown, it is a logic determination diagram for calibrating easily sticky sub-interfaces according to an embodiment of the invention. The adhesion analysis module in this embodiment includes,

[0094] The easy-adhesion calibration sub-unit is used to calibrate sub-interfaces with boundary gap missing features exceeding the reference gap missing value or block height deviation features exceeding the reference deviation value as easy-adhesion sub-interfaces.

[0095] In this embodiment, the reference gap missing value is a reference value used to distinguish between normal boundaries and adhesion boundaries, and the reference gap missing value is predetermined. As an example, those skilled in the art can determine the median of the gap missing features of each sub-interface boundary within the current target interface as the reference gap missing value, as long as the reference gap missing value can reflect the overall density level of the gap region at the normal boundary in the current target interface. The specific value does not constitute a limitation on the present invention, and will not be elaborated further here.

[0096] In this embodiment, the reference deviation value is a deviation reference value used to distinguish between normal sub-interfaces and easily sticky sub-interfaces, and the reference deviation value is predetermined. As an example, those skilled in the art can determine the median of the block height deviation characteristics of each sub-interface within the current target interface as the reference deviation value, as long as the reference deviation value can reflect the block height stability level of the normal sub-interfaces in the current target interface. The specific value does not constitute a limitation on the present invention, and will not be elaborated further here.

[0097] Specifically, the adhesion analysis module also includes,

[0098] The gap region determination sub-unit is used to determine the gap region between adjacent information blocks;

[0099] The boundary gap missing feature determination subunit is used to determine the ratio of the average pixel density of the gap region to the average pixel density within the information block as the boundary gap missing feature.

[0100] It is understandable that the gap region is the vertical region between adjacent information blocks. As an example, the gap region can be determined by identifying the lowest text pixel position of the upper information block and the highest text pixel position of the lower information block; the vertical interval between these two positions is the gap region.

[0101] It is understandable that the average pixel density of the gap area and the average pixel density of the area inside the information block can be determined by counting the number of pixels belonging to the text in the corresponding area and dividing by the pixel area of ​​the corresponding area. As long as it can reflect the average density of the text pixels in the corresponding area, it will not be elaborated further.

[0102] It is understandable that for any sub-interface, the gap region between each adjacent information block within the sub-interface is treated as a whole, and its average pixel density is calculated. At the same time, the internal region of each information block within the sub-interface is treated as a whole, and its average pixel density is calculated. The ratio of the two is the boundary gap missing feature of the sub-interface, which provides a structural basis for subsequent calibration of easily sticky sub-interfaces.

[0103] In this embodiment, the boundary gap missing feature is the ratio of the average pixel density of the gap region to the average pixel density of the internal region of the information block. When the gap region between adjacent information blocks is normal, the average pixel density of the gap region is much lower than that of the internal region of the information block, and the boundary gap missing feature is small. When the boundaries of adjacent information blocks are stuck together, the gap region is invaded by the text of the adjacent information block, or the gap region is compressed, causing the average pixel density of the gap region to increase relative to the internal region of the information block, and the boundary gap missing feature increases. Therefore, the larger the boundary gap missing feature, the higher the degree of insufficient white space or boundary overstepping between adjacent information blocks, and the greater the risk of adjacent information blocks sticking together.

[0104] Specifically, the adhesion analysis module also includes,

[0105] The information block height determination subunit is used to determine the information block height by the absolute value of the difference between the maximum and minimum vertical coordinates of each pixel within the information block.

[0106] The deviation rate determination subunit is used to determine the deviation rate of each information block as the ratio of the absolute value of the difference between the height of each information block and the visual periodicity to the visual periodicity.

[0107] The block height deviation feature determination sub-unit is used to determine the variance of the deviation rate of information blocks within the sub-interface as the block height deviation feature.

[0108] In this embodiment, the block height deviation feature is the variance of the deviation rate between the height of each information block within the sub-interface and the visual periodicity. When the boundaries of each information block are normal, the heights of the information blocks are concentrated near the visual periodicity, the deviation rate is generally small, and the block height deviation feature is small. When there is adhesion, the height of the incorrectly merged information blocks will deviate significantly from the visual periodicity, or the height of adjacent information blocks will be smaller due to content being scrolled away, leading to an increase in the difference in the deviation rate of each information block, and an increase in the block height deviation feature. Therefore, the larger the block height deviation feature, the greater the degree to which the height of the information blocks within the sub-interface deviates from the page standard, and the higher the risk of adhesion of information block boundaries; the smaller the block height deviation feature, the more consistent the height of each information block is with the visual periodicity, and the more reliable the information block boundaries.

[0109] Please see Figure 3 As shown, it is a logic block diagram for determining the extraction method according to an embodiment of the invention. The adhesion removal module in this embodiment includes,

[0110] The average distance determination sub-unit is used to determine the average distance between the center points of each easily sticky sub-interface;

[0111] The extraction method determines the sub-unit, which is used to determine aggregate extraction when the average distance does not exceed the benchmark distance value, and to determine independent extraction when the average distance exceeds the benchmark distance value;

[0112] Among them, aggregate extraction involves extracting the entire region enclosed by the outer boundaries of each easily sticky sub-interface, while independent extraction involves extracting the easily sticky sub-interfaces separately.

[0113] In this embodiment, the average distance between the center points of each easily sticky sub-interface can be obtained by determining the position coordinates of the center points of each easily sticky sub-interface and calculating the average distance between two center points. It is sufficient to reflect the overall degree of aggregation of each easily sticky sub-interface in the target interface, which will not be elaborated further.

[0114] It should be noted that when there is only one easily sticky sub-interface, there is no average distance between the center points of each easily sticky sub-interface, and independent extraction is used directly in this case.

[0115] In this embodiment, the reference distance value is a distance reference value used to distinguish between concentrated and dispersed distributions of easily sticky sub-interfaces, and the reference distance value is predetermined. As an example, those skilled in the art can determine the median distance between the center points of adjacent information blocks in the current target interface as the reference distance value, as long as the reference distance value can reflect the normal spacing level between information blocks in the current target interface. The specific value does not constitute a limitation on the present invention, and will not be elaborated further here.

[0116] It is understandable that aggregate extraction treats the concentrated distribution of easily sticky sub-interfaces as a whole and extracts the smallest continuous region covering the outer boundary of each easily sticky sub-interface, while independent extraction extracts the regions where the dispersed easily sticky sub-interfaces are located separately.

[0117] Please see Figure 4 As shown, this is a logic diagram for determining whether to remove a statement according to an embodiment of the invention. The sticky removal module in this embodiment includes:

[0118] The text extraction determination subunit is used to determine the text to be extracted based on the extraction method;

[0119] The semantic association calculation subunit is used to determine the degree of semantic association between adjacent sentences in the extracted text.

[0120] The statement removal subunit is used to remove the statement that has a semantic relevance lower than the baseline relevance value in adjacent statement pairs within the same information block.

[0121] In this embodiment, the extracted text refers to the text content extracted from the corresponding extraction region based on the extraction method. Specifically, when using aggregate extraction, the extracted text is the text within the smallest continuous region covering the outer boundary of each easily sticky sub-interface; when using independent extraction, the extracted text is the text within the region where each easily sticky sub-interface is located.

[0122] It is understandable that the extracted text can be divided into sentences based on punctuation marks in the text, or it can be divided into sentences based on the natural line breaks of each text line in the target interface. As long as the divided sentences can reflect the relatively independent sentences in the extracted text, this will not be elaborated further.

[0123] It is understandable that within the overall area corresponding to the aggregation extraction, non-adhesive sub-interfaces do not participate in statement removal; statement removal only applies to adjacent statement pairs within the same information block inside adhesive sub-interfaces.

[0124] It is understandable that the semantic relevance between adjacent statements can be determined by calculating the cosine similarity of the word vectors of adjacent statements. As long as the degree of semantic relevance between adjacent statements can be quantified, this will not be elaborated further.

[0125] In this embodiment, the baseline correlation value is a reference value for distinguishing between normal sentences and contiguous sentences. The baseline correlation value is predetermined. As an example, those skilled in the art can determine the median of the semantic correlation degree between adjacent sentences in the currently extracted text as the baseline correlation value. It is only necessary to ensure that the baseline correlation value can reflect the overall level of semantic correlation between adjacent sentences in the extracted text. The specific value does not constitute a limitation on the present invention, and will not be elaborated here.

[0126] In this embodiment, the data flywheel training set is a dataset used to store labeled text data for subsequent iterative training of the AI ​​agent. The flywheel labeling module labels the text in the target interface that has not been removed, and then inputs the labeling results into the data flywheel training set, thereby providing training data support for the AI ​​agent in subsequent retrieval and labeling processes.

[0127] In normal search result blocks, since the statements within a normal block revolve around the same information object, the semantic connection between adjacent statements is natural and the relevance is high. However, adjoining statements are content that crosses boundaries from adjacent blocks, lacking coherence within the same block as the preceding statement, resulting in a significant decrease in semantic relevance. This embodiment determines the semantic relevance between adjacent statements within the extracted text. It utilizes the semantic break caused by adjoining statements originating from different blocks to eliminate the latter statement in adjacent statement pairs with semantic relevance below the benchmark value. The calculation is only performed when adjacent statement pairs belong to the same block. Adjoining statements at the boundaries of different blocks are not included in the semantic relevance comparison, thus identifying adjoining risks during the annotation stage, improving the accuracy of information annotation, and enhancing the reliability of the agent's operation in retrieval scenarios.

[0128] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A data flywheel annotation quality assessment system for AI intelligent agents, characterized in that, include, The information acquisition module is used to acquire the target interface retrieved by the intelligent agent, as well as the page visual data of the target interface; The interface segmentation module is used to segment the target interface based on visual periodicity to obtain several sub-interfaces. The visual periodicity is determined based on the vertical spacing of the most frequent occurrence of adjacent text density peak lines, and the text density peak lines are determined based on the pixel density of the horizontal strips satisfying the baseline density condition. The information block determination module is used to determine several information blocks based on the pixel spacing in the sub-interface; The adhesion analysis module is used to identify easily adhered sub-interfaces based on the boundary gap missing feature and the block height deviation feature. The boundary gap missing feature is determined based on the ratio of the average pixel density of the gap region between adjacent information blocks to the average pixel density within the information block. The block height deviation feature is determined based on the variance of the deviation rate between the height of each information block and the visual periodicity within the sub-interface. The adhesion removal module is used to determine the extraction method based on the relative positional relationship of each easily adhesive sub-interface, and to determine whether to remove a sentence based on the semantic correlation between adjacent sentences in the extracted text; the extraction method includes aggregate extraction and independent extraction, and the extracted text is determined based on the extraction method. The flywheel annotation module is used to annotate the text that has not been removed from the target interface and input the annotation results into the flywheel training set.

2. The data flywheel annotation quality assessment system for AI agents according to claim 1, characterized in that, The interface segmentation module includes, Interface segmentation units are used to segment the target interface starting from the first text density peak line and using visual periodicity as the segmentation interval to obtain several sub-interfaces.

3. The data flywheel annotation quality assessment system for AI agents according to claim 2, characterized in that, The interface segmentation module also includes, The vertical spacing determination sub-unit is used to determine the vertical spacing between adjacent lines of text density peaks; The visual periodicity determination subunit is used to determine the visual periodicity by identifying the vertical spacing with the highest frequency among the vertical spacing of adjacent text density peak lines.

4. The data flywheel annotation quality assessment system for AI agents according to claim 3, characterized in that, The interface segmentation module also includes, The horizontal strip division sub-unit is used to divide the target interface into several horizontal strips from top to bottom; The pixel density determination subunit is used to determine the pixel density within each horizontal strip; The peak row determination sub-unit is used to determine the horizontal strip as the text density peak row when the pixel density of the horizontal strip meets the reference density condition. The baseline density condition is that the pixel density of the horizontal strip exceeds the baseline density value, and the difference between the pixel density of the horizontal strip and the pixel density of its adjacent horizontal strip exceeds the baseline difference value.

5. The data flywheel annotation quality assessment system for AI agents according to claim 1, characterized in that, The information block determination module includes, The information block defines the sub-unit, which is used to group pixels in the sub-interface whose pixel spacing is lower than the reference spacing value into the same information block.

6. The data flywheel annotation quality assessment system for AI agents according to claim 1, characterized in that, The adhesion analysis module includes, The easy-adhesion calibration sub-unit is used to calibrate sub-interfaces with boundary gap missing features exceeding the reference gap missing value or block height deviation features exceeding the reference deviation value as easy-adhesion sub-interfaces.

7. The data flywheel annotation quality assessment system for AI agents according to claim 1, characterized in that, The adhesion analysis module also includes, The gap region determination sub-unit is used to determine the gap region between adjacent information blocks; The boundary gap missing feature determination subunit is used to determine the ratio of the average pixel density of the gap region to the average pixel density within the information block as the boundary gap missing feature.

8. The data flywheel annotation quality assessment system for AI agents according to claim 1, characterized in that, The adhesion analysis module also includes, The information block height determination subunit is used to determine the information block height by the absolute value of the difference between the maximum and minimum vertical coordinates of each pixel within the information block. The deviation rate determination subunit is used to determine the deviation rate of each information block as the ratio of the absolute value of the difference between the height of each information block and the visual periodicity to the visual periodicity. The block height deviation feature determination sub-unit is used to determine the variance of the deviation rate of information blocks within the sub-interface as the block height deviation feature.

9. The data flywheel annotation quality assessment system for AI agents according to claim 1, characterized in that, The adhesion removal module includes, The average distance determination sub-unit is used to determine the average distance between the center points of each easily sticky sub-interface; The extraction method determines the sub-unit, which is used to determine aggregate extraction when the average distance does not exceed the benchmark distance value, and to determine independent extraction when the average distance exceeds the benchmark distance value; Among them, aggregate extraction involves extracting the entire region enclosed by the outer boundaries of each easily sticky sub-interface, while independent extraction involves extracting the easily sticky sub-interfaces separately.

10. The data flywheel annotation quality assessment system for AI agents according to claim 9, characterized in that, The adhesion removal module includes, The extracted text determination subunit is used to determine the extracted text based on the extraction method; The semantic association calculation subunit is used to determine the semantic association degree between adjacent sentences in the extracted text; The statement removal subunit is used to remove the statement that is less than the baseline relevance value in the semantic relevance of adjacent statement pairs within the same information block.

Images