Page element positioning method based on visual semantic fusion of large models

Abstract

The present invention provides a page element positioning method based on visual semantic fusion of a large model, which belongs to the field of data processing technology. Specifically, it includes: determining a trusted feature group in a feature group based on the similarity between the features of different feature groups and other page elements, taking other page elements with similar features in the trusted feature group as associated page elements, determining the confidence of features of different dimensions in the trusted feature group based on the similarity between features of associated page elements in different dimensions and page elements in the trusted feature group, determining the deviation of associated page elements between different trusted feature groups, and determining a positioning processing method for page elements in combination with the confidence of features of different dimensions in the trusted feature group, thereby improving the accuracy of positioning processing.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a page element positioning method based on visual semantic fusion of a large model. Background Art

[0002] To achieve positioning processing on the front-end network interface, the invention patent application CN202410077624.7, "Page Element Positioning Method, Apparatus, Device, and Medium," calculates a first similarity between a target image and a first image in a target area, and determines whether the element to be positioned is successfully positioned in the page image based on the first similarity. However, this method has the following technical problems:

[0003] The front-end page is not static. Not only are there a large number of dynamic page elements, but as the front-end page is revised, due to the changes in image elements, the use of images often cannot accurately achieve the positioning processing of page elements, making it difficult to meet the reliability and accuracy of positioning processing.

[0004] In order to solve the above technical problems, the present application provides a page element positioning method based on visual semantic fusion of a large model. Summary of the Invention

[0005] To achieve the purpose of the present invention, the present invention adopts the following technical solutions:

[0006] Specifically, this application provides a page element positioning method based on visual semantic fusion of a large model, which specifically includes:

[0007] S1 uses the large model to determine the page element associated with the user instruction, and uses the recognition result of the dynamic page element of the front-end page corresponding to the page element, combined with the deviation of the page element during the dynamic page element change process, to determine that the recognition deviation of the page element does not meet the requirements, then proceed to the next step;

[0008] S2 extracts features of different dimensions of the page elements based on the associated data of the front-end page, constructs multiple feature groups based on the features, and determines credible feature groups among the feature groups based on the similarities between the features of different feature groups and other page elements;

[0009] S3: taking other page elements with similar features to the page element in the trusted feature group as associated page elements, so as to associate features of the page element in different dimensions in the trusted feature group with the page element, and determine the confidence of the features in different dimensions in the trusted feature group;

[0010] S4 determines the deviation of the associated page elements between different credible feature groups, and determines the positioning processing method of the page elements in combination with the confidence of the features of different dimensions in the credible feature groups.

[0011] The beneficial effects of the present invention are:

[0012] The confidence of features of different dimensions in the trusted feature group is determined based on the similarity between the features of associated page elements in different dimensions and the page elements in the trusted feature group. This fully takes into account the deviation of features of associated page elements with interference risks in a certain dimension, which leads to differences in the recognition accuracy of features for associated page elements with interference risks. It also lays the foundation for evaluating the confidence of features based on the differences in the recognition accuracy of associated page elements with interference risks, thereby improving the accuracy of the confidence assessment process.

[0013] The positioning and processing method of page elements is determined based on the deviation of associated page elements between different trusted feature groups and the confidence of features of different dimensions in trusted feature groups. It not only takes into account the difference in confidence of trusted feature groups for identification processing, but also takes into account the identification interference of associated page elements when combined with other trusted feature groups, thereby improving the reliability of identification processing of page elements.

[0014] A further technical solution is that the associated page elements are determined according to the user's operation instructions using the semantic recognition results of the large model.

[0015] A further technical solution is that the deviation of the page element during the change of the dynamic page element is determined according to the deviation of the page image of the page element in different front-end pages during the change.

[0016] A further technical solution is to determine that the recognition deviation of the page element does not meet the requirements, specifically including:

[0017] Determining the number of dynamic page elements in the front-end page based on the recognition result of the dynamic page elements of the front-end page corresponding to the page;

[0018] According to the change of the dynamic page elements of the front-end page of different dynamic page elements during the change process, the dynamic page element whose image position of the page element of the front-end page changes is determined and used as the changed page element;

[0019] Based on the change update data of the changed page element, it is determined whether the recognition deviation of the page element meets the requirement.

[0020] A further technical solution is that, when there is a changed page element whose change update cycle is less than a preset update cycle threshold, it is determined that the recognition deviation of the page element does not meet the requirements.

[0021] A further technical solution is that the method for determining the positioning processing method of the page element is:

[0022] Determining the group confidence of different credible feature groups by summing the confidences of features of different credible feature groups in different dimensions;

[0023] Determine the same associated page elements between different trusted feature groups based on the deviation of the associated page elements between different trusted feature groups, and treat them as the same page elements;

[0024] A positioning processing method for the page elements is determined according to the group confidences of different credible feature groups and the same page elements between different credible feature groups.

[0025] A further technical solution is to determine a method for locating page elements based on the group confidences of different trusted feature groups and the same page elements between different trusted feature groups, specifically including:

[0026] When there is a credible feature group whose group confidence is greater than a preset group confidence threshold, the features corresponding to the credible feature group with the largest group confidence are used to perform positioning processing on the page element;

[0027] When there is no credible feature group whose group confidence is greater than the preset group confidence threshold, the credible feature groups are freely combined based on the preset group confidence threshold to obtain alternative positioning schemes, and the credible feature groups corresponding to the alternative positioning schemes are used as alternative feature groups. Based on the same page elements between the alternative feature groups, the positioning processing method of the page elements is determined.

[0028] A further technical solution is that the alternative positioning solution is constructed on the basis that the sum of the group confidences of the constituted credible feature groups is greater than a preset group confidence threshold.

[0029] A further technical solution is to determine a positioning processing method for the page elements based on the same page elements between the candidate feature groups, specifically including:

[0030] Determine the same page elements between candidate feature groups, and determine the average similarity coefficients of the same page elements in different registered feature groups based on the average of the feature similarity coefficients between the features corresponding to the same page elements in different candidate feature groups;

[0031] With the goal of minimizing the number of identical page elements whose minimum similarity coefficient average in different registered feature groups is greater than the preset value of the similarity feature coefficient, an identification scheme in the alternative positioning scheme is determined, and the positioning processing of the page elements is performed using the identification scheme.

[0032] A further technical solution is to utilize the identification scheme to perform positioning processing on the page elements, specifically including:

[0033] Based on the positioning results of the candidate feature groups in the identification scheme, the positioning result corresponding to the largest number of candidate feature groups is used as the positioning processing result of the page element.

[0034] Other features and advantages will be described in the following description. The objectives and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description and drawings.

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the accompanying drawings.

[0037] Figure 1 It is a flow chart of a page element positioning method based on visual semantic fusion of a large model;

[0038] Figure 2 It is a flow chart for determining whether the recognition deviation of the page elements does not meet the requirements;

[0039] Figure 3 is a flow chart of a method for determining a credible feature group among feature groups;

[0040] Figure 4 It is a flow chart of a method for determining a positioning processing method of a page element. DETAILED DESCRIPTION

[0041] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this specification without creative work should fall within the scope of protection of this specification.

[0042] In this application, page elements are positioned based on visual features, text features, and structural features.

[0043] For example, the user command is: "Reduce the price of all products with inventory greater than 100 by 10%";

[0044] Execution process:

[0045] User->>System: Input natural language instructions;

[0046] System->>Visual Module: Intercept the product list page;

[0047] Vision Module-->>System: Return the coordinates of the price input box (conf=0.91);

[0048] System->Semantic Module: Parse "Inventory > 100" and "Price reduced by 10%";

[0049] Semantic module -->> system: maps to DOM attributes data-stock and data-price;

[0050] System -> Fusion Engine: α=0.3, β=0.6, γ=0.1, where α is the weight of visual features, β is the weight of text features, and γ is the weight of structural features. Visual and structural features are extracted using the visual recognition model, and text features are extracted using the semantic recognition model.

[0051] Fusion Engine-->>System: Comprehensive confidence 0.87;

[0052] System->RPA Executor: Execute batch price change operations.

[0053] Dynamic page elements whose image positions of page elements of the front-end page change are regarded as changing page elements. When there are changing page elements whose change update cycle is less than 3 seconds, it is determined that the recognition deviation of the page element does not meet the requirements.

[0054] It should be noted that the large model in this application was built using deepSeek R1.

[0055] The features of the page elements of the feature group are used as group matching features, and the feature similarity coefficients between the page elements and other page elements in different group matching features are determined. The group matching features whose feature similarity coefficients are greater than the preset similarity threshold are used as similar features, and other page elements whose group matching features all belong to similar features are used as interference page elements. When the number of interfering page elements is less than 3, the feature group is determined to be a credible feature group.

[0056] The associated page elements whose features belong to similar features are used as screening associated elements to determine the proportion of the number of screening associated elements in the associated page elements of the trusted feature group, and the similarity interference coefficient of the feature in the trusted feature group is determined. The average similarity coefficient is determined based on the average value of the feature similarity coefficients of the associated page elements of the feature in the trusted feature group and the page elements. The interference value is determined based on the product of the similarity interference coefficient and the average similarity coefficient. The confidence of the feature is determined according to the difference between 1 and the interference value of different features.

[0057] The group confidence of different trusted feature groups is determined by the sum of the confidences of features in different dimensions of different trusted feature groups. According to the deviations of the associated page elements between different trusted feature groups, the same associated page elements between different trusted feature groups are determined and treated as the same page elements. The positioning processing method of the page elements is determined according to the group confidences of different trusted feature groups and the same page elements between different trusted feature groups.

[0058] like Figure 1 As shown, the present application provides a page element positioning method based on visual semantic fusion of a large model, which specifically includes:

[0059] S1 uses the large model to determine the page element associated with the user instruction, and uses the recognition result of the dynamic page element of the front-end page corresponding to the page element, combined with the deviation of the page element during the dynamic page element change process, to determine that the recognition deviation of the page element does not meet the requirements, then proceed to the next step;

[0060] Furthermore, the associated page elements are determined according to the user's operation instructions using the semantic recognition results of the large model.

[0061] Specifically, the deviation of the page element during the change of the dynamic page element is determined according to the deviation of the page image of the page element in different front-end pages during the change.

[0062] Specifically, such as Figure 2 As shown, determining that the recognition deviation of the page element does not meet the requirements specifically includes:

[0063] Determining the number of dynamic page elements in the front-end page based on the recognition result of the dynamic page elements of the front-end page corresponding to the page;

[0064] According to the change of the dynamic page elements of the front-end page of different dynamic page elements during the change process, the dynamic page element whose image position of the page element of the front-end page changes is determined and used as the changed page element;

[0065] Based on the change update data of the changed page element, it is determined whether the recognition deviation of the page element meets the requirement.

[0066] Furthermore, when there is a changed page element whose change update cycle is less than a preset update cycle threshold, it is determined that the recognition deviation of the page element does not meet the requirement.

[0067] It is understandable that when the recognition deviation of the page element meets the requirements, the positioning processing of the page element is performed by using the image of the front page.

[0068] In another possible embodiment, determining that the recognition deviation of the page element does not meet the requirement specifically includes:

[0069] Determining the number of dynamic page elements in the front-end page based on the recognition result of the dynamic page elements of the front-end page corresponding to the page;

[0070] According to the change of the dynamic page elements of the front-end pages of the different dynamic page elements during the change process, determining the number of times the image position of the page element of the front-end page changes during the change process of the different dynamic page elements, and using the number as the number of changes;

[0071] Based on the number of changes, it is determined whether the recognition deviation of the page element meets the requirements.

[0072] Furthermore, when the number of changes is greater than a preset change number threshold, it is determined that the recognition deviation of the page element does not meet the requirements.

[0073] S2 extracts features of different dimensions of the page elements based on the associated data of the front-end page, constructs multiple feature groups based on the features, and determines credible feature groups among the feature groups based on the similarities between the features of different feature groups and other page elements;

[0074] Furthermore, the associated data includes a page image of the front-end page and a front-end code.

[0075] It should be noted that the features include image visual features, semantic features, and image structural features.

[0076] Table 1 is a comparison table of the features of a certain page element and other page elements in multiple dimensions.

[0077]

[0078] It can be understood that the feature group is determined by freely combining different features.

[0079] Specifically, such as Figure 3 As shown, the method for determining the credible feature group in the feature group is:

[0080] Taking the features of the page elements of the feature group as group matching features, and determining feature similarity coefficients between the page elements and other page elements with different group matching features;

[0081] The group matching features whose feature similarity coefficient is greater than the preset similarity threshold are regarded as similar features, and the other page elements whose group matching features all belong to similar features are regarded as interference page elements;

[0082] Whether the feature group is a credible feature group is determined according to the number of the interfering page elements.

[0083] Furthermore, when the number of the interfering page elements is greater than a preset threshold number of interfering page elements, it is determined that the feature group does not belong to a credible feature group.

[0084] It can be understood that the feature similarity coefficient is determined according to the Euclidean distance function.

[0085] In another possible embodiment, a method for determining a credible feature group in the feature group is:

[0086] Taking the features of the page elements of the feature group as group matching features, and determining feature similarity coefficients between the page elements and other page elements with different group matching features;

[0087] Determining the comprehensive similarity coefficient of the other page elements by taking an average value of the feature similarity coefficients between different groups of matching features;

[0088] Whether the feature group is a credible feature group is determined based on the comprehensive similarity coefficients of the other page elements.

[0089] Furthermore, when the number of its page elements whose comprehensive similarity coefficient is within the preset similarity coefficient range does not meet the requirement, it is determined that the feature group does not belong to the credible feature group.

[0090] In another possible embodiment, a method for determining a credible feature group in the feature group is:

[0091] S21 uses the features of the page element of the feature group as group matching features, and determines feature similarity coefficients between the page element and other page elements with different group matching features;

[0092] Optionally, in the above steps, if it is necessary to further determine that no other page elements have group matching features with feature similarity coefficients greater than a preset similarity threshold, then the feature group can be directly determined to be a credible feature group.

[0093] S22: group matching features whose feature similarity coefficients are greater than a preset similarity threshold are regarded as similar features, and other page elements whose group matching features all belong to similar features are regarded as interference page elements;

[0094] It is understandable that in the above steps, it is necessary to further determine that when the number of the interfering page elements does not meet the requirement, it is determined that the feature group does not belong to the trusted feature group. Specifically, it is determined that it does not meet the requirement when the number is greater than a certain number threshold.

[0095] S23: other page elements with similar features are used as screening interference elements, and comprehensive similarity coefficients of different screening interference elements are determined based on the average values ​​of feature similarity coefficients of different screening interference elements between different group matching features;

[0096] Specifically, the above steps include the following three situations:

[0097] Case 1: If the number of the filtered interference elements does not meet the requirement, that is, is greater than the threshold, it is determined that the feature group does not belong to the credible feature group;

[0098] Case 2: If the number of the screening interference elements meets the requirement, if there is a screening interference element whose comprehensive similarity coefficient does not meet the requirement, that is, if the comprehensive similarity coefficient is greater than a certain threshold, then it is determined that the feature group does not belong to the credible feature group;

[0099] Case 3: When the number of the screening interference elements whose comprehensive similarity coefficient is within the preset similarity coefficient range is too large, that is, greater than a fixed threshold, it is determined that the feature group does not belong to the credible feature group.

[0100] When the number of screening interference elements with a comprehensive similarity coefficient within the preset similarity coefficient range meets the requirements:

[0101] S24 determines the recognition credibility coefficient of the feature group according to the comprehensive similarity coefficients of different screening interference elements and the number of similar features, and determines whether the feature group is a credible feature group based on the recognition credibility coefficient.

[0102] It is understandable that, in one possible embodiment, the recognition credibility coefficient of the feature group is determined based on the difference between a preset value and the product of an average value of comprehensive similarity coefficients of different screening interference elements and the number of similar features.

[0103] In one embodiment, when the recognition credibility coefficient of the feature group is greater than a preset recognition credibility coefficient threshold, it is determined that the feature group belongs to a credible feature group.

[0104] Specifically, the similar feature is a feature whose feature similarity coefficient with the page element is greater than a preset value of the feature similarity coefficient.

[0105] S3: taking other page elements with similar features to the page element in the trusted feature group as associated page elements, so as to associate features of the page element in different dimensions in the trusted feature group with the page element, and determine the confidence of the features in different dimensions in the trusted feature group;

[0106] Furthermore, the method for determining the confidence of the feature is:

[0107] Using the associated page elements of the feature that are similar to the feature as screening associated elements, and determining the similarity interference coefficient of the feature in the credible feature group based on the proportion of the screening associated elements in the associated page elements of the credible feature group;

[0108] Determining an average similarity coefficient based on an average of feature similarity coefficients between the associated page elements of the feature in the trusted feature group and the page element;

[0109] An interference value is determined based on a product of the similarity interference coefficient and an average similarity coefficient, and a confidence level of the feature in the trusted feature group is determined based on the interference value.

[0110] It should be noted that the confidence of the feature is based on the difference between 1 and the interference value of different features.

[0111] S4 determines the deviation of the associated page elements between different credible feature groups, and determines the positioning processing method of the page elements in combination with the confidence of the features of different dimensions in the credible feature groups.

[0112] The visual recognition model and semantic recognition model are constructed based on the CNN model and the NLP semantic model. The input of the visual recognition model is the page image of the front-end page, and the input of the semantic recognition model is the front-end code.

[0113] Specifically, such as Figure 4 As shown, the method for determining the positioning processing method of the page element is:

[0114] Determining the group confidence of different credible feature groups by summing the confidences of features of different credible feature groups in different dimensions;

[0115] Determine the same associated page elements between different trusted feature groups based on the deviation of the associated page elements between different trusted feature groups, and treat them as the same page elements;

[0116] A positioning processing method for the page elements is determined according to the group confidences of different credible feature groups and the same page elements between different credible feature groups.

[0117] Furthermore, according to the group confidences of different credible feature groups and the same page elements between different credible feature groups, a method for locating the page elements is determined, specifically including:

[0118] When there is a credible feature group whose group confidence is greater than a preset group confidence threshold, the features corresponding to the credible feature group with the largest group confidence are used to perform positioning processing on the page element;

[0119] When there is no credible feature group whose group confidence is greater than the preset group confidence threshold, the credible feature groups are freely combined based on the preset group confidence threshold to obtain alternative positioning schemes, and the credible feature groups corresponding to the alternative positioning schemes are used as alternative feature groups. Based on the same page elements between the alternative feature groups, the positioning processing method of the page elements is determined.

[0120] It can be understood that the alternative positioning solution is constructed on the basis that the sum of the group confidences of the constituted credible feature groups is greater than a preset group confidence threshold.

[0121] Furthermore, based on the same page elements between the candidate feature groups, a method for positioning the page elements is determined, specifically including:

[0122] Determine the same page elements between candidate feature groups, and determine the average similarity coefficients of the same page elements in different registered feature groups based on the average of the feature similarity coefficients between the features corresponding to the same page elements in different candidate feature groups;

[0123] With the goal of minimizing the number of identical page elements whose minimum similarity coefficient average in different registered feature groups is greater than the preset value of the similarity feature coefficient, an identification scheme in the alternative positioning scheme is determined, and the positioning processing of the page elements is performed using the identification scheme.

[0124] It should be noted that the positioning process of the page element using the identification scheme specifically includes:

[0125] Based on the positioning results of the candidate feature groups in the identification scheme, the positioning result corresponding to the largest number of candidate feature groups is used as the positioning processing result of the page element.

[0126] In another possible embodiment, the method for determining the positioning processing method of the page element is:

[0127] S41 determines the group confidence of different credible feature groups by summing the confidences of features of different credible feature groups in different dimensions;

[0128] It should be noted that, in the above steps, if it is determined that there is a credible feature group whose group confidence is greater than the preset group confidence threshold, the features corresponding to the credible feature group with the largest group confidence are used to locate the page element.

[0129] S42 determines the same associated page elements between different credible feature groups based on the deviation of the associated page elements between different credible feature groups, and treats them as the same page elements. Based on a preset group confidence threshold, the credible feature groups are freely combined to obtain alternative positioning solutions. The credible feature groups corresponding to the alternative positioning solutions are used as alternative feature groups. The average value of the feature similarity coefficients of the same page elements between different alternative feature groups is used to determine the average value of the similarity coefficients of the same page elements in different registered feature groups.

[0130] In one possible embodiment, the above steps include the following two situations:

[0131] Case 1: When the number of identical page elements between the candidate feature groups of the candidate positioning solution is too large, that is, greater than a threshold, it is determined that the candidate positioning solution does not belong to the identification solution;

[0132] Scenario 2: When the number of identical page elements between the alternative feature groups of the alternative positioning scheme is not too large, if the number of identical page elements whose minimum similarity coefficient mean in different registered feature groups is greater than the preset value of the similarity feature coefficient is too large, then the reliability of its positioning processing is difficult to meet the requirements. Specifically, it can be determined by means of a threshold. At this time, it is directly determined that the alternative positioning scheme does not belong to the identification scheme.

[0133] S43 determines the recognition deviation probability of the alternative positioning scheme based on the number of the same page elements and the minimum similarity coefficient mean in different registered feature groups, and determines the recognition scheme among the alternative positioning schemes based on the recognition deviation probability.

[0134] In one embodiment, based on the number of the same page elements and the minimum similarity coefficient means in different registered feature groups, the sum of the minimum similarity coefficient means of different same page elements in different registered feature groups is determined, and the recognition deviation probability is determined by multiplying the sum of the number by a preset proportional factor.

[0135] Furthermore, the identification scheme is an alternative positioning scheme with the smallest probability of identification deviation.

[0136] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0137] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0138] The foregoing description is merely one or more embodiments of this specification and is not intended to limit this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of one or more embodiments of this specification are intended to be within the scope of the claims of this specification.

Claims

1. A page element positioning method based on visual semantic fusion of a large model, characterized by: Specifically include: The large model is used to determine the page element associated with the user instruction. Based on the recognition result of the dynamic page element of the front-end page corresponding to the page element and the deviation of the page element during the dynamic page element change process, if it is determined that the recognition deviation of the page element does not meet the requirements, the next step is entered; Based on the associated data of the front-end page, features of different dimensions of the page elements are extracted, multiple feature groups are constructed based on the features, and credible feature groups among the feature groups are determined based on the similarity between the features of different feature groups and other page elements; Other page elements with similar features to the page element in the trusted feature group are used as associated page elements, so as to associate the features of the page element in different dimensions in the trusted feature group with the similarity of the page element, and determine the confidence of the features in different dimensions in the trusted feature group; Determine the deviation of associated page elements between different trusted feature groups, and determine the location processing method of the page elements based on the confidence of features of different dimensions in the trusted feature groups; The method for determining the positioning processing method of the page element is: Determining the group confidence of different credible feature groups by summing the confidences of features of different credible feature groups in different dimensions; Determine the same associated page elements between different trusted feature groups based on the deviation of the associated page elements between different trusted feature groups, and treat them as the same page elements; Determining a location processing method for the page elements according to the group confidences of different credible feature groups and the same page elements between different credible feature groups; Determining a method for locating page elements based on group confidences of different trusted feature groups and identical page elements between different trusted feature groups specifically includes: When there is a credible feature group whose group confidence is greater than a preset group confidence threshold, the features corresponding to the credible feature group with the largest group confidence are used to perform positioning processing on the page element; When there is no credible feature group whose group confidence is greater than the preset group confidence threshold, the credible feature groups are freely combined based on the preset group confidence threshold to obtain alternative positioning schemes, and the credible feature groups corresponding to the alternative positioning schemes are used as alternative feature groups. Based on the same page elements between the alternative feature groups, the positioning processing method of the page elements is determined.

2. The page element positioning method based on large model visual semantic fusion as claimed in claim 1 is characterized in that: The associated page elements are determined according to the user's operation instructions and the semantic recognition results of the large model.

3. The page element positioning method based on large model visual semantic fusion as claimed in claim 1 is characterized in that: The deviation of the page elements of the dynamic page elements during the change process is determined according to the deviation of the page images of the page elements in different front-end pages during the change process.

4. The page element positioning method based on large model visual semantic fusion as claimed in claim 1 is characterized in that: Determining that the recognition deviation of the page element does not meet the requirements includes: Determining the number of dynamic page elements in the front-end page based on the recognition result of the dynamic page elements of the front-end page corresponding to the page; According to the change of the dynamic page elements of the front-end page of different dynamic page elements during the change process, the dynamic page element whose image position of the page element of the front-end page changes is determined and used as the changed page element; Based on the change update data of the changed page element, it is determined whether the recognition deviation of the page element meets the requirement.

5. The page element positioning method based on large model visual semantic fusion as claimed in claim 4 is characterized in that: When the recognition deviation of the page element meets the requirement, the page element is positioned using the image of the front page.

6. The page element positioning method based on large model visual semantic fusion as claimed in claim 1, characterized in that: The associated data includes a page image of the front-end page and a front-end code.

7. The page element positioning method based on large model visual semantic fusion as claimed in claim 1 is characterized in that: The features include image visual features, semantic features, and image structural features.

8. The page element positioning method based on large model visual semantic fusion as claimed in claim 1 is characterized in that: The similar feature is a feature whose feature similarity coefficient with the page element is greater than a preset value of the feature similarity coefficient.

Images