Training method of multi-task detection model, page detection method, device and product

Abstract

This specification provides a training method for a multi-task detection model, a page detection method, an equipment, and a product. The multi-task detection model includes a shared feature extraction network, a cross-task related feature extraction network, and multiple detection layers. The page detection method includes: acquiring page information of the page to be detected; processing the page information of the page to be detected through the shared feature extraction network to generate shared features; processing the shared features through the cross-task related feature extraction network, and for each of the multiple detection tasks, extracting the task features of that detection task from the shared features and fusing related features from other detection tasks to output the enhanced features corresponding to that detection task; detecting the corresponding enhanced features through the detection layers corresponding to each detection task, and outputting the detection results of each detection task; the multiple detection tasks include at least the following two types: translation detection task, format conversion detection task, and layout detection task.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a training method for a multi-task detection model, a page detection method based on a multi-task detection model, electronic devices, and computer program products. Background Technology

[0002] In the process of global software development and localization adaptation, software needs to undergo internationalization transformation for multiple languages ​​and regions. Ensuring that the user interface (UI) is presented correctly and aesthetically pleasing under different language and regional settings is a key aspect of guaranteeing product quality and user experience.

[0003] This process typically introduces a variety of internationalization issues. Exemplary issues include, but are not limited to: translation problems, such as missing text, mistranslation, or semantic mismatch; layout problems, such as text overflow, overlapping elements, or misalignment; format conversion problems, such as currency, date, and number formats not being adjusted according to the target region's specifications; and other issues that may arise due to language switching, changes in text length, or cultural differences.

[0004] Currently, automated detection solutions for the aforementioned internationalization issues typically treat different problem types as independent tasks. A common approach is to employ multiple single-purpose detection models; for example, one model might specifically check translation accuracy, while another analyzes layout compatibility. Alternatively, multiple such models might be organized in series into a processing pipeline. These solutions treat the detection of various problems as separate, sequentially executed tasks.

[0005] Because each task operates independently and does not share information, it not only leads to redundant consumption of computing resources and efficiency bottlenecks in the detection process, but also makes it difficult to effectively diagnose and comprehensively report complex problems that frequently intertwine in real-world scenarios. Summary of the Invention

[0006] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a page detection method based on a multi-task detection model is proposed, the multi-task detection model including a shared feature extraction network, a cross-task associated feature extraction network, and multiple detection layers corresponding one-to-one with multiple detection tasks; the method includes: Obtain page information of the page to be tested; wherein, the page to be tested is obtained by converting the page presented in the original language with the conversion direction of adapting to the target language; The shared feature extraction network processes the page information of the page to be detected to generate shared features. The shared features are processed by the cross-task association feature extraction network. For each of the multiple detection tasks, the task features of the detection task itself are extracted from the shared features and the association features from other detection tasks are fused together to output the enhanced features corresponding to the detection task. The corresponding enhanced features are detected through the detection layer corresponding to each detection task, and the detection results of each detection task are output; wherein, the plurality of detection tasks include at least the following two types: translation detection task, format conversion detection task, and layout detection task.

[0007] According to a second aspect of one or more embodiments of this specification, a method for training a multi-task detection model is provided, wherein the multi-task detection model is applied to the method described in the first aspect, the method comprising: Acquire training samples, which include page images, page structured data, raw language information and target language information, and are labeled with the ground truth values ​​of the problems corresponding to multiple detection tasks. The training samples are input into the multi-task detection model to be trained, and the prediction results of each detection task output by the model are obtained. The difference between the prediction result of each detection task and the corresponding true value of the problem is calculated separately to obtain the task loss for each detection task. For training samples in the same image region labeled with at least two types of questions, if the set of question types predicted by the multi-task detection model for the image region is inconsistent with the set of labeled question types, an association penalty loss is generated. Calculate the total loss of model training based on the task loss and the correlation penalty loss described above; The parameters of the multi-task detection model are optimized with the goal of minimizing the total loss.

[0008] According to a third aspect of one or more embodiments of this specification, a method for training a multi-task detection model is proposed, wherein the multi-task detection model is applied to the method described in the first aspect, the method comprising: Acquire training samples, which include page images, page structured data, raw language information and target language information, and are labeled with the ground truth values ​​of the problems corresponding to multiple detection tasks. The training samples are input into a multi-task detection model that serves as the agent, and the prediction results of each detection task output by the model are obtained. Based on the difference between the prediction results of each detection task and the true value of the corresponding problem, calculate the precision bonus, false alarm penalty and false negative penalty for the current sample; For training samples in the same image region labeled with at least two types of questions, an association reward or association penalty is generated based on the comparison between the set of question types predicted by the multi-task detection model for the image region and the set of labeled question types. The total reward is calculated based on the accuracy reward, the false alarm penalty, the missed detection penalty, and the correlation reward or the correlation penalty. The parameters of the multi-task detection model are updated with the goal of maximizing the total reward.

[0009] According to a fourth aspect of the embodiments of this specification, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect, the second aspect, or the third aspect.

[0010] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the first, second, or third aspect.

[0011] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the first, second, or third aspect.

[0012] As can be seen from the above embodiments, this specification achieves synchronous and collaborative detection of at least three types of problems: translation, format, and layout by extracting and fusing the features of each detection task and its related tasks in parallel. Compared with the traditional execution of multiple independent models in series or parallel, this significantly reduces the overall computational overhead and memory usage, and improves detection efficiency.

[0013] The cross-task association feature extraction network explicitly performs cross-task information fusion at the feature level, enabling each detection task to utilize features from other detection tasks, thereby significantly enhancing the ability to detect association problems. For example, it can more accurately detect layout overflow issues caused by translation text expansion, or identify formatting errors that have not been updated with language switching.

[0014] The enhanced features of each detection task output by the cross-task associated feature extraction network contain richer feature information, enabling subsequent dedicated detection layers to make judgments based on the enhanced features. This improves the detection rate and accuracy of complex and multifaceted international problems while maintaining the detection accuracy of each detection task.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0016] Figure 1 This is a flowchart of a training method for a multi-task detection model provided in an exemplary embodiment.

[0017] Figure 2 This is a structural diagram of a multi-task detection model provided in an exemplary embodiment.

[0018] Figure 3 This is another structural diagram of a multi-task detection model provided in an exemplary embodiment.

[0019] Figure 4 This is a flowchart of another training method for a multi-task detection model provided in an exemplary embodiment.

[0020] Figure 5 This is a flowchart of a page detection method based on a multi-task detection model, provided in an exemplary embodiment.

[0021] Figure 6 This is a schematic diagram of the structure of a device provided in an exemplary embodiment. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0023] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with relevant laws, regulations and standards, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0024] In the process of global software development and localization adaptation, software needs to undergo internationalization transformation for multiple languages ​​and regions. Ensuring that the user interface (UI) is presented correctly and aesthetically pleasing under different language and regional settings is a key aspect of guaranteeing product quality and user experience.

[0025] This process typically introduces a variety of internationalization issues. Exemplary issues include, but are not limited to: translation problems, such as missing text, mistranslation, or semantic mismatch; layout problems, such as text overflow, overlapping elements, or misalignment; format conversion problems, such as currency, date, and number formats not being adjusted according to the target region's specifications; and other issues that may arise due to language switching, changes in text length, or cultural differences.

[0026] In related technologies, automated detection solutions typically break down different problem types into independent detection tasks to address various issues arising during software internationalization (such as translation, layout, and formatting). This is mainly based on the following reasons: 1. Technological Development Path Dependence and Task Characteristics Differences: The technological development in Computer Vision (CV) and Natural Language Processing (NLP) has long followed a specialized path. Translation problems are essentially text semantic understanding and comparison tasks, relying on NLP techniques; layout problems (such as overflow and overlap) are visual spatial structure analysis tasks, relying on object detection or image segmentation techniques in CV; formatting problems are closer to rule-based text pattern matching. Different tasks have inherent differences in data types, feature spaces, and solution paradigms, making it difficult for early technologies to effectively handle them with a unified model architecture.

[0027] 2. Inertia in Project Implementation and Integration: Employing independent, single-purpose models aligns with the traditional "modular" approach in software engineering. Each model can be developed, tested, and deployed independently, facilitating reuse and integration into existing automated testing pipelines. Translation detection, UI layout analysis, and format validation are typically handled by teams with diverse technical backgrounds (NLP engineers, CV engineers, QA engineers). Using independent models facilitates technology stack selection and clear division of responsibilities among teams.

[0028] 3. Limitations in understanding and modeling the interrelationships between problems: Early solutions primarily treated internationalization issues as discrete, static defects for detection, lacking a systematic understanding of the dynamic relationships and causal connections between problems. Therefore, cross-task information interaction and collaborative reasoning were not considered core requirements in the technical design. Furthermore, deep learning techniques capable of effectively modeling such complex cross-modal and cross-task relationships (such as complex attention mechanisms and multi-task joint optimization frameworks) are themselves still under development.

[0029] In summary, the use of multiple independent models in related technologies is a practical choice constrained by factors such as the inherent differences in tasks, the complexity of model design, inertia in engineering practices, and insufficient early understanding of the interrelationships between problems. While this approach is easy to implement, it leads to computational redundancy, inefficiency, and difficulty in effectively diagnosing and comprehensively reporting complex problems that frequently intertwine in real-world scenarios.

[0030] Based on this, the embodiments of this specification provide a training method for a multi-task detection model and a page detection method based on the multi-task detection model.

[0031] Please see Figure 1 This paper illustrates a training method for a multi-task detection model, which can be executed by an electronic device, including but not limited to servers, workstations, personal computers, or distributed computing clusters. The training method for the multi-task detection model includes: In S100, training samples are obtained, which include page images, page structured data, raw language information and target language information, and are labeled with the ground truth values ​​of the problems corresponding to multiple detection tasks.

[0032] This step obtains the training dataset for model training. Each training sample in the training dataset is a multivariate dataset, whose core components include: ① Page Image: Screenshot or rendered image of the user interface to be detected, presented in the target language, providing the model with pixel-level information such as layout, visual style and text rendering results.

[0033] ② Page structured data: This is machine-readable data that corresponds to page images and describes the hierarchical structure and attributes of interface elements. It usually exists in serialized form of XML, JSON, or a specific UI description language. This data provides semantic and structural information such as the type, text content, spatial location (bounding box), parent-child relationship, and accessibility identifiers of interface elements.

[0034] ③ Original language information, including but not limited to at least one of the following: original language identifier, original language text content corresponding to the page to which the training sample belongs, engineering implementation code adapted to the original language, inherent attribute information of the original language, etc.

[0035] The original language identifier refers to the original language itself, such as zh-CN for Simplified Chinese and en-US for American English. The original language text content covers both statically displayed text and dynamically rendered text (API responses, configuration file reads). The engineering implementation code adapted to the original language includes the original language configuration section in the internationalization configuration file, the character encoding setting code for the original language text, and the page rendering logic code for the original language text. The inherent attribute information of the original language includes character length characteristics and character attributes without accents / special symbols.

[0036] ④ Target language information, including but not limited to at least one: target language identifier, target language text content corresponding to the page to which the training sample belongs, localization code adapted to the target language, inherent attribute information of the target language, translation mapping relationship between the target language and the original language, etc.

[0037] The target language identifier refers to the target language itself, such as de-DE for German and ar-SA for Arabic. The target language text content includes the translated text of the original language, supplementary text after localization adaptation, and numerical / currency / time text after format conversion. The localization code adapted to the target language includes configuration code for the target language translated text, conversion code for localized formats (currency / time / units of measurement), page layout adaptation code for RTL (Right-to-Left) languages, and rendering support code for special characters in less commonly spoken languages. The inherent attribute information of the target language includes character length characteristics, accent / special symbol attributes, and vocabulary spelling rules. The translation mapping relationship includes a one-to-one correspondence between the original language and target language texts, and translation selection rules for ambiguous texts.

[0038] Each training sample is associated with ground truth values ​​for multiple preset detection tasks, pre-labeled by humans or automated tools. For each detection task, the ground truth labels include: (a) Categorical ground truth: Indicates whether the sample has a specific type of problem in the task. For example, in the translation detection task, categorical ground truth includes: {0: missing translation, 1: incorrect translation, 2: no language translation problem}; in the format conversion detection task, categorical ground truth includes: {0: currency format error, 1: time format error, 2: number format error; 3: no format conversion problem}; in the layout detection task, categorical ground truth includes: {0: text overflow, 1: text overlap, 2: no control layout problem}.

[0039] (b) Locating the truth value: If a problem exists, provide the precise location of the problem in the page image, usually represented by a bounding box in normalized coordinates, such as [x,y,w,h], where each value ranges from 0 to 1. (x,y) indicates the center coordinates of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box.

[0040] The page corresponding to each training sample, presented in the target language, is usually obtained by converting the original language page through a localization process (including automatic or manual translation, layout adjustment, and format replacement).

[0041] For example, training samples may further include device type identifiers, such as mobile phone (portrait mode), tablet (landscape mode), and desktop. Introducing this information helps the model learn the different visual representations that similar problems (such as text overflow) may present under different screen sizes, resolutions, and interaction modes, thereby enhancing the model's generalization ability.

[0042] For example, before inputting the model, page images can be standardized, such as adjusting to a uniform size and normalizing pixel values. Structured data needs to be encoded, such as serializing XML / JSON into vectors that the model can process and converting language identifiers into embedded vectors.

[0043] In S102, the training samples are input into the multi-task detection model to be trained, and the prediction results of each detection task output by the model are obtained.

[0044] This step involves inputting the training samples into the multi-task detection model to be optimized, performing forward propagation calculations, and obtaining the model's prediction results for each detection task.

[0045] The multi-task detection model is an end-to-end deep learning network that achieves collaborative learning and inference for multiple related detection tasks by sharing low-level feature extraction and introducing inter-task information interaction at high levels. Please refer to [link / reference]. Figure 2 The multi-task detection model includes a shared feature extraction network, a cross-task associated feature extraction network, and multiple detection layers that correspond one-to-one with each of the multiple detection tasks. These multiple detection tasks include at least two of the following: translation detection task, format conversion detection task, and layout detection task.

[0046] The shared feature extraction network is used to process the page information of the page to be detected and generate shared features.

[0047] Cross-task association feature extraction network is used to process shared features. For each of multiple detection tasks, it extracts the task features of the detection task itself from the shared features and fuses the association features from other detection tasks to output the enhanced features corresponding to the detection task.

[0048] The detection layer corresponding to each detection task is used to detect the corresponding enhanced features and output the detection results of each detection task.

[0049] (1) For information on shared feature extraction networks, please refer to [link / reference]. Figure 3 It includes: a visual encoder, a text encoder, and a cross-modal fusion layer.

[0050] The visual encoder employs a pre-trained visual Transformer model, such as the CLIP-ViT model based on the Vision Transformer architecture. CLIP (Contrastive Language-Image Pretraining) is a method for pre-training images and text through contrastive learning. ViT (Vision Transformer) is a computer vision model based on the Transformer architecture. The visual encoder receives a page image as input and extracts deep visual features of the image through multi-layer self-attention mechanisms and convolutional operations. The output is a high-dimensional feature vector or feature map, which encodes information such as objects, layout, and visual attributes of text regions in the image, collectively referred to as image features.

[0051] The text encoder employs a pre-trained language model, such as the LLaMA model based on the Transformer architecture or the BERT (Bidirectional Encoder Representations from Transformers) model. The text encoder receives concatenated or spliced ​​structured page data text, original language information, and target language information. Through a multi-layer Transformer encoder, it extracts deep semantic and contextual features of the text, outputting a high-dimensional feature vector. This output encodes information such as the interface text content, element attributes, and language context, collectively referred to as text features.

[0052] The cross-modal fusion layer effectively integrates features from different modalities (images and text). Taking a cross-attention mechanism as an example, the processing of this layer can include: using text feature vectors as query vectors and image feature vectors as both key and value vectors. Through attention calculation, the similarity between the query vector (text features) and the key vector (image features) is first calculated and normalized to obtain the attention weight distribution; then, the value vector (image features) is weighted and summed according to this weight distribution to generate a visual context vector modulated by text features. Similarly, reverse interactive calculation can be performed, using image features as query vectors and text features as key and value vectors. Through cross-attention calculation, this layer can output a unified shared feature representation that integrates visual appearance information and text semantic information, thus providing a complementary joint feature foundation for subsequent multi-task detection.

[0053] (2) For information on cross-task related feature extraction networks, please refer to [link / reference]. Figure 3 It includes multiple feature extraction structures that correspond one-to-one with multiple detection tasks.

[0054] The feature extraction structure for each detection task includes: The feature projection layer is used to project shared features to generate the query vector, key vector, and value vector corresponding to the detection task. This module contains three independent learnable linear transformation matrices: the query transformation matrix ( ), key transformation matrix ( ) and value transformation matrix ( This module will input shared features (denoted as...). The query vector (Q), key vector (K), and value vector (V) are multiplied by these three matrices respectively and linearly projected to generate the query vector (Q), key vector (K), and value vector (V) specific to the detection task.

[0055] Mathematically represented as: , , , where i represents the i-th detection task. In the subsequent cross-task attention calculation, the query vector ( ) is used to calculate similarity with the key vectors of all tasks to generate attention weights, key vectors ( The value vector serves as a reference for the query comparison. This serves as the foundational information for generating fused features through weighted summation based on attention weights.

[0056] The cross-task attention layer performs attention calculations between the query vector of the current detection task and the key vectors of all detection tasks to obtain attention weights representing cross-task associations. These attention weights are calculated by normalizing the similarity between the query vector of the current detection task and the key vectors of all detection tasks. Next, based on the attention weights, the value vectors of all detection tasks are weighted and fused to generate and output the enhanced features for the corresponding detection task.

[0057] For the i-th detection task: a. Attention weight calculation: Calculate the query vector of the detection task. The key vector of all detection tasks (including detection task i itself). Similarity is calculated, where N is the number of detection tasks. Dot product attention can be used, followed by scaling and normalization. Specifically, first calculate... With each The dot product is then divided by a scaling factor, such as the square root of the key vector dimension, and finally normalized using the Softmax function to obtain a set of attention weights. These weights precisely quantify how much relevant information the i-th detection task needs to obtain from other tasks (and itself) when making decisions.

[0058] b. Weighted Fusion and Output: Using the attention weights calculated in the previous step, calculate the value vector for all tasks […]. Perform a weighted summation to generate the enhanced features for the i-th detection task. Through this mechanism, the final features of each detection task are... It not only includes task-related information extracted from shared features (through...) Furthermore, it dynamically and selectively incorporates relevant information from other tasks. For example, in the enhanced features of the layout detection task, key information about text length from the translation task features may be incorporated due to the high attention weight.

[0059] Traditional multi-task learning often only performs joint optimization at the loss function level, while each task remains relatively independent during the feature extraction stage. The cross-task associated feature extraction network proposed in the embodiments of this specification achieves deep fusion of information between tasks by introducing a dynamic cross-task attention mechanism into the feature space.

[0060] For each detection task (such as layout detection), the network not only extracts its own task features, but also queries the feature information of other tasks (such as translation detection and format detection) through an attention mechanism.

[0061] Taking layout detection as an example, the generation process of its enhanced features demonstrates significant advantages in relational modeling: Causal correlation capture: The query vector of the layout detection task is similar to the key vector of the translation detection task. If the length of the translated text in a certain region increases significantly (characterized by the features of the translation task), the layout detection task will receive higher attention weight, thus incorporating the semantic information of text expansion into the feature representation of that region. This allows the model to predict and more accurately detect text overflow or control overlap caused by the increased length of the translation, rather than relying solely on visual pixel truncation.

[0062] Coupling and correlation utilization: Features related to currency symbol width or date string length in the format detection task are dynamically fused into the layout detection task. For example, when a long date format is detected, the model automatically adjusts the tolerance threshold for the layout compactness of that area, reducing false alarms caused by normal format changes.

[0063] Inverse interference suppression: If the layout task detects severe text overlap or occlusion, this visual feature will be fed back to the translation detection task to reduce the weight of the text recognition confidence in that area, thus preventing the model from making incorrect omissions or mistranslations of the occluded incomplete text.

[0064] This explicit cross-task feature interaction mechanism enables the model to move beyond a simple stack of three independent detectors and understand the complex coupling relationships in internationalization issues, thereby significantly improving the recall rate and root cause localization accuracy for composite defects (such as layout disorder caused by mistranslation).

[0065] (3) Regarding the detection layer, please refer to Figure 3 Each detection task's corresponding detection layer includes a classifier and a locator.

[0066] The classifier consists of one or more fully connected layers, with the softmax activation function applied at the ends. The classifier is based on augmented features of the input. The model outputs a probability distribution vector as a classification identifier. For example, in a translation detection task, this vector might contain probability values ​​in three dimensions, corresponding to "missed translation," "incorrect translation," and "no problem." The dimension with the highest probability value represents the problem type (or no problem) predicted by the model.

[0067] The localizer consists of one or more fully connected layers, ending with a sigmoid activation function (ensuring the output is between 0 and 1). The localizer is based on the augmented features of the input. It outputs a four-dimensional vector as the predicted location information. This four-dimensional vector typically represents the normalized center point coordinates (x, y), width (w), and height (h) of the bounding box of the predicted problem region.

[0068] Therefore, the model's prediction result for the i-th detection task is composed of the classification label (class probability) and location information (bounding box coordinates) output by the detection layer for that task. The prediction results for all tasks are obtained together in this step, providing a basis for subsequent loss calculation and model optimization.

[0069] In S104, the difference between the prediction result of each detection task and the corresponding true value of the problem is calculated to obtain the task loss corresponding to each detection task.

[0070] This step, through loss calculation, quantifies the deviation between the model's predictions for each detection task and the actual labels, providing a clear gradient direction for model optimization. Task loss calculation is typically performed independently for each detection task, and the loss function for each task consists of a weighted average of classification loss and regression loss, simultaneously supervising the determination of the problem type and the location of the problem.

[0071] Classification loss: Used to measure the difference between the model's predicted question category labels (probability distribution) and the true category labels. For example, the cross-entropy loss function can be used. For each detection task, the model outputs a probability vector processed by the softmax function, representing the confidence level of belonging to each question category (e.g., missed, incorrect, no question). The classification loss is obtained by calculating the cross-entropy between this predicted probability distribution and the true category labels. This loss function drives the model's output category probabilities to approach the true labels.

[0072] Regression loss measures the spatial deviation between the model's predicted problem location bounding box and the ground truth bounding box. Examples include the SmoothL1 loss function or the IoU (Intersection over Union) series of loss functions. The model outputs a four-dimensional vector representing the normalized coordinates of the predicted bounding box. Regression loss is obtained by calculating the differences between the predicted and ground truth boxes in terms of center point coordinates, width, and height. For example, the SmoothL1 loss applies a linear penalty to errors with small differences and a saturated L2 penalty to errors with large differences, making it more robust to outliers.

[0073] The final task loss for each detection task (such as translation problem detection, format conversion problem detection, layout problem detection) ), usually its classification loss ( ) and regression loss ( The weighted sum of ), that is: ,in and These are preset weighting coefficients used to balance the importance of the two losses.

[0074] In S106, for training samples in the same image region labeled with at least two types of questions, if the set of question types predicted by the multi-task detection model for the image region is inconsistent with the set of labeled question types, an association penalty loss is generated.

[0075] This step introduces supervision for cross-task correlation modeling at the loss function level, encouraging the model to learn and correctly predict co-occurrence and correlation patterns between problems, rather than treating each task as completely independent.

[0076] This loss term is generated for samples in the training data that are labeled with two or more different types of problems within the same image region (e.g., the same UI control or closely adjacent regions). These labels reflect the complex defects commonly found in real-world problems.

[0077] For such samples, the electronic device compares all problem types predicted by the model in that image region (i.e., the set of categories that each detection task classifies as having a problem in that region) with the labeled set of true problem types. If the two sets are inconsistent, including the prediction missing a certain truly existing problem type or the false positive reporting of a non-existent problem type, an association penalty loss is generated. ).

[0078] In one possible implementation, this correlation penalty loss can be quantified as a function of the degree of difference between the two sets. For example, the size of the symmetric difference between the true problem type set and the predicted problem type set can be calculated, i.e., the sum of the number of incorrectly predicted problem types and the number of false positive problem types, and this value, or the result after scaling / transformation, can be used as the correlation penalty loss. For example, the multi-task detection problem in this image region can be regarded as a multi-label classification problem. A loss function suitable for multi-label scenarios (such as binary cross-entropy) can be used to calculate the overall deviation between the prediction and the ground truth across all task categories, and this deviation can be used as an association penalty.

[0079] The design of this loss term requires the model to not only minimize the error of each independent task during training, but also to further optimize its recognition accuracy for the pattern of "multiple problems appearing together in the same area".

[0080] In S108, the total loss of model training is calculated based on the loss of each task and the correlation penalty loss.

[0081] Among them, the total loss of model training ( ) is the weighted sum of the task loss and correlation penalty loss for each detection task. The expression is: ; where i iterates through all detection tasks, The task loss for the i-th task. These are the weighting coefficients assigned to the loss of this task; For related penalty losses, Its corresponding weighting coefficient.

[0082] Weighting coefficient and These are hyperparameters used to coordinate the relative importance of different loss terms in the total loss. Their settings can be determined based on prior knowledge, task importance, or through performance tuning on the validation set. For example, if you want the model to focus more on learning the relationships between tasks, you can appropriately increase the hyperparameters. If a certain detection task has a higher priority in the current scenario, its priority can be increased accordingly. .

[0083] By calculating the total loss, the training process obtains a scalar index that comprehensively reflects the model's performance in both single-task accuracy and cross-task correlation modeling ability.

[0084] In S110, the parameters of the multi-task detection model are optimized with the goal of minimizing the total loss.

[0085] This step uses the calculated total loss to iteratively update all learnable parameters in the model using the gradient descent algorithm and its variants, optimizing the model in the direction of reducing loss.

[0086] First, based on total loss The gradient of the loss with respect to the parameters of each layer of the model (including the visual encoder, text encoder, transformation matrices, and weights of fully connected layers) is calculated using the backpropagation algorithm. The gradient indicates the direction and magnitude of the effect of small changes in each parameter on the total loss.

[0087] Then, optimization algorithms are used to update the parameters based on the calculated gradients. Commonly used optimization algorithms include stochastic gradient descent (SGD), Adam, and AdamW. Taking the Adam optimizer as an example, its update rule not only considers the current gradient but also incorporates the exponential moving average of historical gradients, and performs bias correction and adaptive learning rate adjustment, thereby achieving more stable and efficient convergence in complex optimization spaces. The update process can be summarized as follows: , where θ represents the model parameters, η is the learning rate, and UpdateRule is the update rule defined for a specific optimizer.

[0088] Steps S102 to S110 described above are performed on a batch of training data, constituting one training iteration. The training process will traverse the training dataset multiple times until the total loss converges to a low level, or the model's performance on the independent validation set no longer improves significantly. At this point, the model training is considered complete.

[0089] For example, in the training process of a multi-task detection model, the setting of key hyperparameters is crucial to the model's convergence speed, stability, and final performance. For instance: Training typically employs the AdamW optimizer, with a base learning rate recommended to be set in the range of 1e-5 to 5e-5 to balance convergence and prevent catastrophic forgetting when fine-tuning the pre-trained backbone network.

[0090] The learning rate scheduling strategy uses cosine annealing in conjunction with the warm-up phase, where the number of warm-up steps accounts for about 5% to 10% of the total training steps, so that the learning rate rises slowly to the base value and then decreases according to the cosine curve.

[0091] The batch size can be set to 8, 16, or 32 depending on the computing hardware memory. When memory is limited, gradient accumulation can be used to simulate a larger effective batch.

[0092] The total number of training rounds is typically between 10 and 50, depending on the size of the dataset, and overfitting should be avoided by monitoring the validation set loss.

[0093] Model regularization is achieved through weight decay (recommended values ​​are 0.01 to 0.05) and gradient clipping (the threshold is usually set to 1.0).

[0094] The weight coefficients of each loss function term (such as classification loss weight, regression loss weight, and association penalty loss weight) need to be allocated and optimized according to the importance of the task.

[0095] All parameters need to be calibrated on the validation set to obtain better model performance.

[0096] Through this optimization process, the model parameters are systematically adjusted, and the resulting multi-task detection model can collaboratively process multiple detection tasks and has the ability to identify related internationalization issues.

[0097] In some embodiments, please refer to Figure 4 This specification also provides another method for training a multi-task detection model, which can be executed by an electronic device. This method treats the multi-task detection model as an agent, models the page detection process as an interaction between the agent and the environment (training data), and guides the model to learn a better detection strategy through reward signals. The training method for the multi-task detection model includes: In S400, training samples are acquired, which include page images, page structured data, raw language information and target language information, and are labeled with ground truth values ​​for each of the multiple detection tasks.

[0098] This step has the same purpose and content as step S100 in the aforementioned supervised learning embodiment, aiming to prepare training data for interaction. Each training sample contains a page image, page structured data, raw language information and target language information, and is labeled with the ground truth values ​​for each detection task (such as translation detection task, format conversion detection task, layout detection task), including classification and localization information.

[0099] In S402, the training samples are input as the agent's multi-task detection model, and the prediction results of each detection task are obtained from the model output.

[0100] The architecture of the multi-task detection model (including the shared feature extraction network, the cross-task associated feature extraction network, and the detection layers for each task) is the same as described in the previous embodiments, and will not be repeated here. After receiving the input, the multi-task detection model outputs the prediction results of each detection task through forward propagation, that is, the predicted problem classification identifier and location information for each task.

[0101] In S404, the precision bonus, false alarm penalty, and false negative penalty for the current sample are calculated based on the difference between the prediction results of each detection task and the true value of the corresponding problem.

[0102] This step provides fine-grained, real-time feedback for each action (i.e., prediction) of the agent (model). This feedback is calculated based on a point-by-point comparison between the prediction and the true value of the problem. Precision reward (R_precision): This reward is designed to encourage the model to make correct positive class predictions and is calculated based on the problem instances correctly identified by the model. For example, in a detection task, if the model correctly predicts the presence of a specific type of problem in a certain region and the localization is basically accurate, such as if the Intersection over Union (IoU) between the predicted bounding box and the ground truth box exceeds a threshold, a positive reward value is given. This reward is directly related to the detection precision.

[0103] False positive penalty (P_false_positive): This is designed to suppress the model from making incorrect positive category predictions. When the model predicts that there is a problem in a certain area, but the area is labeled as "no problem" or the problem type does not match in the ground truth, a negative reward, i.e., a penalty, is generated. This penalty helps to reduce the false positive rate.

[0104] False-negative penalty (P_false_negative): This penalty aims to prevent the model from missing real-world problem instances. When a region is marked as having a problem in the ground truth, but the model fails to detect it, predicting "no problem" or predicting a completely incorrect problem type, a negative reward is generated. This penalty is designed to improve recall.

[0105] These three calculations can be performed independently for each detection task, and then summed or weighted averaged to serve as the net reward / penalty for the current training samples in terms of basic detection performance.

[0106] In one possible implementation, for each detection task i, the model's basic detection performance on that sample is fed back. R_detection_i This can be constructed by quantifying the consistency between its predictions and the true value of the problem. Specifically, it can be done by statistically analyzing the correctly matched instances under this task ( TP_i False alarm examples () FP_i ) and examples of missed detections ( FN_i The contribution of each instance is assigned a positive coefficient to the correctly matched instance. w_tp ), assign a negative coefficient to false alarm instances ( w_fp ), assign a negative coefficient to the missed detection instance ( w_fn The net feedback is calculated by weighted summation. The formula is as follows: .in: TP_i (Correct Instance Contribution): Measures the degree to which the model correctly detects a problem. For example, when the Intersection over Union (IoU) between the model's predicted problem bounding box and the ground truth bounding box exceeds a preset threshold (e.g., 0.5), and the classification is correct, it can be counted as one instance contribution. TP_i contribute. w_tp A positive value means that a correct detection will generate a positive reward.

[0107] FP_i (False Alarm Contribution): Measures the degree of false alarms raised by the model. For example, if the model predicts a problem, but there is no corresponding problem at that location in the ground truth (or the IoU is too low), it is counted as one false alarm. FP_i contribute. w_fp A negative value means that a false alarm will result in a negative penalty.

[0108] FN_i (Missed Instance Contribution): Measures the degree to which the model misses problems. For example, if the model fails to detect a problem in the ground truth annotation or the bounding box IoU does not reach a threshold, it is counted as one missed instance contribution. FN_i contribute. w_fn A negative value means that a missed detection will result in a negative penalty.

[0109] In S406, for training samples in the same image region labeled with at least two types of questions, an association reward or association penalty is generated based on the comparison between the set of question types predicted by the multi-task detection model for the image region and the set of labeled question types.

[0110] This step introduces specialized supervision of cross-task correlation prediction capabilities, which is a core mechanism for promoting multi-task collaboration within the reinforcement learning framework.

[0111] This step is performed when the same image region in the training samples is labeled with two or more different types of questions. The electronic device compares the set of true question types labeled for that image region with the set of all question types predicted by the model for that image region.

[0112] If the two sets match perfectly, meaning the model correctly predicted all problem types in the region and did not falsely report other types, then a positive correlation reward is generated. This encourages models to accurately identify complex problems.

[0113] If the two sets partially match or do not match (e.g., the model misses a real problem type or falsely reports a non-existent problem class), a negative correlation penalty is generated. The magnitude of the penalty can be proportional to the degree of set difference (such as the size of a symmetric difference set), thereby precisely quantifying the prediction error.

[0114] In S408, the total reward is calculated based on the accuracy reward, false alarm penalty, missed detection penalty, and correlation reward or correlation penalty.

[0115] This step aggregates the above feedback signals into a scalar total reward (R_total), which serves as the final metric for evaluating the overall quality of the agent's actions in the current state. The total reward is typically a weighted sum of the individual components: .

[0116] in, This represents the basic net reward for detection calculated by S404 (i.e., the sum of the accuracy reward, false alarm penalty, and false negative penalty). Representing the associated reward or penalty generated by S406, β is a hyperparameter used to adjust the associated supervision weights. The larger the value of the total reward R_total, the closer the model's current predictive behavior is to the desired optimal policy.

[0117] In S410, the parameters of the multi-task detection model are updated with the goal of maximizing the total reward.

[0118] Based on the calculated total reward, this step uses a reinforcement learning algorithm to update the model parameters of the agent, with the goal of learning a policy that maximizes the long-term cumulative reward (approximately maximizing the single-step R_total expectation in this task).

[0119] Algorithms based on policy gradients, such as Proximal Policy Optimization (PPO), can be used. This involves comparing the probabilities of the current policy and the old policy in the same state, and incorporating the advantage gained from that action (calculated by R_total and the baseline value), to construct an alternative loss function. Maximizing this function through gradient ascent then pushes the model parameters towards actions that yield higher rewards.

[0120] The difference from supervised learning: Unlike the gradient descent minimization loss in the previous embodiment, this step maximizes the reward through gradient ascent. Parameter updates depend not only on the absolute error between the prediction and the true value, but also on the relative utility evaluation (reward) obtained by taking a specific prediction action under specific environmental conditions. This allows the model to learn complex, multi-objective collaborative decision-making strategies more flexibly.

[0121] By iteratively executing S400 to S410, the model gradually optimizes its parameters under the guidance of the reward signal, and finally converges into an intelligent detection model that can effectively coordinate multiple tasks and accurately identify related problems.

[0122] It's understandable that multi-task detection models offer flexibility in their training methods. They can employ various approaches. Figure 1 The supervised fine-tuning method described in the embodiments, or the method employed... Figure 4 The examples describe training using a reinforcement learning-based method. Alternatively, a phased training strategy can be employed, such as first obtaining the model's basic capabilities through supervised fine-tuning, and then further optimizing its decision-making strategy using reinforcement learning methods. This specification does not limit the specific training method used in the examples.

[0123] In some embodiments, issues such as language translation, layout, and format conversion involved in software internationalization are not isolated but rather interconnected, exhibiting causal, coupled, and even inverse relationships. These relationships are unique to multilingual scenarios, primarily stemming from differences in text length, writing direction, character types, and regional formatting standards between different languages, and can potentially lead to chain reactions. Specific manifestations include: (1) Causal relationship: Language translation problems directly lead to layout problems. For example, the length of the translated text increases, the writing direction changes (such as from left to right to right to left) or the character type changes (such as letters and pictographs), which may not match the fixed layout in the original design, thus causing layout problems such as text overflow and control misalignment.

[0124] (2) Strong Coupling: Although there is no strong direct causal relationship between format conversion problems (such as currency, time, and number formats) and translation or layout problems, they are highly coupled in the same detection area. For example, incorrect regional formatting may occur simultaneously with language translation errors; on the other hand, changes in the width of format symbols (such as currency symbols) may indirectly lead to layout problems. At the same time, translation problems (such as incorrect target language settings) may also lead to misjudgments of format problems.

[0125] (3) Secondary correlation: Severe layout problems may have a reverse effect on the detectability of translation or formatting problems. For example, when text is truncated due to layout overflow or occluded due to element overlap, the model may not be able to fully identify the text content or formatting symbols in the occluded area, thus causing underlying translation or formatting errors to be missed.

[0126] Based on the understanding of the complex relationships described above, this specification provides a page detection method that can effectively identify and utilize these relationships. Please refer to... Figure 5 This specification also provides a page detection method based on a multi-task detection model in its embodiments. Please refer to [link / reference]. Figure 2 The multi-task detection model includes a shared feature extraction network, a cross-task associated feature extraction network, and multiple detection layers that correspond one-to-one with multiple detection tasks; this method can be executed by an electronic device, and the method includes: In S500, page information of the page to be detected is obtained; the page to be detected is obtained by converting the page presented in the original language with the target language as the conversion direction.

[0127] For example, the page information of the page to be detected includes: the page image of the page to be detected, the page structured data, the original language information, and the target language information.

[0128] The original language information includes, but is not limited to, at least one of the following: original language identifier, original language text content of the page to be detected, character encoding / rendering code adapted to the original language, and inherent attributes such as original language writing direction / character length.

[0129] Target language information includes, but is not limited to, at least one of the following: target language identifier, target language translated text content of the page to be tested, target language localization format conversion / layout adaptation code, inherent attributes such as special characters / writing rules of the target language, and translation mapping relationship between the original language and the target language.

[0130] In S502, the shared feature extraction network processes the page information of the page to be detected to generate shared features.

[0131] For example, please refer to Figure 3The shared feature extraction network includes a visual encoder, a text encoder, and a cross-modal fusion layer.

[0132] Electronic devices encode page images using a visual encoder to extract pixel-level image features such as layout, style, and text rendering effects; they also encode page structured data, raw language information, and target language information using a text encoder to extract text features such as element attributes, text semantics, and language context; and finally, they fuse image and text features through a cross-modal fusion layer to obtain shared features.

[0133] The cross-modal fusion layer establishes semantic alignment and information complementarity between image modalities and text modalities. The final output shared feature is a dense vector representation that combines visual appearance information and textual semantic information, providing a unified and information-rich foundation for various detection tasks that require simultaneous understanding of vision and semantics.

[0134] In S504, shared features are processed through a cross-task association feature extraction network. For each of the multiple detection tasks, the task features of that detection task are extracted from the shared features and the association features from other detection tasks are fused together to output the enhanced features corresponding to that detection task.

[0135] For example, please refer to Figure 3 The cross-task associated feature extraction network includes multiple feature extraction structures that correspond one-to-one with multiple detection tasks.

[0136] The feature extraction structure for each detection task includes: The feature projection layer is used to project shared features to generate query vectors, key vectors, and value vectors corresponding to the detection task. Each feature projection layer includes a query transformation matrix, a key transformation matrix, and a value transformation matrix. The query vector is obtained by multiplying the query transformation matrix with the shared features, the key vector is obtained by multiplying the key transformation matrix with the shared features, and the value vector is obtained by multiplying the value transformation matrix with the shared features.

[0137] The cross-task attention layer performs attention calculations between the query vector of the current detection task and the key vectors of all detection tasks. This evaluates the relevance of the current detection task to each of the other detection tasks, yielding attention weights that characterize cross-task associations. Based on these attention weights, the value vectors of all detection tasks are weighted and fused to generate and output the enhanced features for the corresponding detection task. The attention weight for any given detection task is obtained by normalizing the similarity between the query vector of the current detection task and the key vectors of all detection tasks.

[0138] By using a cross-task attention layer, the augmented features ultimately used for each detection task not only include the information that the detection task itself focuses on in the shared features, but also dynamically and on demand incorporate relevant information provided by other detection tasks.

[0139] For example, the enhanced features of layout detection tasks may incorporate the "text length variation" feature from translation detection tasks, thereby enabling more accurate prediction of potential overflow issues.

[0140] For example, the enhanced features of format conversion detection tasks may incorporate the "language region" features provided by translation tasks, thereby more accurately determining whether formats such as currency and dates conform to the target region specifications.

[0141] For example, the enhancement features of translation detection tasks may incorporate the "text truncation or occlusion" features identified by layout tasks, thereby enabling careful assessment of the completeness of the translated content in the region and avoiding incorrect judgments about incomplete text.

[0142] For example, the enhanced features of the layout detection task may incorporate the "symbol width or format length" feature provided by the format conversion task (such as the width difference between long date format and short date format), so as to more accurately detect layout misalignment problems caused by changes in the size of format elements.

[0143] For example, the enhanced features of translation detection tasks may incorporate the "number or date format pattern" features discovered by format conversion tasks, thereby enabling collaborative verification of the consistency between text content and embedded formats, and helping to identify legacy format issues that were not converted synchronously with translation.

[0144] In S506, the corresponding enhanced features are detected through the detection layer corresponding to each detection task, and the detection results of each detection task are output; among them, multiple detection tasks include at least the following two types: translation detection task, format conversion detection task, and layout detection task.

[0145] For example, each detection task's corresponding detection layer includes a classifier and a locator. The classifier is used to output a classification label for the detected problem under that detection task based on the corresponding enhanced features. The locator is used to output the problem's location information in the page image based on the corresponding enhanced features.

[0146] All detection layers work in parallel, ultimately outputting the detection results of all tasks synchronously (e.g., simultaneously outputting lists of translation errors, layout problems, and formatting problems). Because these results are generated based on shared features enhanced through cross-task correlation, they possess inherent logical consistency and naturally reflect the potential relationships between problems. Furthermore, multiple detection tasks can be completed in parallel within a single forward propagation, significantly reducing overall computational overhead and memory usage compared to schemes using multiple independent models connected in series or parallel, thus improving detection efficiency.

[0147] Since the detection results of each detection task originate from shared and correlated enhanced features, they have inherent consistency in space and logic, making it possible to automatically analyze the co-occurrence, causal, and other relationships between problems, thereby supporting the generation of more insightful comprehensive diagnostic reports, rather than simple problem lists.

[0148] In one possible implementation, the electronic device can analyze the detection results corresponding to each detection task to identify multiple problems coexisting in the same image region within the page image. For example, the electronic device performs spatial clustering analysis on the problems output by each detection task, which have precise location information (bounding boxes). By calculating the spatial relationships (such as the Intersection over Union (IoU)) between different problem prediction boxes, it identifies multiple problems coexisting in the same or adjacent image regions within the page image. For instance, a button control area may be simultaneously marked with both "incorrect translation" and "layout overflow" problems. This step initially integrates the discrete detection results based on spatial location to obtain at least one problem set, where each problem set includes multiple problems coexisting in the same image region.

[0149] For each identified set of problems, the electronic device calls predefined association rules to analyze the set of problems, and determines the relationships between multiple problems in the set and the root cause of the problem.

[0150] For example, each association rule can be represented as an "IF-THEN" logical assertion, typically containing the following key fields: ① Triggering condition group: Defines the prerequisites for the rule to take effect, such as the attributes of the image area corresponding to the problem set, such as control boxes, text areas, numerical display areas, etc. ② Root problem: Refers to the most fundamental defect that is upstream in the causal chain. It is usually the target point that needs to be fixed first. ③ Associated problems: Refers to derivative problems that are directly or indirectly caused by the root problem, or that are highly coupled with it in the same context.

[0151] For example, association rules include, but are not limited to: Rule 1: {Root cause: Mistranslation (long text languages ​​such as German / Russian), Related issue: Text overflow, Detection area: Same control box}.

[0152] Rule 2: {Root cause: Missed translation (source + target text coexist), Related issues: Text overflow / overlap, Detection area: Same control box}.

[0153] Rule 3: {Root cause: Currency format error (symbol position / separator error), Related issue: Text overflow, Detection area: Same text area}.

[0154] Rule 4: {Root cause: Translation of less commonly spoken languages ​​introduces accented characters (such as French é, Spanish ñ), related issues: text overflow / control border truncation, detection area: the same text display control}.

[0155] Rule 5: {Root cause: Incorrect translation of units of measurement (e.g., “centimeter” translated as “centimeter” instead of “cm”), Related issues: Text overlap, Detection area: Area displaying the same value}.

[0156] Rule 6: {Root cause: Missing text truncation rules in the target region (e.g., Japanese vertical text does not adapt to line break logic), Related issues: Text overlap / semantic break, Detection area: Vertical text container}.

[0157] Rule 7: {Root cause: Multilingual font compatibility error (e.g., Arabic text rendered with Latin font), Related issues: Layout misalignment / text overlap, Detection area: Entire page}.

[0158] Rule 8: {Root cause: Incorrect date format (incorrect order, such as using MM / DD / YYYY for DD / MM / YYYY regions), Related issues: Semantic confusion / user misunderstanding, Detection area: Display area for the same date}.

[0159] Finally, based on the detection results of each detection task and the correlations and root causes derived from the above analysis, the electronic device generates a comprehensive problem list. This list may, for example, use the root cause as the main entry, with indented subheadings listing the derivative problems it causes; or, it may provide a summary description for each problem cluster, clearly stating that "problem A leads to problem B, and it is recommended to fix problem A first."

[0160] For example, a comprehensive problem list can be organized around root causes, clearly showing the problem chain. For example, as shown below: This embodiment, through correlation analysis, can clearly identify the root cause of the problem. This allows developers or testers to directly pinpoint the essential defects that need to be prioritized for fixing, such as correcting German translation text first, rather than blindly dealing with superficial issues like adjusting button widths. This significantly improves the efficiency and accuracy of the fixes. Furthermore, by automatically performing correlation analysis and root cause localization through predefined rules, it replaces the tedious comparison and reasoning work that originally relied on human experts. This not only greatly improves analysis efficiency but also avoids inconsistencies in conclusions caused by the subjectivity or fatigue of human judgment, ensuring the objectivity and repeatability of the problem diagnosis results.

[0161] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0162] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 6 As shown, device 600 mainly consists of a communication interface 602, a user interface 604, a processor 606, and a data storage 608. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 610. The communication interface 602 enables device 600 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 602 may include an antenna and related processing devices for wireless communication with a radio access network or access point. Furthermore, the communication interface 602 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 602 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 602 may also include multiple physical communication interfaces, such as Wi-Fi interfaces, Bluetooth interfaces, and wide-area wireless interfaces.

[0163] User interface 604 includes receiving user input and providing output to the user. Therefore, user interface 604 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 604 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 604 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 600 may support remote access from other devices via communication interface 602 or another physical interface (not shown). User interface 604 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 604 may also be configured as a display device for rendering or displaying text fragments.

[0164] Processor 606 may contain one or more general-purpose processors and / or special-purpose processors.

[0165] Data storage 608 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 606. Data storage 608 may include removable and non-removable components.

[0166] Processor 606 is capable of executing program instructions 618 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 608 to perform the various functions described herein. Data storage 608 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 600, enable device 600 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 618 by processor 606 may result in processor 606 using data 612.

[0167] For example, program instructions 618 may include an operating system 622 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 600 and one or more applications 620 (e.g., a browser, social application, or game application). Similarly, data 612 may include operating system data 616 and application data 614. Operating system data 616 is primarily accessible to the operating system 622, while application data 614 is primarily accessible to one or more applications 620. Application data 614 may reside in a file system visible or hidden from the user of device 600.

[0168] Application 620 can communicate with operating system 622 through one or more application programming interfaces (APIs). These APIs help application 620 read and / or write application data 614, transmit or receive information via communication interface 602, receive or display information on user interface 604, etc.

[0169] In some terminology, application 620 may be simply referred to as "app". Furthermore, application 620 can be downloaded to device 600 through one or more online app stores or app markets. However, applications can also be installed on device 600 in other ways, such as through a web browser or a physical interface on device 600 (e.g., a USB port).

[0170] In some embodiments, the page detection device based on a multi-task detection model can be applied to, for example... Figure 6 The device shown implements the technical solution described in this specification. The multi-task detection model includes a shared feature extraction network, a cross-task associated feature extraction network, and multiple detection layers, each corresponding to a specific detection task. The device may include: The page information acquisition module is used to acquire the page information of the page to be tested; the page to be tested is obtained by converting the page presented in the original language with the target language as the conversion direction.

[0171] The shared feature generation module is used to process the page information of the page to be detected through the shared feature extraction network and generate shared features.

[0172] The enhanced feature generation module is used to process shared features through a cross-task associated feature extraction network. For each of multiple detection tasks, it extracts the task features of the detection task itself from the shared features and fuses the associated features from other detection tasks to output the enhanced features corresponding to the detection task.

[0173] The detection module is used to detect the corresponding enhancement features through the detection layer corresponding to each detection task and output the detection results of each detection task; among them, the multiple detection tasks include at least the following two types: translation detection task, format conversion detection task, and layout detection task.

[0174] In other embodiments, the training apparatus for the multi-task detection model can be applied to, for example... Figure 6 The device shown implements the technical solution described in this specification. The multi-task detection model includes a shared feature extraction network, a cross-task associated feature extraction network, and multiple detection layers, each corresponding to a specific detection task. The device may include: The sample acquisition module is used to acquire training samples, which include page images, page structured data, raw language information and target language information, and are labeled with the ground truth values ​​of the problems corresponding to multiple detection tasks.

[0175] The training module is used to input training samples into the multi-task detection model to be trained and obtain the prediction results of each detection task output by the model.

[0176] The training module is also used to calculate the difference between the prediction results of each detection task and the corresponding true value of the problem, so as to obtain the task loss corresponding to each detection task.

[0177] The training module is also used to generate an association penalty loss if the set of question types predicted by the multi-task detection model for the same image region is inconsistent with the set of question types labeled.

[0178] The training module is also used to calculate the total loss of model training based on the loss of each task and the correlation penalty loss.

[0179] The training module is also used to optimize the parameters of the multi-task detection model with the goal of minimizing the total loss.

[0180] In yet other embodiments, the training apparatus for the multi-task detection model can be applied to, for example... Figure 6 The device shown implements the technical solution described in this specification. The multi-task detection model includes a shared feature extraction network, a cross-task associated feature extraction network, and multiple detection layers, each corresponding to a specific detection task. The device may include: The sample acquisition module is used to acquire training samples, which include page images, page structured data, raw language information and target language information, and are labeled with the ground truth values ​​of the problems corresponding to multiple detection tasks.

[0181] The training module is used to input training samples into a multi-task detection model that acts as an agent, and to obtain the prediction results of each detection task output by the model.

[0182] The training module is also used to calculate the precision reward, false alarm penalty, and false negative penalty for the current sample based on the difference between the prediction results of each detection task and the corresponding true value of the problem.

[0183] The training module is also used to generate an association reward or association penalty for training samples labeled with at least two types of questions for the same image region, based on the comparison results between the set of question types predicted by the multi-task detection model for the image region and the set of labeled question types.

[0184] The training module is also used to calculate the total reward based on the accuracy reward, false alarm penalty, false negative penalty, and correlation reward or correlation penalty.

[0185] The training module is also used to update the parameters of the multi-task detection model with the goal of maximizing the total reward.

[0186] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0187] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor executes the executable instructions to implement the steps of the method as described in any of the above embodiments.

[0188] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0189] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0190] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0191] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0192] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0193] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0194] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0195] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0196] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A page detection method based on a multi-task detection model, wherein the multi-task detection model includes a shared feature extraction network, a cross-task associated feature extraction network, and multiple detection layers corresponding one-to-one with multiple detection tasks; the method includes: Obtain page information of the page to be tested; wherein, the page to be tested is obtained by converting the page presented in the original language with the conversion direction of adapting to the target language; The shared feature extraction network processes the page information of the page to be detected to generate shared features. The shared features are processed by the cross-task association feature extraction network. For each of the multiple detection tasks, the task features of the detection task itself are extracted from the shared features and the association features from other detection tasks are fused together to output the enhanced features corresponding to the detection task. The corresponding enhanced features are detected through the detection layer corresponding to each detection task, and the detection results of each detection task are output; wherein, the plurality of detection tasks include at least the following two types: translation detection task, format conversion detection task, and layout detection task.

2. The method according to claim 1, wherein the cross-task associated feature extraction network includes multiple feature extraction structures that correspond one-to-one with multiple detection tasks; wherein The feature extraction structure corresponding to each of the detection tasks includes: The feature projection layer is used to project the shared features to generate query vectors, key vectors, and value vectors corresponding to the detection task; The cross-task attention layer is used to perform attention calculations on the query vector of this detection task and the key vectors of all detection tasks to obtain attention weights that represent cross-task associations. Based on the attention weights, the value vectors of all detection tasks are weighted and fused to generate and output the enhanced features of the corresponding detection tasks.

3. The method according to claim 2, Each of the feature projection layers includes a query transformation matrix, a key transformation matrix, and a value transformation matrix; wherein The query vector is obtained by multiplying the query transformation matrix by the shared feature; the key vector is obtained by multiplying the key transformation matrix by the shared feature; and the value vector is obtained by multiplying the value transformation matrix by the shared feature. The attention weight corresponding to any detection task is obtained by normalizing the similarity between the query vector of this detection task and the key vectors of all detection tasks.

4. The method according to claim 1, wherein the page information includes the page image of the page to be detected; The detection layer corresponding to each of the aforementioned detection tasks includes a classifier and a locator; The classifier is used to output a classification label of the problem detected in the detection task based on the corresponding enhanced features. The locator is used to output the location information of the problem in the page image based on the corresponding enhanced features.

5. The method according to claim 1, wherein the page information includes: The page image, page structured data, raw language information, and target language information of the page to be detected; The shared feature extraction network includes a visual encoder, a text encoder, and a cross-modal fusion layer; The step of processing the page information of the page to be detected through the shared feature extraction network to generate shared features includes: The page image is encoded using the visual encoder to extract image features; The text encoder encodes the page structured data, the original language information, and the target language information to extract text features; The image features and text features are fused through the cross-modal fusion layer to obtain the shared features.

6. The method according to any one of claims 1 to 5, wherein the page information includes a page image of the page to be detected; the method further includes: Analyze the detection results corresponding to each of the detection tasks to identify multiple problems coexisting in the same image region in the page image; Based on predefined association rules, the association relationships and root causes among the multiple issues are determined; Based on the detection results of each detection task, as well as the identified correlations and root causes, a comprehensive list of issues for the page image is generated.

7. A method for training a multi-task detection model, wherein the multi-task detection model is applied to the method according to any one of claims 1 to 6, the method comprising: Acquire training samples, which include page images, page structured data, raw language information and target language information, and are labeled with the ground truth values ​​of the problems corresponding to multiple detection tasks. The training samples are input into the multi-task detection model to be trained, and the prediction results of each detection task output by the model are obtained. The difference between the prediction result of each detection task and the corresponding true value of the problem is calculated separately to obtain the task loss for each detection task. For training samples in the same image region labeled with at least two types of questions, if the set of question types predicted by the multi-task detection model for the image region is inconsistent with the set of labeled question types, an association penalty loss is generated. Calculate the total loss of model training based on the task loss and the correlation penalty loss described above; The parameters of the multi-task detection model are optimized with the goal of minimizing the total loss.

8. A training method for a multi-task detection model, wherein the multi-task detection model is applied to the method according to any one of claims 1 to 6, the method comprising: Acquire training samples, which include page images, page structured data, raw language information and target language information, and are labeled with the ground truth values ​​of the problems corresponding to multiple detection tasks. The training samples are input into a multi-task detection model that serves as the agent, and the prediction results of each detection task output by the model are obtained. Based on the difference between the prediction results of each detection task and the true value of the corresponding problem, calculate the precision bonus, false alarm penalty and false negative penalty for the current sample; For training samples in the same image region labeled with at least two types of questions, an association reward or association penalty is generated based on the comparison between the set of question types predicted by the multi-task detection model for the image region and the set of labeled question types. The total reward is calculated based on the accuracy reward, the false alarm penalty, the missed detection penalty, and the correlation reward or the correlation penalty. The parameters of the multi-task detection model are updated with the goal of maximizing the total reward.

9. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-8 by executing the executable instructions.

10. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-8.

Images