Scene image testing method, apparatus, device, and medium

Abstract

The present disclosure provides a scene image testing method, device and equipment and a storage medium. The method comprises: obtaining a scene image to be tested; using a scene classifier in a pre-trained testing model to perform scene detection on the scene image to be tested to obtain a scene detection result corresponding to the scene image to be tested; and using a text recognition network in the pre-trained testing model to perform text recognition on the scene image to be tested to obtain a text recognition result corresponding to the scene image to be tested, wherein the text recognition comprises at least one or a combination of control text detection, text content recognition, text language classification and text direction classification. Thus, the multifunctional testing model is used to test the scene image to be tested in multiple dimensions to obtain a testing result meeting multiple testing requirements, without stacking multiple testing services, thereby reducing the consumption of testing resources and improving the testing efficiency of the scene image.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, device and medium for testing scene images. Background Technology

[0002] Scene image testing refers to testing the scene images generated by an application before it goes live, making it an important task in the application development process. For example, for various game applications developed by game development teams, in order to ensure that users can play the game normally when the application goes live, it is usually necessary to test the scene images generated by the game application before its release.

[0003] Current scene image testing methods typically involve layering multiple different testing services to address varying scene image testing needs. For example, if scene image testing requirements include scene classification, control text detection, text content recognition, text language classification, and text direction classification, then five different testing services need to be layered to perform comprehensive testing of the scene image and obtain the results. However, layering multiple testing services consumes significant testing resources, and each service can only produce one result at a time, resulting in low testing efficiency for scene images. Therefore, proposing a low-energy-consumption and high-efficiency scene image testing method is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a scene image testing method, apparatus, device, and medium.

[0005] In a first aspect, this disclosure provides a scene image testing method, the method comprising:

[0006] Obtain the image of the scene to be tested;

[0007] Using the scene classifier in the pre-trained test model, scene detection is performed on the scene image to be tested to obtain the scene detection result corresponding to the scene image to be tested;

[0008] Using a pre-trained text recognition network in a test model, text recognition is performed on the test scene image to obtain the text recognition result corresponding to the test scene image. The text recognition includes at least one or more combinations of control text detection, text content recognition, text language classification, and text direction classification.

[0009] Secondly, this disclosure provides a scene image testing device, the device comprising:

[0010] The acquisition module is used to acquire images of the scene to be tested.

[0011] The scene detection module is used to perform scene detection on the scene image to be tested using the scene classifier in the pre-trained test model, and obtain the scene detection result corresponding to the scene image to be tested.

[0012] The text recognition module is used to perform text recognition on the test scene image using a text recognition network in a pre-trained test model, and obtain the text recognition result corresponding to the test scene image. The text recognition includes at least one or more combinations of control text detection, text content recognition, text language classification, and text direction classification.

[0013] Thirdly, this disclosure provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the above-described method.

[0014] Fourthly, this disclosure provides an apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described above.

[0015] Fifthly, this disclosure provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the method described above.

[0016] The technical solution provided in this disclosure has at least the following advantages compared with the prior art:

[0017] This disclosure provides a method, apparatus, device, and medium for testing scene images. The method includes: acquiring a scene image to be tested; performing scene detection on the scene image to be tested using a scene classifier in a pre-trained test model to obtain a scene detection result corresponding to the scene image to be tested; and performing text recognition on the scene image to be tested using a text recognition network in a pre-trained test model to obtain a text recognition result corresponding to the scene image to be tested. The text recognition includes at least one or more combinations of control text detection, text content recognition, text language classification, and text direction classification. Therefore, when testing scene images, a multi-functional test model is used to test the scene image to be tested in multiple dimensions, obtaining test results that meet multiple testing requirements, thus achieving end-to-end testing. This eliminates the need to stack multiple test services, reducing the consumption of test resources and improving the testing efficiency of scene images. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a scene image testing method provided in some embodiments of this disclosure;

[0021] Figure 2 This is a schematic diagram of the structure of the test model provided in some embodiments of this disclosure;

[0022] Figure 3 This is a schematic diagram of the structure of a multilayer sensor provided in some embodiments of this disclosure;

[0023] Figure 4 This is a schematic diagram of the structure of the control detector provided in some embodiments of this disclosure;

[0024] Figure 5 This is a schematic diagram of the structure of a converter provided in some embodiments of this disclosure;

[0025] Figure 6 This is a schematic diagram of the structure of a character recognition device provided in some embodiments of this disclosure;

[0026] Figure 7 This is a schematic diagram of the structure of a scene image testing device provided in some embodiments of this disclosure;

[0027] Figure 8 This is a schematic diagram of the structure of a scene image testing device provided in some embodiments of this disclosure. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0029] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0030] Figure 1A schematic flowchart of a scene image testing method provided in an embodiment of this disclosure is shown. Figure 1 As shown, the scene image testing method includes the following steps.

[0031] S110. Obtain the image of the scene to be tested.

[0032] In this embodiment, when automated testing of games or virtual reality scenes is required, page images under different testing scenarios are obtained based on the business requirements of automated testing, and used as the test scene images.

[0033] The scene image to be tested refers to the original page image used for testing. Specifically, the scene image to be tested can include various scenes, controls, and text. During scene image testing, different scene categories, control information, and text information are identified.

[0034] Optionally, the test scene image can be a two-dimensional image or a three-dimensional image; there is no restriction here.

[0035] S120. Using the scene classifier in the pre-trained test model, perform scene detection on the test scene image to obtain the scene detection result corresponding to the test scene image.

[0036] The test model is pre-trained on a preset network based on multiple sets of training samples, thus obtaining a test model that can be tested uniformly across multiple dimensions.

[0037] Specifically, multiple sets of training samples include sample scene images and scene labels. During model training, the scene detection module in the preset network is trained using the sample scene images and scene labels to obtain a scene classifier, which is then used to perform scene detection on the test image based on the scene classifier in the test model.

[0038] S130. Using the text recognition network in the pre-trained test model, perform text recognition on the test scene image to obtain the text recognition result corresponding to the test scene image. The text recognition includes at least one or more combinations of control text detection, text content recognition, text language classification, and text direction classification.

[0039] Specifically, the multiple training samples also include one or more of the following: control text labels, text content labels, text language labels, and text direction labels. The text recognition network in the preset network is trained in advance using one or more of the control text labels, text content labels, text language labels, and text direction labels, as well as the sample scene image, to obtain the text recognition network, which enables the text recognition network in the test model to perform text recognition on the test image.

[0040] In some embodiments, the text recognition network may include one or more combinations of a control detector, a text recognizer, a language recognizer, and a direction recognizer. The control detector is used to detect control text in the test scene image to obtain control detection results; the text recognizer is used to recognize text content in the test scene image to obtain content recognition results; the language recognizer is used to classify the language of the text in the test scene image to obtain language recognition results; and the direction recognizer is used to classify the language of the text in the test scene image to obtain direction recognition results.

[0041] In other embodiments, the text recognition network may include one or more of a control detector, a character recognizer, a language recognizer, a direction recognizer, and a converter. The converter connects the control detector and the character recognizer, enabling joint supervision by the converter to jointly optimize the control detector and the character recognizer, thereby improving the performance of the entire test model and increasing the interaction between the control detector and the character recognizer.

[0042] In other scenarios, the test model may also include a backbone network and a region extraction network. The backbone network is used to downsample the image of the scene to be tested, enabling the scene classifier to perform scene detection on the downsampled features output by the backbone network, and enabling the text recognition network to perform text recognition on the downsampled features output by the backbone network. The region extraction network further extracts features from the downsampled features generated by the backbone network, enabling the scene classifier to perform scene detection on the target region features output by the region extraction network, and enabling the text recognition network to perform text recognition on the target region features output by the region extraction network.

[0043] This disclosure provides a scene image testing method, which includes: acquiring a scene image to be tested; using a scene classifier in a pre-trained testing model to perform scene detection on the scene image to be tested, obtaining a scene detection result corresponding to the scene image to be tested; and using a text recognition network in the pre-trained testing model to perform text recognition on the scene image to be tested, obtaining a text recognition result corresponding to the scene image to be tested. The text recognition includes at least one or more combinations of control text detection, text content recognition, text language classification, and text direction classification. Therefore, when conducting scene image testing, a multi-functional testing model is used to test the scene image to be tested in multiple dimensions, obtaining test results that meet multiple testing requirements, thus achieving end-to-end testing. Therefore, it eliminates the need to stack multiple testing services, reducing the consumption of testing resources and improving the testing efficiency of scene images.

[0044] In another embodiment of this disclosure, the test model further includes a backbone network, which is used to downsample the image of the scene to be tested, so that the scene classifier performs scene detection on the downsampled features output by the backbone network, and the text recognition network performs text recognition on the downsampled features output by the backbone network.

[0045] In this embodiment of the disclosure, optionally, the test model further includes a backbone network; correspondingly, before S120, the method further includes:

[0046] S111. Based on the backbone network in the test model, downsampling is performed on the image of the scene to be tested to obtain the downsampling features corresponding to the image of the scene to be tested; S120 specifically includes:

[0047] S1201. Based on the scene classifier in the test model, perform scene detection on the downsampled features to obtain the scene detection results; S130 specifically includes:

[0048] S1301. Based on the text recognition network in the test model, perform text recognition on the downsampled features to obtain the text recognition result.

[0049] The backbone network includes multiple attention modules built on moving windows; S111 specifically includes: using multiple attention modules built on moving windows to perform multiple feature interaction calculations on the test scene image between consecutive self-attention layers to generate downsampled features.

[0050] See Figure 2 The diagram shows the structure of the test model. The image of the scene to be tested is input into the test model. The backbone network in the test model performs downsampling processing on the image of the scene to be tested to obtain downsampled features, which are then used for scene detection and text recognition.

[0051] In this embodiment, the backbone network can be Swin-Transformer. Since Swin-Transformer can divide image features into multiple patches, and these patches have interactive relationships, when using Swin-Transformer to downsample the test scene image, it can effectively utilize the interactive relationships between the patches to achieve global downsampling. When using Swin-Transformer as the backbone network, assuming the size of the test scene image is H*W*3, firstly, the patch partitioning module in Swin-Transformer divides the test scene image into multiple equal patches, each with a size of H*W*3. Next, the Swin-Transformer downsamples multiple blocks sequentially through four stages. Specifically, the first stage downsamples blocks of size [size missing]. Downsampling is performed on multiple blocks to generate a size of Multiple blocks, after the second stage, with a size of Downsampling is performed on multiple blocks to generate a size of Multiple blocks, after the second stage, with a size of Downsampling is performed on multiple blocks to generate a size of Multiple blocks, after stage 4, with a size of Multiple blocks are downsampled to generate a size of The system divides the network into multiple blocks and uses this feature as the downsampled feature of the backbone network output. The first stage includes a linear embedding layer and a Swin Transformer block. The linear embedding layer is used to divide the network into multiple blocks of size 1. The dimensions of multiple blocks are transformed into preset dimensions. The Swin Transformer module is used to perform self-attention processing on the features, generating a size of [size missing]. The second stage consists of multiple blocks; it includes a patch merging layer and a Swin Transformer module. The patch merging layer performs pooling-like operations without losing relevant information, and the Swin Transformer module continues to perform self-attention processing on the features, generating a size of [size missing]. The third stage includes a block merging layer and a Swing Transformer module, and after passing through the block merging layer and the Swing Transformer module, a block of size is generated. The fourth stage includes a block merging layer and a Swing Transformer module, and after passing through the block merging layer and the Swing Transformer module, a size of [size missing] is generated. Multiple blocks.

[0052] Specifically, the Swin Transformer module comprises two cascaded structures. The first structure includes two linear layers, one window multi-head self-attention (W-MSA) layer, and one multilayer perceptron (MLP). The second structure includes two linear layers, one shifted window W-MSA layer, and one MLP. The linear layers perform linear processing on the input features to obtain linear features. The MLP performs feature mapping. The window and shifted window W-MSA layers capture the information interaction brought about by pane movement. Information between different panes in the previous layer is effectively connected in the next layer, so that feature maps within independent panes are assigned to the new panes after movement, realizing a more complex interaction mechanism to achieve multiple feature interaction calculations between consecutive self-attention layers.

[0053] Therefore, since the Swin-Transformer takes into account the interaction characteristics between multiple blocks, when the Swin-Transformer is used as the backbone network, it can effectively utilize the global downsampling capability and computational efficiency, thus ensuring the downsampling effect and efficiency of the backbone network.

[0054] The scene classifier is a multilayer perceptron. Specifically, S1201 includes: using the multilayer perceptron to predict the scene category of the downsampled features, obtaining at least one scene category, and using the at least one scene category as the scene detection result.

[0055] See also Figure 2 The downsampled features are used as input data for a scene classifier. The scene classifier then predicts the scene category based on the downsampled features to obtain predicted labels. Different predicted labels represent different scene categories, and these different scene categories are used as the scene detection results. Optionally, scene categories may include, but are not limited to, login, pop-up, settings, payment, editing, and general categories.

[0056] Figure 3 A schematic diagram of a multilayer perceptron is shown. Figure 2 As shown, the multilayer perceptron specifically includes two linear layers, one activation function (GELU), and two random deactivation functions (Dropout). The linear layers perform linear processing on the input features to obtain linear features, the activation function performs regularization on the input features to obtain regularized features, and the random deactivation functions perform random deactivation on the input features to obtain the retained features.

[0057] Therefore, by using a multilayer perceptron as a scene classifier, the downsampled features are processed in multiple stages to obtain at least one scene category, thus ensuring the reliability of the scene classification results.

[0058] Before executing S1301, the method further includes: using the region extraction network in the test model to extract target region features from the downsampled features; then, S1301 specifically includes: performing text recognition on the target region features based on the text recognition network to obtain the text recognition result.

[0059] See also Figure 2 The downsampled features are used as input data for the Region Extraction Network (RoI Align), and the RoI Align network is used to extract features from the downsampled features to obtain the target region features. Then, the text recognition network is used to perform text recognition on the target region features to obtain the text recognition result.

[0060] The target area can be a local area containing features such as controls and text.

[0061] Therefore, a region extraction network is added before the text recognition network, so that the region extraction network can identify local regions in the image, and the text recognition network can perform text recognition only on local regions, thereby improving the text recognition efficiency.

[0062] In summary, by employing multiple attention modules based on moving windows, multilayer perceptrons, region extraction networks, and text recognition networks, scene detection and text recognition are performed on the test scene images. This effectively utilizes the interactive characteristics between image features while ensuring the model's processing performance and reliability.

[0063] In another embodiment of this disclosure, the text recognition network includes a control detector, a text recognizer, a language classifier, and an orientation classifier to perform text detection, text content recognition, text language classification, and text orientation classification on the image of the scene to be tested.

[0064] As described in the foregoing embodiments, before executing S1301, the method further includes: using a region extraction network in the test model to extract target region features from downsampled features; S1301 specifically includes: performing text recognition on the target region features based on a text recognition network to obtain text recognition results.

[0065] In this embodiment of the disclosure, the optional step of "performing text recognition on the features of the target region based on a text recognition network to obtain text recognition results" specifically includes:

[0066] S13011. Based on the control detector in the text recognition network, control text detection is performed on the target region features to obtain control detection results;

[0067] S13012. Based on the text recognizer in the text recognition network, perform text content recognition on the target area features and control detection results to obtain the content recognition result;

[0068] S13013. Based on the language recognizer in the text recognition network, the target region features and control detection results are classified into text languages ​​to obtain the language recognition result;

[0069] S13014. Based on the orientation recognizer in the text recognition network, the text orientation is classified according to the target area features and control detection results to obtain the orientation recognition result;

[0070] The text recognition result consists of control detection results, content recognition results, language recognition results, and direction recognition results.

[0071] Specifically, S13011 includes: starting from the first detection stage, for the current detection stage, using the control detector to perform control text detection based on the target region features corresponding to the previous detection stage and the confidence level predicted by the previous detection stage, to obtain the confidence level corresponding to the current detection stage; for the next detection stage, using the control detector to continue performing control text detection based on the target region features corresponding to the current detection stage and the confidence level corresponding to the current detection stage, until the next detection stage is the last detection stage, to obtain the control detection result.

[0072] In this embodiment, the control detector, like the conventional Transformer, uses a multi-stage query-based detection method to detect text. It designs multiple detection stages, such as six or more, and performs control text detection based on the target region features predicted by the previous detection stage and the confidence level predicted by the previous detection stage in each detection stage. The control detector is applicable to detecting text of any shape.

[0073] See also Figure 2 The control detector includes a dynamic adaptive Transformer encoder, a flattening module, and a detection head module. For details, see [link to documentation]. Figure 4The schematic diagram of the control detector shown indicates that the dynamic adaptive Transformer encoder includes two convolutional layers (Conv), two normalization and activation layers (Norm & Activation), one linear layer, one fusion and normalization layer (Add & Norm), and one multi-head self-attention layer (MSA). k The flattening module is used to output the confidence score, and the detection head module includes three linear layers. Specifically, in the current detection stage, the target region features corresponding to the previous detection stage and the confidence score predicted in the previous detection stage are input into the control detector. These features are processed through two convolutional layers, two standardization and activation layers, one linear layer, one fusion and standardization layer, and one multi-head self-attention layer in the dynamic adaptive Transformer encoder. This processes the target region features and confidence scores from the previous detection stage. The resulting features are then processed by the flattening module to calculate the confidence score for the current detection stage. Finally, the control detection result for the current detection stage is output through the three linear layers in the detection head module. Optionally, the control detection result for the current detection stage includes the control category, control position, and control coordinates. Further, based on the control coordinates in the control detection result of the current detection stage, the target region features corresponding to the current detection stage are determined. The target region features and confidence scores corresponding to the current detection stage are then input into the control detector, and the above steps are repeated until the final control detection result is obtained.

[0074] Therefore, by utilizing the control detector in the text recognition network, control text detection is performed cyclically in multiple control detection stages, thereby improving the accuracy of control text detection.

[0075] Prior to S13012, the method further includes: fusing target region features and control detection results based on a converter in a text recognition network to generate target fusion features;

[0076] Accordingly, S13012 specifically includes: based on the text recognition device, performing text content recognition on the target fusion features and the control positions contained in the control detection results to obtain the content recognition result;

[0077] Accordingly, S13013 specifically includes: classifying the target fusion features into text languages ​​based on a language recognizer to obtain language recognition results;

[0078] Accordingly, S13014 specifically includes: classifying text directions based on the target fusion features using a direction recognizer to obtain direction recognition results.

[0079] Specifically, "based on the converter in the text recognition network, the target region features and control detection results are fused to generate target fusion features" includes: based on the converter, the target region features and control detection results are sequentially encoded, upsampled, activated and fused to obtain target fusion features.

[0080] Understandably, in order to jointly supervise the control detector and the text recognizer to achieve joint optimization, a converter is connected between the control detector and the text recognizer. This allows the high-level semantic information of the text to be encoded using the converter, which facilitates the improvement of the text recognizer's recognition performance, thereby improving the overall performance of the model and greatly increasing the interaction between the control detector and the text recognizer.

[0081] See also Figure 2 The converter is connected after the control detector and before the text recognizer, language recognizer, and direction recognizer. For ease of understanding, Figure 5 A schematic diagram of the converter is shown; see [link / reference]. Figure 5 First, the control detection results and the downsampled features (including a1, a2, a3) corresponding to the target region features are input into the converter. The Transformer Encoder in the converter encodes some of the downsampled features (a1, a2) and the control detection results to generate coded features di, where di includes d1, d2, and d3. Then, a series of upsampling and feature activation processes are performed on d1, d2, and d3 to generate mask features M. i If i = 1, 2, 3, then M i Including M1, M2, and M3; furthermore, M1, M2, M3, and a1, a2, and a3 are fused to generate the target fusion feature r3.

[0082] Optional, d1, d2, d3, M i It can be determined in the following way:

[0083] d1 = TrE(f det )

[0084] d2=(Eu(d1)+a2)

[0085] d3=(Eu(d2)+a1)

[0086] M i =δ(di), i = 1, 2, 3

[0087] Optionally, r3 can be determined in the following way:

[0088] r1 = M 1* a3

[0089] r2=M 2* (Eu(r1)+a2)

[0090] r3=M 3* (Eu(r2)+a1)

[0091] Furthermore, since the converter effectively suppresses background noise, thus obtaining more accurate text region boundaries, a high text recognition accuracy can be achieved using only a simple text recognizer.

[0092] In this embodiment, the text recognizer can be a Transformer decoder. For ease of understanding, Figure 6 A schematic diagram of the structure of a text recognition device provided in an embodiment of this disclosure is shown. (In conjunction with...) Figure 2 and Figure 6 The target fusion features and the control positions contained in the control detection results are used as input data for the text recognizer. The text content is then recognized by two standard layers (Norm), one multi-head attention layer, and one multilayer perceptron (MLP) in the recognizer, and the target fusion features and control positions are used to obtain the content recognition result.

[0093] Furthermore, to predict the language and direction of the text, two multilayer perceptrons (MLPs) were used as the language identifier and direction identifier, respectively. The specific structure of each MLP is similar to that of the scene classifier, and will not be elaborated here. Specifically, the language identifier uses two linear layers, one activation function (GELU), and one dropout function to classify the target fusion features into the language of the text, obtaining the language identification result. The language identification result can include, but is not limited to, English, Chinese, Japanese, and Korean. Similarly, the direction identifier uses two linear layers, one activation function, and one dropout function to classify the target fusion features into the direction of the text, obtaining the direction identification result. The direction identification result can include, but is not limited to, horizontal and vertical text.

[0094] In summary, by connecting a converter between the control detector and the text recognizer, joint optimization of the control detector and the text recognizer can be achieved, which facilitates the improvement of the recognition effect of the text recognizer and thus improves the performance of the entire model. Furthermore, based on the text recognizer, text content recognition is performed on the target fusion features and the control positions contained in the control detection results to ensure the accuracy of the content recognition results. At the same time, a language recognizer and a direction recognizer are used to perform text language classification and text direction classification respectively to ensure the accuracy of language classification and direction classification.

[0095] Based on the same inventive concept as the above-described method embodiments, this disclosure also provides a scene image testing device, with reference to... Figure 7 This is a schematic diagram of the structure of a scene image testing device provided in an embodiment of the present disclosure. The scene image testing device 700 includes:

[0096] The acquisition module 710 is used to acquire images of the scene to be tested;

[0097] The scene detection module 720 is used to perform scene detection on the scene image to be tested using a scene classifier in a pre-trained test model, and obtain the scene detection result corresponding to the scene image to be tested.

[0098] The text recognition module 730 is used to perform text recognition on the test scene image using a text recognition network in a pre-trained test model, and obtain the text recognition result corresponding to the test scene image. The text recognition includes at least one or more combinations of control text detection, text content recognition, text language classification, and text direction classification.

[0099] This disclosure provides a scene image testing apparatus, comprising: acquiring a scene image to be tested; performing scene detection on the scene image to be tested using a scene classifier in a pre-trained testing model to obtain a scene detection result corresponding to the scene image to be tested; and performing text recognition on the scene image to be tested using a text recognition network in a pre-trained testing model to obtain a text recognition result corresponding to the scene image to be tested. The text recognition includes at least one or more combinations of control text detection, text content recognition, text language classification, and text direction classification. Therefore, when performing scene image testing, a multi-functional testing model is used to test the scene image to be tested in multiple dimensions, obtaining test results that meet multiple testing requirements, thus achieving end-to-end testing. Therefore, it eliminates the need to stack multiple testing services, reducing the consumption of testing resources and improving the testing efficiency of scene images.

[0100] In one optional implementation, the test model further includes a backbone network; the device also includes:

[0101] The downsampling module is used to downsample the image of the scene to be tested based on the backbone network in the test model, thereby obtaining the downsampling features corresponding to the image of the scene to be tested; accordingly,

[0102] The scene detection module 720 is specifically used to: perform scene detection on the downsampled features based on the scene classifier in the test model, and obtain the scene detection result;

[0103] The text recognition module 730 is specifically used to: perform text recognition on the downsampled features based on the text recognition network in the test model, and obtain the text recognition result.

[0104] In one optional implementation, the backbone network includes multiple attention modules built based on moving windows; the downsampling module is specifically used for:

[0105] By utilizing the multiple attention modules constructed based on moving windows, the test scene image is subjected to multiple feature interaction calculations between consecutive self-attention layers to generate the downsampling features.

[0106] In one optional implementation, the scene classifier is a multilayer perceptron; the scene detection module 720 is specifically used to: use the multilayer perceptron to predict the scene category of the downsampled features to obtain at least one scene category, and use the at least one scene category as the scene detection result.

[0107] In one optional embodiment, the device further includes:

[0108] The feature extraction module is used to extract target region features from the downsampled features using the region extraction network in the test model;

[0109] Accordingly, the text recognition module 730 is specifically used for:

[0110] Based on the text recognition network, text recognition is performed on the features of the target region to obtain the text recognition result.

[0111] In one optional implementation, the text recognition module 730 includes:

[0112] The detection unit is used to perform control text detection on the features of the target region based on the control detector in the text recognition network, and obtain control detection results;

[0113] The recognition unit is used to perform text content recognition based on the text recognizer in the text recognition network, the features of the target area and the control detection results, and obtain the content recognition result.

[0114] The first classification unit is used to classify the target region features and the control detection results based on the language recognizer in the text recognition network to obtain the language recognition result.

[0115] The second classification unit is used to classify the text direction based on the direction recognizer in the text recognition network, the target region features and the control detection results, and obtain the direction recognition result.

[0116] The control detection result, the content recognition result, the language recognition result, and the direction recognition result constitute the text recognition result.

[0117] In one optional implementation, the detection unit is specifically used for:

[0118] Starting from the first detection stage, for the current detection stage, the control detector is used to detect control text based on the target region features corresponding to the previous detection stage and the confidence level predicted by the previous detection stage, so as to obtain the confidence level corresponding to the current detection stage.

[0119] For the next detection stage, the control detector is used to continue detecting control text based on the target region features and the confidence level corresponding to the current detection stage, until the next detection stage is the last detection stage, and the control detection result is obtained.

[0120] In an optional implementation, the text recognition module 730 further includes:

[0121] The fusion unit is used to fuse the target region features and the control detection results based on the converter in the text recognition network to generate target fusion features;

[0122] Accordingly, the recognition unit is specifically used to: based on the text recognizer, perform text content recognition on the target fusion features and the control positions contained in the control detection results to obtain the content recognition result;

[0123] Accordingly, the first classification unit is specifically used to: classify the target fusion features into text languages ​​based on the language recognizer, and obtain the language recognition result;

[0124] Accordingly, the second classification unit is specifically used to: classify the text direction based on the target fusion features using the direction recognizer, and obtain the direction recognition result.

[0125] In one optional implementation, the fusion unit is specifically used to: based on the converter, sequentially perform encoding processing, upsampling processing, feature activation processing, and fusion processing on the target region features and the control detection results to obtain the target fused features.

[0126] In addition to the methods and apparatus described above, this disclosure also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the scene image testing method of this disclosure.

[0127] This disclosure also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the scene image testing method of this disclosure.

[0128] In addition, this disclosure also provides a scene image testing device, see [link to relevant documentation]. Figure 8 As shown, the scene image testing device may include:

[0129] The scene image testing device includes a processor 801, a memory 802, an input device 803, and an output device 804. The number of processors 801 in the device can be one or more. Figure 8 Taking a processor as an example. In some embodiments of this disclosure, the processor 801, memory 802, input device 803, and output device 804 can be connected via a bus or other means, wherein, Figure 8 Taking the example of a connection between China and Israel via a bus.

[0130] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing of the scene image testing equipment by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc. In addition, the memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The input device 803 can be used to receive input digital or character information, and generate signal inputs related to user settings and function control of the scene image testing equipment.

[0131] Specifically in this embodiment, the processor 801 will load the executable files corresponding to the processes of one or more applications into the memory 802 according to the following instructions, and the processor 801 will run the applications stored in the memory 802 to realize the various functions of the above-mentioned scene image testing device.

[0132] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0133] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for testing scene images, characterized in that, The method includes: Obtain the image of the scene to be tested; Using the scene classifier in the pre-trained test model, scene detection is performed on the scene image to be tested to obtain the scene detection result corresponding to the scene image to be tested; Using a pre-trained text recognition network in a test model, text recognition is performed on the test scene image to obtain the text recognition result corresponding to the test scene image. The text recognition includes at least one or more combinations of control text detection, text content recognition, text language classification, and text direction classification. Using a pre-trained text recognition network in a test model, control text detection is performed on the test scene image to obtain the text recognition result corresponding to the test scene image, including: Starting from the first detection stage, for the current detection stage, the control detector in the text recognition network is used to detect control text based on the target region features corresponding to the previous detection stage and the confidence level predicted by the previous detection stage, so as to obtain the confidence level corresponding to the current detection stage. For the next detection stage, the control detector is used to continue detecting control text based on the target region features and the confidence level corresponding to the current detection stage, until the next detection stage is the last detection stage, and the control detection result is obtained. Here, the target region features refer to the local features in the scene image to be tested.

2. The method according to claim 1, characterized in that, The test model further includes a backbone network; before using the scene classifier in the pre-trained test model to perform scene detection on the scene image to be tested and obtain the scene detection result corresponding to the scene image to be tested, the method further includes: Based on the backbone network in the test model, the image of the scene to be tested is downsampled to obtain the downsampled features corresponding to the image of the scene to be tested. The step of using a scene classifier in a pre-trained test model to perform scene detection on the scene image to be tested, and obtaining the scene detection result corresponding to the scene image to be tested, includes: Based on the scene classifier in the test model, scene detection is performed on the downsampled features to obtain the scene detection result; The step of using a pre-trained text recognition network in a test model to perform text recognition on the image to be tested, and obtaining the text recognition result corresponding to the image to be tested, includes: Based on the text recognition network in the test model, text recognition is performed on the downsampled features to obtain the text recognition result.

3. The method according to claim 2, characterized in that, The backbone network includes multiple attention modules built based on moving windows; the backbone network in the test model is used to downsample the image of the scene to be tested to obtain downsampled features corresponding to the image of the scene to be tested, including: By utilizing the multiple attention modules constructed based on moving windows, the test scene image is subjected to multiple feature interaction calculations between consecutive self-attention layers to generate the downsampling features.

4. The method according to claim 2, characterized in that, The scene classifier is a multilayer perceptron; the scene detection based on the downsampled features using the scene classifier in the test model, to obtain the scene detection result, includes: Using the multilayer perceptron, scene category prediction is performed on the downsampled features to obtain at least one scene category, and the at least one scene category is used as the scene detection result.

5. The method according to claim 2, characterized in that, Before performing text recognition on the downsampled features using the text recognition network based on the test model to obtain the text recognition result, the method further includes: The target region features are extracted from the downsampled features using the region extraction network in the test model. The text recognition network based on the test model performs text recognition on the downsampled features to obtain the text recognition result, including: Based on the text recognition network, text recognition is performed on the features of the target region to obtain the text recognition result.

6. The method according to claim 5, characterized in that, The process of performing text recognition on the target region features based on the text recognition network to obtain the text recognition result includes: Based on the control detector in the text recognition network, control text detection is performed on the features of the target region to obtain control detection results; Based on the text recognizer in the text recognition network, text content recognition is performed on the target region features and the control detection results to obtain the content recognition result; Based on the language recognizer in the text recognition network, the target region features and the control detection results are classified into text languages ​​to obtain the language recognition result. Based on the orientation recognizer in the text recognition network, the target region features and the control detection results are classified into text orientations to obtain orientation recognition results. The control detection result, the content recognition result, the language recognition result, and the direction recognition result constitute the text recognition result.

7. The method according to claim 6, characterized in that, Before performing text content recognition on the target region features and the control detection results based on the text recognizer in the text recognition network to obtain the content recognition result, the method further includes: Based on the converter in the text recognition network, the target region features and the control detection results are fused to generate target fusion features; The text recognizer based on the text recognition network performs text content recognition on the target region features and the control positions in the control detection results to obtain content recognition results, including: Based on the text recognition device, text content recognition is performed on the target fusion features and the control positions contained in the control detection results to obtain the content recognition result; The language identifier based on the text recognition network performs text language classification on the target region features and the control detection results to obtain language recognition results, including: Based on the language identifier, the target fusion features are classified into different languages ​​to obtain the language identification result. The orientation recognizer in the text recognition network performs text orientation classification on the target region features and the control detection results to obtain orientation recognition results, including: Based on the orientation recognizer, the target fusion features are classified into text orientations to obtain the orientation recognition result.

8. The method according to claim 7, characterized in that, The converter based on the text recognition network fuses the target region features and the control detection results to generate target fusion features, including: Based on the converter, the target region features and the control detection results are sequentially encoded, upsampled, activated, and fused to obtain the target fused features.

9. A scene image testing device, characterized in that, The device includes: The acquisition module is used to acquire images of the scene to be tested. The scene detection module is used to perform scene detection on the scene image to be tested using the scene classifier in the pre-trained test model, and obtain the scene detection result corresponding to the scene image to be tested. The text recognition module is used to perform text recognition on the test scene image using a text recognition network in a pre-trained test model, and obtain the text recognition result corresponding to the test scene image. The text recognition includes at least one or more combinations of control text detection, text content recognition, text language classification, and text direction classification. The text recognition module includes a detection unit; The detection unit is used to detect control text by using the control detector in the text recognition network to detect the target region features and the confidence level predicted in the previous detection stage, starting from the first detection stage, for the current detection stage, so as to obtain the confidence level corresponding to the current detection stage. For the next detection stage, the control detector is used to continue detecting control text based on the target region features and the confidence level corresponding to the current detection stage, until the next detection stage is the last detection stage, and the control detection result is obtained. Here, the target region features refer to the local features in the scene image to be tested.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method as described in any one of claims 1-8.

11. A device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method as described in any one of claims 1-8.

Images