Cloud desktop character scene coding method and device, equipment, medium and product

Abstract

The invention discloses a cloud desktop character scene coding method and device, equipment, a medium and a product, and the method comprises the steps: carrying out the character region detection of a current to-be-processed character scene image through a target connection text proposal network CTPN model, and obtaining a plurality of character blocks and non-character blocks; respectively coding the character blocks and the non-character blocks to obtain coded data of the character scene image; wherein the target CTPN model takes a lightweight neural network as a spatial feature extraction network, and takes a Transform encoder as a sequence feature extraction network. According to the method, the detection efficiency and the detection precision of the character area in the character scene encoding process are remarkably improved, the problem that character area detection is omitted or the character block contains excessive non-character parts is effectively avoided, and the encoding requirement of the cloud desktop character scene can be met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image coding, in particular to a cloud desktop text scene coding method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] In the coding mode of the text scene, the text part and the non-text part are usually divided according to the color and texture features of the text scene image, such as color histogram, color moment, etc., and then the text part is losslessly coded using Huffman coding or arithmetic coding.

[0003] In the coding process of the text scene in the prior art, the text scene image is usually divided into blocks with a fixed size, which easily destroys the connectivity of the original text area, causes the detection of the text part to be missed or the text block to contain too much non-text part, and results in low detection efficiency and detection accuracy of the text part. In the cloud desktop scene, complex text scenes or multiple fonts are usually involved, which has high requirements for the detection efficiency and detection accuracy of the text part. Therefore, the prior art is difficult to meet the coding requirements of the cloud desktop text scene. SUMMARY

[0004] The present application provides a cloud desktop text scene coding method, device, equipment, medium and product to solve the technical problem that the detection efficiency and detection accuracy of the text part are low in the coding process of the text scene in the prior art, thereby being difficult to meet the coding requirements of the cloud desktop text scene.

[0005] To solve the above technical problem, the first aspect of the embodiment of the present application provides a cloud desktop text scene coding method, comprising: detecting the text area of the current text scene image to be processed by using a target connection text proposal network (CTPN) model to obtain a plurality of text blocks and non-text blocks; coding each of the text blocks and the non-text blocks to obtain coding data of the text scene image; The target CTPN model uses a lightweight neural network as a spatial feature extraction network and a Transformer encoder as a sequence feature extraction network.

[0006] As a preferred scheme, the text area of the text scene image is detected by using the target CTPN model to obtain a plurality of text blocks and non-text blocks, which specifically comprises: The text scene image is sequentially subjected to high-level feature extraction and spatial feature extraction by the lightweight neural network to obtain a high-level feature map and a first feature map with spatial feature information, respectively; perform sequence feature extraction on the high-level feature map through the Transformer encoder to obtain a second feature map with sequence feature information; fuse the first feature map and the second feature map to obtain a fusion feature map; perform text region detection on the fusion feature map based on a preset plurality of prior boxes with the same width and different heights corresponding to each feature point in the fusion feature map to obtain a plurality of the text blocks and the non-text blocks; wherein the feature points are determined based on feature sizes of the fusion feature map by splitting the fusion feature map.

[0007] As a preferred solution, the high-level feature map and the first feature map with spatial feature information are obtained by sequentially performing high-level feature extraction and spatial feature extraction on the text scene image through the lightweight neural network, specifically including: perform low-level feature extraction on the text scene image using a convolutional layer to obtain a low-level feature map; perform high-level feature extraction on the low-level feature map using a plurality of reverse residual blocks to obtain the high-level feature map; perform spatial feature extraction on the high-level feature map using a sliding window with a preset size to obtain the first feature map.

[0008] As a preferred solution, the second feature map with sequence feature information is obtained by performing sequence feature extraction on the high-level feature map through the Transformer encoder, specifically including: obtain a plurality of feature sequences in the high-level feature map based on each row of feature vectors in the high-level feature map after dimension deformation processing; add a position embedding vector to each of the feature sequences to obtain an input feature sequence with position information; perform sequence feature extraction on each of the input feature sequences using a plurality of Transformer basic blocks to obtain the second feature map; wherein the Transformer basic block includes a multi-head attention layer and a positional feedforward network.

[0009] As a preferred solution, the plurality of the text blocks and the non-text blocks are obtained by performing text region detection on the fusion feature map based on a preset plurality of prior boxes with the same width and different heights corresponding to each feature point in the fusion feature map, specifically including: perform position regression and foreground or background classification on each of the prior boxes through an RPN layer in the target CTPN model based on a plurality of the prior boxes corresponding to each of the feature points to obtain a position offset and a confidence score of the prior box; adjust each of the prior boxes based on the position offset, to obtain an adjusted prior box; generate a plurality of text regions based on each of the adjusted prior boxes and the confidence score, and determine a plurality of the text blocks and the non-text block based on the text regions.

[0010] As a preferred solution, the method specifically generates a plurality of text regions based on each of the adjusted prior boxes and the confidence score, and specifically includes: perform non-maximum suppression on each of the adjusted prior boxes, to obtain a plurality of candidate prior boxes; filter each of the candidate prior boxes based on the confidence score, using a preset confidence score threshold, to obtain a plurality of candidate text boxes; merge a plurality of the candidate text boxes based on position information of each of the candidate text boxes and the confidence score, to generate a plurality of the text regions.

[0011] As a preferred solution, the method specifically encodes each of the text blocks through the following steps: perform quadtree partitioning on the text block based on color information of each pixel in the text block, to obtain a plurality of leaf nodes; wherein the color information of each pixel included in the leaf node is the same or the size of the pixel region corresponding to the leaf node is a preset minimum partitioning size; generate run block information corresponding to each of the leaf nodes using a two-dimensional run encoding method based on the position, size and color information of the pixel region corresponding to each of the leaf nodes; encode each of the text blocks based on each of the run block information, to obtain encoding data of each of the text blocks.

[0012] As a preferred solution, the method specifically encodes each of the non-text blocks through the following steps: encode each of the non-text blocks based on an H.264 encoding format, to obtain encoding data of each of the non-text blocks.

[0013] The second aspect of the embodiment of the application provides a cloud desktop text scene encoding device, which comprises: a text region detection module configured to perform text region detection on a current text scene image to be processed using a target connection text proposal network (CTPN) model, to obtain a plurality of text blocks and non-text blocks; an encoding module configured to encode each of the text blocks and the non-text blocks respectively, to obtain encoding data of the text scene image; The target CTPN model takes a lightweight neural network as a spatial feature extraction network and a Transformer encoder as a sequence feature extraction network.

[0014] The third aspect of the embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the cloud desktop text scene encoding method of any one of the first aspect when executing the computer program.

[0015] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, including a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the cloud desktop text scene encoding method of any one of the first aspect when the computer program is running.

[0016] The fifth aspect of the embodiment of the present application provides a computer program product, including computer programs / instructions, and the computer programs / instructions implement the steps of the cloud desktop text scene encoding method of any one of the first aspect when executed by a processor.

[0017] Compared with the prior art, the embodiment of the present application has the beneficial effects that by using the target CTPN model to extract the spatial features and sequence features of the text scene image and perform text region detection, compared with the fixed size text scene image division method, the detection efficiency and detection accuracy of the text region are significantly improved, the problem of missing text region detection or containing too many non-text parts in the text block is effectively avoided, and the coding demand of the cloud desktop text scene can be met. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart of the cloud desktop text scene encoding method in the embodiment of the present application; Figure 2 is an architecture diagram of the Transformer encoder in the embodiment of the present application; Figure 3 is a structure diagram of the cloud desktop text scene encoding device in the embodiment of the present application; Figure 4 is a structure diagram of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0019] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0020] Referring to Figure 1 The first aspect of the embodiments of the present application provides a cloud desktop text scene coding method, comprising the following steps S1 and S2: Step S1: using a target connection text proposal network (CTPN) model to detect the text region of a current text scene image to be processed, to obtain a plurality of text blocks and non-text blocks; Step S2: respectively coding each of the text blocks and the non-text blocks to obtain coding data of the text scene image; The target CTPN model uses a lightweight neural network as a spatial feature extraction network, and uses a Transformer encoder as a sequence feature extraction network.

[0021] It is worth noting that the traditional CTPN model (Connectionist Text Proposal Network) is a deep learning model specially used for scene text detection, and its core idea is to regard text detection as a sequence labeling problem, which combines convolutional neural networks and recurrent neural networks, especially bidirectional long short-term memory networks, to improve the accuracy and robustness of text detection. However, the traditional CTPN model has high complexity and consumes a lot of computing resources, which may cause delays and congestion problems for real-time transmission in the cloud desktop scenario. Therefore, in order to adapt the CTPN model to the real-time transmission requirements in the cloud desktop scenario, the embodiment replaces the original convolutional neural network in the CTPN model with a lightweight neural network to reduce the amount of calculation and the number of parameters. It can be understood that the lightweight neural network, also known as the micro neural network, is a neural network model with few parameters and low computational cost, which includes MobileNet, ShuffleNet, SqueezeNet, etc. This embodiment will not be described in more detail, and preferably, the target CTPN model in this embodiment uses MobileNet V2 as a spatial feature extraction network. MobileNet V2 uses deep separable convolution to build a lightweight deep neural network, significantly reducing model size and computational complexity, while it uses an inverted residual structure as its core structure, which helps to reduce model size while maintaining performance. In addition, the bidirectional long short-term memory network used in the traditional CTPN model does not have parallel processing capability, which will reduce the running efficiency of the network and increase the detection time of the text region. In order to further adapt the CTPN model to the real-time transmission requirements in the cloud desktop scenario, the embodiment uses a Transformer encoder with parallel processing capability as the sequence feature extraction network of the target CTPN model to improve the efficiency of text region detection while maintaining the accuracy of text region detection.

[0022] In this embodiment, by using the target CTPN model to extract the spatial features and sequence features of the current to-be-processed text scene image, it can provide the basis for "accurate positioning" and "local differentiation" of the text region, and provide support for "continuous integration" and "global correlation" of the text region, realize high-precision detection of the text region, and finally divide the current to-be-processed text scene image into a plurality of text blocks and non-text blocks. Then each text block and non-text block is encoded to obtain the encoding data of the text scene image. It can be understood that the encoding data of the text scene image includes the encoding data of the text block and the encoding data of the non-text block.

[0023] The cloud desktop text scene coding method provided by the embodiment of the application can extract spatial features and sequence features of a text scene image by using a target CTPN model for text region detection, significantly improve the detection efficiency and detection accuracy of the text region compared with a fixed size text scene image division method, effectively avoid the problems of text region detection omission or too many non-text parts in the text block, and meet the coding needs of the cloud desktop text scene.

[0024] As a preferred solution, the text region detection of the text scene image by using the target connection text proposal network (CTPN) model obtains a plurality of text blocks and non-text blocks, and specifically includes the following steps: The light-weight neural network is used to sequentially perform high-level feature extraction and spatial feature extraction on the text scene image to obtain a high-level feature map and a first feature map with spatial feature information. The sequence feature extraction of the high-level feature map is performed by the Transformer encoder to obtain a second feature map with sequence feature information. The first feature map and the second feature map are fused to obtain a fused feature map. The text region detection of the fused feature map is performed based on a plurality of preset prior boxes with the same width and different heights corresponding to each feature point in the fused feature map to obtain a plurality of text blocks and non-text blocks; wherein the feature points are determined by splitting the fused feature map based on the feature size of the fused feature map.

[0025] Specifically, for the current text scene image to be processed, the light-weight neural network in the target CTPN model is first used to sequentially perform high-level feature extraction and spatial feature extraction on the text scene image to obtain a high-level feature map and a first feature map with spatial feature information, which can capture the local spatial attributes of the text in the text scene image, such as the edges, thickness, and corners of strokes, the outline and color distribution of the text, and the like, thereby helping to distinguish the text from the background during subsequent text region detection.

[0026] Further, since the spatial feature and the sequence feature are both obtained based on further extraction of the high-level feature map, the extraction of the spatial feature and the sequence feature in the embodiment can be carried out simultaneously after the high-level feature map is obtained. Specifically, the high-level feature map is subjected to sequence feature extraction by using a Transformer encoder, so as to obtain a second feature map with sequence feature information. It is worth noting that in the cloud desktop text scenario, there may be complex situations such as “multi-row arrangement”, “chaotic background” and “various fonts”. The sequence feature information can improve the robustness of text region detection through global vision, and can capture the context association between texts in the text scene image, so as to ensure the text continuity and text integrity of the finally recognized text block.

[0027] Further, the embodiment sets a prior box based on the extracted spatial feature and sequence feature. First, the first feature map and the second feature map are fused, so that the fused feature map contains both the spatial feature and the sequence feature. Then, a plurality of prior boxes with the same width and different heights are set for each feature point in the fused feature map, for example, 10 prior boxes are set for each feature point, so that the various text regions with different heights arranged horizontally in the text scene image can be considered. Specifically, the embodiment uses each prior box to predict only the position in the vertical direction of the text, but does not predict the position in the horizontal direction. A plurality of prior boxes with fixed width are used in the horizontal position, and the height corresponding to each prior box is predicted accurately. Finally, the text blocks are merged into complete text blocks. After determining a plurality of text blocks, the remaining regions other than the text blocks are non-text blocks.

[0028] It is worth noting that the feature points in the embodiment are determined by splitting the fused feature map based on the feature size of the fused feature map. Assuming that the feature size of the fused feature map is N x M, where N represents the height of the feature map, i.e. the number of pixel rows in the vertical direction, and M represents the width of the feature map, i.e. the number of pixel columns in the horizontal direction, the fused feature map is divided into N x M segments, and each segment has a feature point.

[0029] As a preferred scheme, the high-level feature extraction and the spatial feature extraction on the text scene image by the lightweight neural network are specifically as follows: a low-level feature map is obtained by using a convolutional layer to extract low-level features from the text scene image; the high-level feature map is obtained by using a plurality of reverse residual blocks to extract high-level features from the low-level feature map; the first feature map is obtained by using a sliding window with a preset size to extract spatial features from the high-level feature map.

[0030] Specifically, the lightweight neural network in the embodiment is a MobileNet V2 network. First, a character scene image with a pixel size of (batch, W, H) is input into a convolutional layer with a size of 3x3 for low-level feature extraction, where W represents the width of the original character scene image, i.e., the number of pixels in the horizontal direction, and H represents the height of the original character scene image, i.e., the number of pixels in the vertical direction. The low-level features are features directly extracted from the original character scene image, representing basic visual attributes such as color, texture, edge, and corner feature information, thereby obtaining a low-level feature map.

[0031] Further, the low-level feature map is input into a plurality of inverted residual blocks to extract high-level features. The high-level features are abstract information further extracted from the low-level features, which can extract semantic information of the character scene image, such as size, shape, and other feature information. Each inverted residual block in the MobileNet V2 network includes a 1x1 convolutional layer for dimensionality increase to ensure that sufficient information is extracted, and a depthwise convolution and a pointwise convolution for depthwise separable convolution. These inverted residual blocks can gradually extract high-level features of the character scene image while reducing the size of the low-level feature map, thereby obtaining a high-level feature map.

[0032] Further, the last feature layer of the inverted residual structure is taken as an effective feature layer, with a dimension of (batch, K, N, M), where batch is the batch size representing the number of samples currently being processed, K is the number of channels, and NxM is the feature size. Then, a sliding window with a preset size is used to extract spatial features from the high-level feature map output by the inverted residual structure, i.e., each point in the high-level feature map is combined with the features in the sliding window region, thereby obtaining a feature vector with a length of 3x3xK. Finally, a first feature map with a dimension of (batch, LxK, N, M) is output, where L is the size of the sliding window. For example, if the sliding window is 3x3, the final output is a first feature map with a dimension of (batch, 9xK, N, M), which enables the network to learn spatial features.

[0033] As a preferred solution, the sequence feature extraction on the high-level feature map by the Transformer encoder to obtain a second feature map with sequence feature information includes: Based on each row of feature vectors in the high-level feature map after dimension deformation processing, a plurality of feature sequences in the high-level feature map are obtained; A position embedding vector is added to each feature sequence to obtain an input feature sequence with position information; Utilize several Transformer basic blocks to perform sequence feature extraction on each of the input feature sequences to obtain the second feature map; wherein the Transformer basic block includes a multi-head attention layer and a position feedforward network.

[0034] Specifically, in order to meet the input data requirements of the Transformer encoder, this embodiment first needs to perform dimensionality deformation processing on the high-level feature map. For example, assuming that the dimension of the high-level feature map is (batch, K, N, M), its feature dimension can be converted into (batch×N, M, K) through the Reshape deformation process to obtain several feature sequences in the high-level feature map. Then, the sequence features are learned through the multi-head attention layer of the Transformer encoder, which can learn the contextual relationship between each segment in the sequence features of each row, thereby improving the accuracy of text area detection. In order to make up for the defect that the Transformer encoder itself is not good at processing the order of elements in the sequence, this embodiment adds a position embedding vector as a position code in front of each feature sequence, which is used to represent the absolute position of each element in the feature sequence, and each element corresponds to a unique integer.

[0035] Furthermore, several Transformer basic blocks are used to extract sequence features from each input feature sequence, where Figure 2 As shown, the basic block of Transformer includes a multi-head attention layer and a position feed-forward network. Figure 2 In the figure, “features Embeddings” means feature embedding, that is, inputting the feature sequence into the Transformer encoder; “PositionEmbedding” means position embedding, that is, adding a position embedding vector to the input feature sequence; “block” means the Transformer basic block, and N is the number of Transformer basic blocks; in the Transformer basic block, “Multi-Head Attention” means the multi-head attention layer; “Add&Normalize” means the addition and normalization layer; “Feed-Forward Network” means the position feedforward network; “Encoder Normalize” means the encoder normalization layer.

[0036] Specifically, the multi-head attention layer is the core of the Transformer basic block to extract sequence features, and its core role is to break the "local recursive dependence" of traditional RNN (Recurrent Neural Network) / LSTM (Long Short-Term Memory), directly establish a connection between any two elements in the sequence, and realize the fusion of global context information. The multi-head attention layer projects the input feature sequence into query vectors (Q), key vectors (K) and value vectors (V) respectively. The similarity between the query vector (Q) and the key vector (K), that is, the attention weight, measures the association degree of each element with all other elements. Then, based on the weight, the value vector (V) is weighted and summed to obtain the feature of a single element after the global context is fused. Among them, "multi-head" means that multiple sets of attention calculations are performed in parallel, each set capturing the context relationship in different dimensions of the sequence, so as to improve the efficiency of sequence feature extraction. The results of multiple attention calculations are spliced, and then integrated into a unified dimension feature through a linear transformation, and output to the next layer to complete the comprehensive capture of the global context information of the text sequence. In addition, the embodiment adds an addition and normalization layer after the multi-head attention layer, which is used to connect the input feature and the output feature of the current layer in residual, avoid gradient disappearance during deep network training, and protect the integrity of feature transmission, and then normalize the features connected in residual.

[0037] After obtaining the global context feature in the multi-head attention layer, the position feedforward network optimizes the feature of each sequence element independently to supplement the local detail information. Specifically, the position feedforward network first raises the feature dimension through linear transformation, introduces nonlinearity through ReLU and other activation functions, and then restores the dimension through linear transformation. The position feedforward network adjusts the local details of the global context feature output by the multi-head attention layer, avoids the loss of local details caused by global information learning, and finally outputs the feature sequence after the current Transformer basic block extracts the features. Since the embodiment includes multiple Transformer basic blocks, the output of the previous Transforme basic block is used as the input of the subsequent Transforme basic block, and the learning of sequence features is deepened layer by layer. After obtaining the feature sequence output by the last Transforme basic block, it is restored to the size of (batch, 256, N, M) through the Reshape operation, and then changed to a second feature map with a dimension of (batch, 512, N, M) through the full connection layer.

[0038] As a preferred solution, the fusion feature map is subjected to text region detection based on a plurality of preset prior boxes of the same width and different heights corresponding to each feature point in the fusion feature map, to obtain a plurality of text blocks and non-text blocks, specifically comprising: Based on the plurality of prior boxes corresponding to each feature point, the position regression and foreground or background classification of the prior boxes are respectively performed by an RPN layer in the target CTPN model, to obtain the position offset and confidence score of the prior boxes; Based on the position offset, the adjustment of each prior box is performed to obtain an adjusted prior box; Based on each adjusted prior box and the confidence score, a plurality of text regions are generated, and based on the text regions, a plurality of text blocks and non-text blocks are determined.

[0039] Specifically, the plurality of prior boxes corresponding to each feature point in the embodiment have the same width, which can be set as the ratio between the width of the original text scene image and the width of the fusion feature map, so as to ensure that the final text block can cover all positions of the original text scene image, and the height of the prior box varies from 11 pixels to 273 pixels, and optionally, each time the height of the previous prior box is divided by 0.7, to form 10 different height values, which can cover text targets of different heights.

[0040] To screen out the text blocks from the several prior boxes, the embodiment utilizes the RPN layer (Region Proposal Network) in the target CTPN model to respectively perform position regression and foreground or background classification on the prior boxes. It is worth noting that the RPN layer is divided into three prediction branches: the first prediction branch performs a 1x1x2k convolution operation, and the dimension of the finally output feature information is (batch, 2k, N, M), where k is the number of prior boxes. The first prediction branch is used to predict the offset of the center point y coordinate of the prior box and the offset of the prior box height h, so 2k represents the offset of the center point y coordinate of each prior box and the offset of the prior box height h. The second prediction branch performs a 1x1x2k convolution operation, and the dimension of the finally output feature information is (batch, 2k, N, M). The second prediction branch is used to predict the probability of the image in the prior box belonging to the foreground and the probability of belonging to the background, so 2k represents the probability of each prior box belonging to the foreground and the probability of belonging to the background, and finally forms a confidence score. The higher the confidence score is, the greater the probability of belonging to the foreground is, that is, the greater the probability of text is. The third prediction branch performs a 1x1xk convolution operation, and the dimension of the finally output feature information is (batch, k, N, M). The third prediction branch is used to predict the offset of the center point x coordinate of the prior box, so as to finely adjust each prior box based on the offset of the center point y coordinate, the offset of the prior box height h and the offset of the center point x coordinate of each prior box, so as to improve the accuracy of text positioning. Then, based on each adjusted prior box and the confidence score, several text regions are generated as text blocks, and the remaining regions are non-text blocks.

[0041] As a preferred solution, the generating of the several text regions based on each of the adjusted prior boxes and the confidence score specifically includes: performing non-maximum suppression on each of the adjusted prior boxes to obtain several candidate prior boxes; performing screening on each of the candidate prior boxes based on the confidence score and the candidate prior boxes by using a preset confidence score threshold to obtain several candidate text boxes; merging the several candidate text boxes according to the position information of each of the candidate text boxes and the confidence score to generate the several text regions.

[0042] Specifically, the embodiment first performs non-maximum suppression on each adjusted prior box to filter out redundant prior boxes and remove prior boxes with high overlap, thereby obtaining a plurality of candidate prior boxes. Then, among the plurality of candidate prior boxes, candidate prior boxes with a confidence score greater than a confidence score threshold are screened to select a plurality of candidate text boxes containing foreground images, i.e., text content. The remaining candidate text boxes are sorted according to the confidence scores, and optionally, a text proposal connector is used to merge adjacent candidate text boxes with high confidence scores based on the position information and confidence scores of the candidate text boxes, thereby generating a text region. Specifically, the merging process of the candidate text boxes in the embodiment includes forward finding and reverse finding. The forward finding is to find, along a horizontal positive direction, remaining candidate text boxes with a horizontal distance from the ith candidate text box less than a horizontal distance threshold (e.g., 50 pixels), and to select, from the remaining candidate text boxes, a jth candidate text box that has a vertical overlap with the ith candidate text box greater than an overlap threshold (e.g., 0.7) and has the highest confidence score. The reverse finding is to find, along a horizontal negative direction, remaining candidate text boxes with a horizontal distance from the jth candidate text box less than the horizontal distance threshold (e.g., 50 pixels), and to select, from the remaining candidate text boxes, a kth candidate text box that has a vertical overlap with the jth candidate text box greater than the overlap threshold (e.g., 0.7) and has the highest confidence score. If the confidence score of the ith candidate text box is greater than or equal to the confidence score of the kth candidate text box, it indicates that the ith candidate text box to the jth candidate text box is a longest connected text block. If the confidence score of the ith candidate text box is less than the confidence score of the kth candidate text box, it indicates that the ith candidate text box to the jth candidate text box is not a longest connected text block, and is contained in a text block corresponding to the kth candidate text box to the jth candidate text box.

[0043] As a preferred solution, the method encodes each of the text blocks by the following steps: According to the color information of each pixel in the text block, the text block is quadtree partitioned to obtain a plurality of leaf nodes; wherein the color information of each pixel contained in the leaf node is the same or the size of the pixel region corresponding to the leaf node is a preset minimum partition size; Based on the position, size and color information of the pixel region corresponding to each of the leaf nodes, a two-dimensional run-length encoding method is used to generate run-length block information corresponding to each of the leaf nodes; Based on each of the run-length block information, each of the text blocks is encoded to obtain encoding data of each of the text blocks.

[0044] It is worth mentioning that the text block has simpler image information than the non-text block, and thus is suitable for lossless coding to achieve high-precision compression. Standard run-length coding usually has good compression effect on images with a single color background or simple repeating patterns, but most of the text images in the text scene have the characteristics of simple text color and complex background color, and thus the run-length coding needs to be adjusted. Two-dimensional run-length coding scans the image by a scanning method such as line-by-line scanning and records the run-length of the continuous pixels of the same color to achieve data compression. It is very sensitive to color changes in the image. When the background color is complex, the boundaries between different colors will form multiple small block continuous pixel regions. Two-dimensional run-length coding can accurately record the position and size of these regions, so as to accurately restore the color boundaries during decoding. On the other hand, two-dimensional run-length coding can achieve higher compression ratio by combining data structures such as quadtree. Therefore, the embodiment adopts two-dimensional run-length coding for lossless coding operation of the text block.

[0045] Specifically, the embodiment regards the entire text block as a root node of a quadtree, and then judges whether each pixel in the current text block region has the same color information, such as whether it has the same gray value. If the current pixel region has the same color information, it is marked as a leaf node of the quadtree. If the current pixel region has different color information, it is divided into sub-regions by four equal parts, and the quadtree splitting process is recursively performed on each sub-region until all the pixels in the pixel region have the same color information, or the size of the pixel region reaches a preset minimum splitting size, thereby obtaining a plurality of leaf nodes.

[0046] Further, based on the position of the pixel region corresponding to each leaf node, such as the coordinates of the starting pixel in the pixel region, the left-top pixel in the pixel region can be taken as the starting pixel, and the size and color information of the pixel region can be used to generate the run-length block information corresponding to each leaf node, thereby realizing the coding of the text block and forming the coding data of the text block.

[0047] As a preferred solution, the method specifically codes each non-text block by the following steps: Each non-text block is coded based on the H.264 coding format to obtain the coding data of each non-text block.

[0048] Specifically, since the non-text block has more complex and rich image information, and the coding method based on the H.264 coding format has excellent compression performance, it can significantly reduce the data amount while maintaining the video quality, and is very suitable for processing complex content such as video and image in the cloud desktop scene.

[0049] Please refer to Figure 3In a second aspect, the embodiment of the present application provides a cloud desktop text scene coding device 100, comprising: A text region detection module 11 is configured to detect text regions of a current text scene image to be processed by using a target connection text proposal network (CTPN) model, and obtain a plurality of text blocks and non-text blocks. An encoding module 12 is configured to encode each of the text blocks and the non-text blocks to obtain coding data of the text scene image. The target CTPN model uses a lightweight neural network as a spatial feature extraction network and uses a Transformer encoder as a sequence feature extraction network.

[0050] As a preferred solution, the text region detection module 11 is configured to detect text regions of a text scene image by using a target connection text proposal network (CTPN) model, and obtain a plurality of text blocks and non-text blocks, and specifically includes the following steps. The lightweight neural network is configured to sequentially perform high-level feature extraction and spatial feature extraction on the text scene image to obtain a high-level feature map and a first feature map with spatial feature information. The Transformer encoder is configured to perform sequence feature extraction on the high-level feature map to obtain a second feature map with sequence feature information. The first feature map and the second feature map are fused to obtain a fused feature map. The fused feature map is detected for text regions based on a plurality of prior boxes with the same width and different heights corresponding to each feature point in the fused feature map, to obtain a plurality of text blocks and non-text blocks, wherein the feature point is determined by splitting the fused feature map based on the feature size of the fused feature map.

[0051] As a preferred solution, the text region detection module 11 is configured to sequentially perform high-level feature extraction and spatial feature extraction on the text scene image by using the lightweight neural network to obtain a high-level feature map and a first feature map with spatial feature information, and specifically includes the following steps. The convolutional layer is configured to perform low-level feature extraction on the text scene image to obtain a low-level feature map. The plurality of reverse residual blocks are configured to perform high-level feature extraction on the low-level feature map to obtain the high-level feature map. The sliding window with a preset size is configured to perform spatial feature extraction on the high-level feature map to obtain the first feature map.

[0052] As a preferred solution, the character region detection module 11 is configured to perform sequence feature extraction on the high-level feature map by the Transformer encoder to obtain a second feature map with sequence feature information, specifically including: Based on each row of feature vectors in the high-level feature map after the dimension deformation processing, a plurality of feature sequences in the high-level feature map are obtained. A position embedding vector is added to each of the feature sequences to obtain an input feature sequence with position information. A plurality of Transformer basic blocks are used to perform sequence feature extraction on each of the input feature sequences to obtain the second feature map; wherein the Transformer basic block includes a multi-head attention layer and a position feedforward network.

[0053] As a preferred solution, the character region detection module 11 is configured to perform character region detection on the fusion feature map based on a plurality of preset prior boxes with the same width and different heights corresponding to each feature point in the fusion feature map to obtain a plurality of character blocks and non-character blocks, specifically including: Based on a plurality of prior boxes corresponding to each feature point, the RPN layer in the target CTPN model is used to perform position regression and foreground or background classification on the prior boxes respectively to obtain a position offset and a confidence score of the prior boxes; Based on the position offset, each of the prior boxes is adjusted to obtain an adjusted prior box; Based on each of the adjusted prior boxes and the confidence score, a plurality of character regions are generated, and a plurality of character blocks and non-character blocks are determined based on the character regions.

[0054] As a preferred solution, the character region detection module 11 is configured to generate a plurality of character regions based on each of the adjusted prior boxes and the confidence score, specifically including: Each of the adjusted prior boxes is subjected to non-maximum suppression to obtain a plurality of candidate prior boxes; Based on each of the candidate prior boxes and the confidence score, each of the candidate prior boxes is screened using a preset confidence score threshold to obtain a plurality of candidate text boxes; According to the position information and the confidence score of each of the candidate text boxes, a plurality of the candidate text boxes are merged to generate a plurality of the character regions.

[0055] As a preferred solution, the encoding module 12 specifically encodes each of the character blocks by the following steps: According to color information of each pixel in the text block, the text block is quadtree partitioned to obtain a plurality of leaf nodes; wherein the color information of each pixel contained in the leaf node is same or a size of a pixel region corresponding to the leaf node is preset minimum partition size; Based on a position, a size and the color information of the pixel region corresponding to each leaf node, a two-dimensional run-length coding method is used to generate run block information corresponding to each leaf node; Based on each run block information, each text block is encoded to obtain encoding data of each text block.

[0056] As a preferred solution, the encoding module 12 specifically encodes each non-text block by the following steps: Each non-text block is encoded based on H.264 encoding format to obtain encoding data of each non-text block.

[0057] The cloud desktop text scene coding apparatus 100 provided by the embodiment of the present application significantly improves the detection efficiency and detection accuracy of the text region by using the target CTPN model to extract the spatial features and sequence features of the text scene image and perform text region detection, compared with the fixed size text scene image division method, effectively avoids the problem of text region detection omission or containing too many non-text parts in the text block, and can meet the coding needs of the cloud desktop text scene.

[0058] Please refer to Figure 4 The third aspect of the embodiment of the present application provides an electronic device 200, which comprises a memory 22, a processor 21, and a computer program stored in the memory 22 and executable on the processor 21, and the processor 21 implements the cloud desktop text scene coding method of any one of the first aspect.

[0059] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device 200.

[0060] The electronic device 200 can include, but is not limited to, a processor 21, a memory 22. Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 200, and does not constitute a limitation on the electronic device 200, and can include more or fewer components than the diagram, or combine certain components, or different components, for example, the electronic device 200 can also include an input / output device, a network access device, a bus, etc.

[0061] The processor 21 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor 21 can also be any conventional processor 21, etc. The processor 21 is the control center of the electronic device 200, and connects all parts of the electronic device 200 through various interfaces and lines.

[0062] The memory 22 can be used to store computer programs and / or modules, and the processor 21 realizes various functions of the electronic device 200 by running or executing computer programs and / or modules stored in the memory 22, and calling data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0063] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, the computer readable storage medium includes a stored computer program, wherein when the computer program runs, the device where the computer readable storage medium is located executes the cloud desktop text scene encoding method of any one of the first aspect.

[0064] The fifth aspect of the embodiments of the present application provides a computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the cloud desktop text scene encoding method of any of the first aspect.

[0065] The modules / units integrated in the electronic device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0066] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A cloud desktop text scene encoding method, characterized in that, The application relates to a text region detection method and device. The method comprises the following steps: detecting a text region of a text scene image by using a target connection text proposal network (CTPN) model to obtain a plurality of text blocks and non-text blocks; encoding each of the text blocks and the non-text blocks to obtain encoding data of the text scene image; 2. The cloud desktop text scene encoding method of claim 1, wherein, wherein the target CTPN model uses a lightweight neural network as a spatial feature extraction network and uses a Transformer encoder as a sequence feature extraction network. The method for detecting a text region of a text scene image by using a target connection text proposal network (CTPN) model to obtain a plurality of text blocks and non-text blocks comprises the following steps: extracting high-level features and spatial features from the text scene image in sequence by using the lightweight neural network to obtain a high-level feature map and a first feature map with spatial feature information; extracting sequence features from the high-level feature map by using the Transformer encoder to obtain a second feature map with sequence feature information; fusing the first feature map and the second feature map to obtain a fused feature map; 3. The cloud desktop text scene encoding method of claim 2, wherein, detecting a text region of the fused feature map based on a plurality of preset prior boxes with the same width and different heights corresponding to each feature point in the fused feature map to obtain a plurality of text blocks and non-text blocks; wherein the feature points are determined by splitting the fused feature map based on the feature size of the fused feature map. The method for extracting high-level features and spatial features from the text scene image in sequence by using the lightweight neural network to obtain a high-level feature map and a first feature map with spatial feature information comprises the following steps: extracting low-level features from the text scene image by using a convolutional layer to obtain a low-level feature map; extracting high-level features from the low-level feature map by using a plurality of reverse residual blocks to obtain the high-level feature map; 4. The cloud desktop text scene encoding method of claim 2, wherein, extracting spatial features from the high-level feature map by using a sliding window with a preset size to obtain the first feature map. The method for extracting sequence features from the high-level feature map by using the Transformer encoder to obtain a second feature map with sequence feature information comprises the following steps: obtaining a plurality of feature sequences in the high-level feature map based on each row of feature vectors in the high-level feature map after dimension deformation processing; adding a position embedding vector to each of the feature sequences to obtain an input feature sequence with position information; 5. The cloud desktop text scene encoding method of claim 2, wherein, extracting sequence features from each of the input feature sequences by using a plurality of Transformer basic blocks to obtain the second feature map; wherein the Transformer basic block comprises a multi-head attention layer and a position feedforward network. The method for detecting a text region of the fused feature map based on a plurality of preset prior boxes with the same width and different heights corresponding to each feature point in the fused feature map to obtain a plurality of text blocks and non-text blocks comprises the following steps: Based on the several prior boxes corresponding to each of the feature points, the position regression and the foreground or background classification of the prior boxes are respectively performed by an RPN layer in the target CTPN model to obtain the position offset and the confidence score of the prior boxes; Based on the position offset, the prior boxes are adjusted to obtain adjusted prior boxes; Based on the adjusted prior boxes and the confidence score, a plurality of text regions are generated, and the text blocks and the non-text blocks are determined based on the text regions.

6. The cloud desktop text scene encoding method of claim 5, wherein, The method specifically generates the text regions based on the adjusted prior boxes and the confidence score, and includes the following steps: The non-maximum suppression is performed on the adjusted prior boxes to obtain a plurality of candidate prior boxes; Based on the candidate prior boxes and the confidence score, the candidate prior boxes are screened by using a preset confidence score threshold to obtain a plurality of candidate text boxes; According to the position information and the confidence score of the candidate text boxes, the candidate text boxes are merged to generate the text regions.

7. The cloud desktop text scene encoding method of claim 1, wherein, The method specifically encodes the text blocks by the following steps: According to the color information of each pixel in the text block, the text block is quadtree partitioned to obtain a plurality of leaf nodes; wherein the color information of each pixel included in the leaf node is the same or the size of the pixel region corresponding to the leaf node is a preset minimum partition size; Based on the position, size and color information of the pixel region corresponding to each leaf node, a two-dimensional run-length encoding method is used to generate run-length block information corresponding to each leaf node; Based on the run-length block information, the text blocks are encoded to obtain the encoding data of the text blocks.

8. The cloud desktop text scene encoding method of claim 1, wherein, The method specifically encodes the non-text blocks by the following steps: Based on the H.264 encoding format, the non-text blocks are encoded to obtain the encoding data of the non-text blocks.

9. A cloud desktop text scene encoding apparatus, characterized in that, It includes: a text region detection module configured to detect text regions in a current text scene image to be processed by using a target connection text proposal network (CTPN) model, and obtain a plurality of text blocks and non-text blocks; an encoding module configured to encode the text blocks and the non-text blocks respectively, and obtain encoding data of the text scene image. The target CTPN model uses a lightweight neural network as a spatial feature extraction network and a Transformer encoder as a sequence feature extraction network.

10. An electronic device, comprising: The computer readable storage medium includes a computer program stored therein, wherein when the computer program is running, the computer readable storage medium controls the device where the computer readable storage medium is located to perform the cloud desktop text scene encoding method of any one of claims 1 to 8.

11. A computer readable storage medium characterized by, The computer readable storage medium includes a computer program stored therein, wherein when the computer program is running, the computer readable storage medium controls the device where the computer readable storage medium is located to perform the cloud desktop text scene encoding method of any one of claims 1 to 8.

12. A computer program product, characterised in that, Computer program / instructions which, when executed by a processor, implement the steps of the cloud desktop text scene encoding method according to any one of claims 1 to 8.

Images