A cloud desktop text scene coding method, device, equipment, medium and product

Abstract

The application discloses a cloud desktop text scene coding method, device, equipment, medium and product. The method comprises the following steps: detecting a text region of a current text scene image to be processed by using a target connection text proposal network (CTPN) model, and obtaining a plurality of text blocks and non-text blocks; and respectively coding each text block and non-text block to obtain coding data of the text scene image. The target CTPN model uses a lightweight neural network as a spatial feature extraction network and uses a Transformer encoder as a sequence feature extraction network. The application significantly improves the detection efficiency and accuracy of the text region in the text scene coding process, effectively avoids the problems of missing text region detection or containing too many non-text parts in the text block, and meets the coding requirements of the cloud desktop text scene.

Description

Technical Field

[0001] This invention relates to the field of image encoding technology, and in particular to a cloud desktop text scene encoding method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] In text scene encoding methods, the text and non-text parts are usually divided based on the color and texture features of the text scene image, such as color histograms and color moments. Then, Huffman coding or arithmetic coding is used to perform lossless encoding on the text part.

[0003] In the encoding process of text scenes, existing technologies typically divide the text scene image into blocks of fixed size. This can easily disrupt the connectivity of the original text regions, causing text parts to be missed or containing too many non-text parts in the text blocks, resulting in low detection efficiency and accuracy for the text parts. However, cloud desktop scenarios often involve complex text scenes or multiple fonts, which places high demands on the detection efficiency and accuracy of the text parts. Therefore, existing technologies are difficult to meet the encoding requirements of cloud desktop text scenes. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and product for encoding text scenes on cloud desktops, in order to solve the technical problem that the detection efficiency and accuracy of text parts are low in the existing technology during the text scene encoding process, thus making it difficult to meet the encoding requirements of text scenes on cloud desktops.

[0005] To address the aforementioned technical problems, a first aspect of this invention provides a cloud desktop text scene encoding method, comprising:

[0006] The Target Connectivity Text Proposal Network (CTPN) model is used to detect text regions in the current text scene image to be processed, and obtain several text blocks and non-text blocks.

[0007] Each of the text blocks and the non-text blocks is encoded separately to obtain the encoded data of the text scene image;

[0008] The target CTPN model uses a lightweight neural network as the spatial feature extraction network and a Transformer encoder as the sequence feature extraction network.

[0009] As a preferred embodiment, the method of using the Target Connectivity Text Proposal Network (CTPN) model to detect text regions in a text scene image and obtain several text blocks and non-text blocks specifically includes:

[0010] The lightweight neural network is used to sequentially perform high-level feature extraction and spatial feature extraction on the text scene image to obtain a high-level feature map and a first feature map with spatial feature information, respectively.

[0011] The Transformer encoder is used to extract sequence features from the high-level feature map to obtain a second feature map with sequence feature information.

[0012] The first feature map and the second feature map are fused to obtain a fused feature map;

[0013] Based on a number of predefined prior boxes with the same width but different heights corresponding to each feature point in the fused feature map, text region detection is performed on the fused feature map to obtain a number of text blocks and non-text blocks; wherein, the feature points are determined by splitting the fused feature map based on the feature size of the fused feature map.

[0014] As a preferred embodiment, the step of sequentially performing high-level feature extraction and spatial feature extraction on the text scene image using the lightweight neural network to obtain a high-level feature map and a first feature map with spatial feature information, specifically includes:

[0015] Low-level feature extraction is performed on the text scene image using convolutional layers to obtain a low-level feature map;

[0016] The high-level feature map is obtained by using several inverted residual blocks to extract high-level features from the low-level feature map.

[0017] Spatial features are extracted from the high-level feature map using a sliding window of a preset size to obtain the first feature map.

[0018] As a preferred embodiment, the step of extracting sequence features from the high-level feature map using the Transformer encoder to obtain a second feature map with sequence feature information specifically includes:

[0019] Based on each row of feature vectors in the high-level feature map after dimensional deformation processing, several feature sequences in the high-level feature map are obtained.

[0020] Add a position embedding vector to each of the aforementioned feature sequences to obtain an input feature sequence with positional information;

[0021] The second feature map is obtained by extracting sequence features from each of the input feature sequences using several Transformer basic blocks; wherein, the Transformer basic blocks include a multi-head attention layer and a position feedforward network.

[0022] As a preferred embodiment, the step of performing text region detection on the fused feature map based on a number of preset prior boxes with the same width but different heights corresponding to each feature point in the fused feature map, to obtain a number of text blocks and non-text blocks, specifically includes:

[0023] Based on several prior boxes corresponding to each feature point, the RPN layer in the target CTPN model is used to perform position regression and foreground or background classification on the prior boxes to obtain the position offset and confidence score of the prior boxes.

[0024] The prior boxes are adjusted based on the position offset to obtain the adjusted prior boxes;

[0025] Based on the adjusted prior boxes and the confidence scores, several text regions are generated, and several text blocks and non-text blocks are determined based on the text regions.

[0026] As a preferred embodiment, the generation of several text regions based on the adjusted prior boxes and the confidence scores specifically includes:

[0027] Non-maximum suppression is applied to each of the adjusted prior boxes to obtain several candidate prior boxes;

[0028] Based on each of the candidate prior boxes and the confidence score, the candidate prior boxes are filtered using a preset confidence score threshold to obtain a number of candidate text boxes.

[0029] Based on the position information and confidence score of each candidate text box, several candidate text boxes are merged to generate several text regions.

[0030] As a preferred embodiment, the method specifically encodes each of the text blocks through the following steps:

[0031] Based on the color information of each pixel in the text block, the text block is divided into a quadtree to obtain several leaf nodes; wherein, the color information of each pixel contained in the leaf node is the same or the size of the pixel region corresponding to the leaf node is a preset minimum segmentation size;

[0032] Based on the position, size, and color information of the pixel region corresponding to each leaf node, run-length encoding is used to generate run-length block information corresponding to each leaf node.

[0033] Based on the run-length block information, each text block is encoded to obtain the encoded data of each text block.

[0034] As a preferred embodiment, the method specifically encodes each of the non-text blocks through the following steps:

[0035] Each of the non-text blocks is encoded using the H.264 encoding format to obtain the encoded data of each non-text block.

[0036] A second aspect of the present invention provides a cloud desktop text scene encoding device, comprising:

[0037] The text region detection module is used to detect text regions in the current text scene image to be processed using the Target Connected Text Proposal Network (CTPN) model, and obtain several text blocks and non-text blocks.

[0038] The encoding module is used to encode each of the text blocks and the non-text blocks respectively to obtain the encoded data of the text scene image;

[0039] The target CTPN model uses a lightweight neural network as the spatial feature extraction network and a Transformer encoder as the sequence feature extraction network.

[0040] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cloud desktop text scene encoding method described in any of the first aspects.

[0041] A fourth aspect of the present invention provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the cloud desktop text scene encoding method described in any of the first aspects.

[0042] A fifth aspect of the present invention provides a computer program product, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, it implements the steps of the cloud desktop text scene encoding method described in any of the first aspects.

[0043] Compared with the prior art, the beneficial effect of the embodiments of the present invention is that by using the target CTPN model to extract spatial and sequence features of text scene images and perform text region detection, compared with the text scene image division method according to fixed size, the detection efficiency and accuracy of text regions are significantly improved, effectively avoiding the problem of missing text regions or containing too many non-text parts in text blocks, and can meet the encoding requirements of cloud desktop text scenes. Attached Figure Description

[0044] Figure 1This is a flowchart illustrating the cloud desktop text scene encoding method in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the Transformer encoder architecture in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the structure of the cloud desktop text scene encoding device in an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the structure of the electronic device in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Please see Figure 1 The first aspect of this invention provides a cloud desktop text scene encoding method, including the following steps S1 and S2:

[0050] Step S1: Use the Target Connected Text Proposal Network (CTPN) model to detect text regions in the current text scene image to be processed, and obtain several text blocks and non-text blocks.

[0051] Step S2: Encode each of the text blocks and the non-text blocks respectively to obtain the encoded data of the text scene image;

[0052] The target CTPN model uses a lightweight neural network as the spatial feature extraction network and a Transformer encoder as the sequence feature extraction network.

[0053] It's worth noting that the traditional CTPN (Connectionist Text Proposal Network) model is a deep learning model specifically designed for scene-based text detection. Its core idea is to treat text detection as a sequence labeling problem, combining convolutional neural networks and recurrent neural networks, particularly bidirectional long short-term memory networks, to improve the accuracy and robustness of text detection. However, the traditional CTPN model is highly complex and consumes significant computational resources, potentially causing latency and congestion issues in real-time transmission scenarios like cloud desktops. Therefore, to adapt the CTPN model to the real-time transmission requirements of cloud desktops, this embodiment replaces the original convolutional neural network in the CTPN model with a lightweight neural network to reduce computational load and the number of parameters. It is understandable that lightweight neural networks, also known as micro-neural networks, are neural network models with few parameters and low computational cost. These include MobileNet, ShuffleNet, and SqueezeNet, which will not be elaborated upon further in this embodiment. Preferably, the target CTPN model in this embodiment uses MobileNet V2 as the spatial feature extraction network. MobileNet V2 uses depthwise separable convolutions to construct lightweight deep neural networks, significantly reducing model size and computational cost. Simultaneously, it employs an inverse residual structure as its core structure, which helps to reduce model size while maintaining performance. Furthermore, the bidirectional long short-term memory network used in traditional CTPN models lacks parallel processing capabilities, which reduces network efficiency and increases text region detection time. To further adapt the CTPN model to the real-time transmission requirements of cloud desktop scenarios, this embodiment uses a Transformer encoder with parallel processing capabilities as the sequence feature extraction network of the target CTPN model, thereby improving text region detection efficiency while maintaining text region detection accuracy.

[0054] In this embodiment, by using the target CTPN model to extract spatial and sequence features from the text scene image to be processed, a basis for the "precise positioning" and "local differentiation" of text regions can be provided, and support for the "continuous integration" and "global association" of text regions can be provided, thereby achieving high-precision detection of text regions. Finally, the text scene image to be processed is divided into several text blocks and non-text blocks, and then each text block and non-text block is encoded to obtain the encoded data of the text scene image. It can be understood that the encoded data of the text scene image includes the encoded data of text blocks and the encoded data of non-text blocks.

[0055] The cloud desktop text scene encoding method provided in this embodiment of the invention extracts spatial and sequence features from text scene images and performs text region detection using a target CTPN model. Compared with the text scene image segmentation method based on fixed size, it significantly improves the detection efficiency and accuracy of text regions, effectively avoids the problem of missing text regions or including too many non-text parts in text blocks, and can meet the encoding requirements of cloud desktop text scenes.

[0056] As a preferred embodiment, the method of using the Target Connectivity Text Proposal Network (CTPN) model to detect text regions in a text scene image and obtain several text blocks and non-text blocks specifically includes:

[0057] The lightweight neural network is used to sequentially perform high-level feature extraction and spatial feature extraction on the text scene image to obtain a high-level feature map and a first feature map with spatial feature information, respectively.

[0058] The Transformer encoder is used to extract sequence features from the high-level feature map to obtain a second feature map with sequence feature information.

[0059] The first feature map and the second feature map are fused to obtain a fused feature map;

[0060] Based on a number of predefined prior boxes with the same width but different heights corresponding to each feature point in the fused feature map, text region detection is performed on the fused feature map to obtain a number of text blocks and non-text blocks; wherein, the feature points are determined by splitting the fused feature map based on the feature size of the fused feature map.

[0061] Specifically, for the text scene image to be processed, this embodiment first uses the lightweight neural network in the target CTPN model to sequentially perform high-level feature extraction and spatial feature extraction on the text scene image, thereby obtaining a high-level feature map and a first feature map with spatial feature information. This spatial feature information can capture the local spatial attributes of the text in the text scene image, such as the edges, thickness, and corners of the strokes, the outline of the text, and the color distribution, which helps to distinguish the text from the background in subsequent text region detection.

[0062] Furthermore, since both spatial features and sequence features are obtained by further extraction based on high-level feature maps, the extraction of spatial features and sequence features can be performed in parallel after obtaining the high-level feature map in this embodiment. Specifically, a Transformer encoder is used to extract sequence features from the high-level feature map to obtain a second feature map with sequence feature information. It is worth noting that in cloud desktop text scenarios, text may have complex situations such as "multi-line arrangement", "cluttered background", and "various fonts". Sequence feature information can improve the robustness of text region detection through global vision. It can capture the contextual relationship between text in the text scene image and ensure the text continuity and text integrity of the finally identified text block.

[0063] Furthermore, this embodiment sets prior boxes based on the extracted spatial and sequence features. First, the first and second feature maps are fused, resulting in a fused feature map that simultaneously contains both spatial and sequence features. Then, several prior boxes with the same width but different heights are set for each feature point in the fused feature map; for example, 10 prior boxes are set for each feature point. This allows for the consideration of various text regions with different heights arranged horizontally in the text scene image. Specifically, this embodiment uses each prior box to predict only the vertical position of the text, not the horizontal position. For the horizontal position, only multiple prior boxes with fixed widths are needed to accurately predict the height corresponding to each prior box, and finally, they are merged into a complete text block. After determining several text blocks, the remaining areas outside the text blocks are considered non-text blocks.

[0064] It is worth noting that the feature points in this embodiment are determined by splitting the fused feature map based on the feature size of the fused feature map. Assuming the feature size of the fused feature map is N×M, where N represents the feature map height, i.e. the number of pixel rows in the vertical direction, and M represents the feature map width, i.e. the number of pixel columns in the horizontal direction, the fused feature map is divided into N×M segments, and each segment contains one feature point.

[0065] As a preferred embodiment, the step of sequentially performing high-level feature extraction and spatial feature extraction on the text scene image using the lightweight neural network to obtain a high-level feature map and a first feature map with spatial feature information, specifically includes:

[0066] Low-level feature extraction is performed on the text scene image using convolutional layers to obtain a low-level feature map;

[0067] The high-level feature map is obtained by using several inverted residual blocks to extract high-level features from the low-level feature map.

[0068] Spatial features are extracted from the high-level feature map using a sliding window of a preset size to obtain the first feature map.

[0069] Specifically, the lightweight neural network in this embodiment is the MobileNet V2 network. First, a text scene image with a pixel size of (batch, W, H) is input into a 3×3 convolutional layer for low-level feature extraction. Here, W refers to the width of the original text scene image, i.e., the number of pixels contained in the horizontal direction; H refers to the height of the original text scene image, i.e., the number of pixels contained in the vertical direction. Low-level features refer to features directly extracted from the original text scene image that represent basic visual attributes, such as color, texture, edges, and corners, thereby obtaining a low-level feature map.

[0070] Furthermore, the low-level feature maps are input into multiple inverted residual blocks to extract high-level features. High-level features are abstract information further extracted from low-level features, which can extract semantic information of text scene images, such as size and shape features. Each inverted residual block in the MobileNet V2 network contains an up-dimensional 1×1 convolutional layer to ensure that enough information is extracted, as well as a depthwise convolution and a pointwise convolution for depthwise separable convolution. By using these inverted residual blocks, the high-level features of the text scene image can be gradually extracted while reducing the size of the low-level feature maps to obtain high-level feature maps.

[0071] Furthermore, the last feature layer of the inverse residual structure is extracted as the effective feature layer, assuming a dimension of (batch, K, N, M), where batch is the batch size, representing the number of samples currently being processed; K is the number of channels; and N×M is the feature size. Then, using a sliding window of a preset size (e.g., a 3×3 sliding window), spatial features are extracted from the high-level feature map output by the inverse residual structure. Each point in the high-level feature map is combined with features within the sliding window region, resulting in a feature vector of length 3×3×K. The final output is a first feature map with dimensions (batch, L×K, N, M), where L is the size of the sliding window. Using a 3×3 sliding window as an example, the final output is a first feature map with dimensions (batch, 9×K, N, M), enabling the network to learn spatial features.

[0072] As a preferred embodiment, the step of extracting sequence features from the high-level feature map using the Transformer encoder to obtain a second feature map with sequence feature information specifically includes:

[0073] Based on each row of feature vectors in the high-level feature map after dimensional deformation processing, several feature sequences in the high-level feature map are obtained.

[0074] Add a position embedding vector to each of the aforementioned feature sequences to obtain an input feature sequence with positional information;

[0075] The second feature map is obtained by extracting sequence features from each of the input feature sequences using several Transformer basic blocks; wherein, the Transformer basic blocks include a multi-head attention layer and a position feedforward network.

[0076] Specifically, to meet the input data requirements of the Transformer encoder, this embodiment first needs to perform dimensional deformation processing on the high-level feature map. For example, assuming the dimensions of the high-level feature map are (batch, K, N, M), it can be transformed into (batch × N, M, K) through reshape deformation to obtain several feature sequences in the high-level feature map. Then, the sequence features are learned through the multi-head attention layer of the Transformer encoder, which can learn the contextual relationships between segments in each line of sequence features, thereby improving the accuracy of text region detection. To compensate for the Transformer encoder's inherent weakness in handling the order of elements in a sequence, this embodiment adds a position embedding vector as a positional encoding before each feature sequence to represent the absolute position of each element in the feature sequence. Each element corresponds to a unique integer.

[0077] Furthermore, several Transformer basic blocks are used to extract sequence features from each input feature sequence, such as... Figure 2 As shown, the basic Transformer block includes a multi-head attention layer and a position feedforward network. Figure 2 In this code, "features Embeddings" refers to feature embedding, which means inputting the feature sequence into the Transformer encoder; "PositionEmbedding" refers to position embedding, which means adding position embedding vectors to the input feature sequence; "block" refers to a Transformer basic block, and N is the number of Transformer basic blocks; within the Transformer basic block, "Multi-Head Attention" refers to a multi-head attention layer; "Add&Normalize" refers to an addition and normalization layer; "Feed-Forward Network" refers to a position feed-forward network; and "Encoder Normalize" refers to an encoder normalization layer.

[0078] Specifically, the multi-head attention layer is the core of the Transformer basic block for extracting sequence features. Its core function is to break the "local recursive dependency" of traditional RNN (Recurrent Neural Network) / LSTM (Long Short-Term Memory) networks, directly establishing a relationship between any two elements in the sequence, and achieving the fusion of global contextual information. The multi-head attention layer projects the input feature sequence into a query vector (Q), a key vector (K), and a value vector (V), respectively. By calculating the similarity between the query vector (Q) and the key vector (K), i.e., the attention weight, measures the degree of association between each element and all other elements; then, based on this weight, the value vector (V) is weighted and summed to obtain the feature of a single element after fusing the global context. Here, "multi-head" means that multiple sets of the above attention calculations are executed in parallel, each set capturing different dimensions of contextual relationships in the sequence, thereby improving the efficiency of sequence feature extraction. The results of multiple sets of attention calculations are concatenated and then integrated into a feature of a unified dimension through a linear transformation, which is then output to the next layer, completing the comprehensive capture of the global contextual information of the text sequence. In addition, this embodiment adds an addition and normalization layer after the multi-head attention layer. This layer is used to perform residual connection between the input features and output features of the current layer to avoid gradient vanishing during deep network training and ensure the integrity of feature propagation. Then, the features after residual connection are normalized.

[0079] After acquiring global context features in the multi-head attention layer, the features of each sequence element are independently optimized through a positional feedforward network to supplement local detail information. Specifically, the positional feedforward network first increases the feature dimension through linear transformation, introduces non-linearity through activation functions such as ReLU, and then restores the dimension through linear transformation. This positional feedforward network performs local detail adjustments on the global context features output by the multi-head attention layer, avoiding the loss of local details caused by global information learning, and finally outputs the feature sequence after the current Transformer basic block extracts features. Since this embodiment contains multiple Transformer basic blocks, the output of the previous Transformer basic block is used as the input of the next Transformer basic block, deepening the learning of sequence features layer by layer. After obtaining the feature sequence output by the last Transformer basic block, it is restored to a size of (batch, 256, N, M) through a reshape operation, and becomes a second feature map with a dimension of (batch, 512, N, M) after passing through a fully connected layer.

[0080] As a preferred embodiment, the step of performing text region detection on the fused feature map based on a number of preset prior boxes with the same width but different heights corresponding to each feature point in the fused feature map, to obtain a number of text blocks and non-text blocks, specifically includes:

[0081] Based on several prior boxes corresponding to each feature point, the RPN layer in the target CTPN model is used to perform position regression and foreground or background classification on the prior boxes to obtain the position offset and confidence score of the prior boxes.

[0082] The prior boxes are adjusted based on the position offset to obtain the adjusted prior boxes;

[0083] Based on the adjusted prior boxes and the confidence scores, several text regions are generated, and several text blocks and non-text blocks are determined based on the text regions.

[0084] Specifically, in this embodiment, the prior boxes corresponding to each feature point have the same width. For example, it can be set to the ratio between the width of the original text scene image and the width of the fused feature map, so as to ensure that the final text block can cover all positions of the original text scene image. The height of the prior boxes varies from 11 pixels to 273 pixels. Optionally, each change is based on the height of the previous prior box divided by 0.7, thereby forming 10 different height values, which can cover text targets of different heights.

[0085] To select text blocks from several prior boxes, this embodiment utilizes the RPN (Region Proposal Network) layer in the target CTPN model to perform position regression and foreground or background classification on the prior boxes. It is worth noting that the RPN layer is divided into three prediction branches: the first prediction branch performs a 1×1×2k convolution operation, and the final output feature information has dimensions of (batch, 2k, N, M), where k is the number of prior boxes. This prediction branch is used to predict the offset of the y-coordinate of the center point of the prior box and the offset of the height h of the prior box. Therefore, 2k represents the offset of the y-coordinate of the center point of each prior box and the offset of the height h of the prior box. The second prediction branch uses a 1×1×2k convolution operation, resulting in a feature information dimension of (batch, 2k, N, M). This branch predicts the probability that the image within the prior bounding box belongs to the foreground and the background. Therefore, 2k represents the probability of each prior bounding box belonging to the foreground and the background, ultimately forming a confidence score. A higher confidence score indicates a greater probability of belonging to the foreground, i.e., a greater probability of being text. The third prediction branch uses a 1×1×k convolution operation, resulting in a feature information dimension of (batch, k, N, M). This branch predicts the offset of the x-coordinate of the center point of the prior bounding box. Based on the offset of the y-coordinate of the center point, the offset of the height h of the prior bounding box, and the offset of the x-coordinate of the center point, the prior bounding boxes are finely adjusted to improve the accuracy of text localization. Then, based on the adjusted prior bounding boxes and the confidence score, several text regions are generated as text blocks, and the remaining regions are non-text blocks.

[0086] As a preferred embodiment, the generation of several text regions based on the adjusted prior boxes and the confidence scores specifically includes:

[0087] Non-maximum suppression is applied to each of the adjusted prior boxes to obtain several candidate prior boxes;

[0088] Based on each of the candidate prior boxes and the confidence score, the candidate prior boxes are filtered using a preset confidence score threshold to obtain a number of candidate text boxes.

[0089] Based on the position information and confidence score of each candidate text box, several candidate text boxes are merged to generate several text regions.

[0090] Specifically, this embodiment first applies non-maximum suppression to each adjusted prior box to filter out redundant prior boxes and remove prior boxes with high overlap, resulting in multiple candidate prior boxes. Then, among these candidate prior boxes, those with confidence scores greater than a confidence score threshold are selected to identify several candidate text boxes containing the foreground image, i.e., the text content. The remaining candidate text boxes are then sorted according to their confidence scores. Optionally, a Text Proposal Connector is used to merge adjacent candidate text boxes with higher confidence scores based on their positional information and confidence scores, thereby generating a text region. Specifically, in this embodiment, the merging process of candidate text boxes is divided into forward search and reverse search. Forward search involves searching along the positive horizontal direction for other candidate text boxes whose horizontal distance to the i-th candidate text box is less than a horizontal distance threshold (e.g., 50 pixels), and selecting the j-th candidate text box from these other candidate text boxes that has a vertical overlap greater than an overlap threshold (e.g., 0.7) and the highest confidence score. Reverse search involves searching along the negative horizontal direction for other candidate text boxes whose horizontal distance to the j-th candidate text box is less than a horizontal distance threshold (e.g., 50 pixels), and selecting the j-th candidate text box from these other candidate text boxes. Select the k-th candidate text box with the highest confidence score and the vertical overlap greater than the overlap threshold (e.g., 0.7) with the j-th candidate text box. If the confidence score of the ith candidate text box is greater than or equal to the confidence score of the k-th candidate text box, then the ith candidate text box to the j-th candidate text box is the longest connected text block. If the confidence score of the ith candidate text box is less than the confidence score of the k-th candidate text box, then the ith candidate text box to the j-th candidate text box is not the longest connected text block, and it is contained in the text block corresponding to the k-th candidate text box to the j-th candidate text box.

[0091] As a preferred embodiment, the method specifically encodes each of the text blocks through the following steps:

[0092] Based on the color information of each pixel in the text block, the text block is divided into a quadtree to obtain several leaf nodes; wherein, the color information of each pixel contained in the leaf node is the same or the size of the pixel region corresponding to the leaf node is a preset minimum segmentation size;

[0093] Based on the position, size, and color information of the pixel region corresponding to each leaf node, run-length encoding is used to generate run-length block information corresponding to each leaf node.

[0094] Based on the run-length block information, each text block is encoded to obtain the encoded data of each text block.

[0095] It's worth noting that text blocks contain simpler image information than non-text blocks, making them suitable for lossless encoding to achieve high-precision compression. Standard run-length encoding typically performs well on images with a single-color background or simple repetitive patterns. However, most text-based images feature simple text colors against complex backgrounds, necessitating adjustments to run-length encoding. Two-dimensional run-length encoding compresses data by scanning the image in a line-by-line manner and recording the run length of consecutive pixels of the same color. It is highly sensitive to color variations in the image. When the background color is complex, the boundaries between different colors form multiple small, consecutive pixel regions. Two-dimensional run-length encoding accurately records the position and size of these regions, allowing for accurate reconstruction of these color boundaries during decoding. Furthermore, by combining two-dimensional run-length encoding with data structures such as quadtrees, a higher compression ratio can be achieved. Therefore, this embodiment employs two-dimensional run-length encoding for lossless encoding of text blocks.

[0096] Specifically, in this embodiment, the entire text block is regarded as the root node of a quadtree. Then, it is determined whether each pixel in the current text block region has the same color information, such as whether it has the same grayscale value. If the current pixel region has the same color information, it is marked as a leaf node of the quadtree. If the current pixel region has different color information, it is divided into four equal sub-regions, and the quadtree segmentation process is recursively executed for each sub-region until the color information of each pixel in all pixel regions is the same, or the size of the pixel region reaches the preset minimum segmentation size, thereby obtaining several leaf nodes.

[0097] Furthermore, based on the position of the pixel region corresponding to each leaf node, such as the coordinates of the starting pixel in the pixel region, the top-left pixel in the pixel region can be used as the starting pixel, along with the size and color information of the pixel region, to generate the run-length block information corresponding to each leaf node, thereby achieving the encoding of the text block and forming the encoded data of the text block.

[0098] As a preferred embodiment, the method specifically encodes each of the non-text blocks through the following steps:

[0099] Each of the non-text blocks is encoded using the H.264 encoding format to obtain the encoded data of each non-text block.

[0100] Specifically, since non-text blocks contain more complex and richer image information, and the encoding method based on the H.264 encoding format has excellent compression performance, it can significantly reduce the amount of data while maintaining video quality, making it very suitable for processing complex content such as videos and images in cloud desktop scenarios.

[0101] Please see Figure 3 A second aspect of the present invention provides a cloud desktop text scene encoding device 100, comprising:

[0102] The text region detection module 11 is used to perform text region detection on the current text scene image to be processed using the Target Connected Text Proposal Network (CTPN) model, and obtain several text blocks and non-text blocks.

[0103] Encoding module 12 is used to encode each of the text blocks and the non-text blocks respectively to obtain the encoded data of the text scene image;

[0104] The target CTPN model uses a lightweight neural network as the spatial feature extraction network and a Transformer encoder as the sequence feature extraction network.

[0105] As a preferred embodiment, the text region detection module 11 is used to perform text region detection on the text scene image using the Target Connected Text Proposal Network (CTPN) model, obtaining several text blocks and non-text blocks, specifically including:

[0106] The lightweight neural network is used to sequentially perform high-level feature extraction and spatial feature extraction on the text scene image to obtain a high-level feature map and a first feature map with spatial feature information, respectively.

[0107] The Transformer encoder is used to extract sequence features from the high-level feature map to obtain a second feature map with sequence feature information.

[0108] The first feature map and the second feature map are fused to obtain a fused feature map;

[0109] Based on a number of predefined prior boxes with the same width but different heights corresponding to each feature point in the fused feature map, text region detection is performed on the fused feature map to obtain a number of text blocks and non-text blocks; wherein, the feature points are determined by splitting the fused feature map based on the feature size of the fused feature map.

[0110] As a preferred embodiment, the text region detection module 11 is used to sequentially perform high-level feature extraction and spatial feature extraction on the text scene image through the lightweight neural network, respectively obtaining a high-level feature map and a first feature map with spatial feature information, specifically including:

[0111] Low-level feature extraction is performed on the text scene image using convolutional layers to obtain a low-level feature map;

[0112] The high-level feature map is obtained by using several inverted residual blocks to extract high-level features from the low-level feature map.

[0113] Spatial features are extracted from the high-level feature map using a sliding window of a preset size to obtain the first feature map.

[0114] As a preferred embodiment, the text region detection module 11 is used to perform sequence feature extraction on the high-level feature map through the Transformer encoder to obtain a second feature map with sequence feature information, specifically including:

[0115] Based on each row of feature vectors in the high-level feature map after dimensional deformation processing, several feature sequences in the high-level feature map are obtained.

[0116] Add a position embedding vector to each of the aforementioned feature sequences to obtain an input feature sequence with positional information;

[0117] The second feature map is obtained by extracting sequence features from each of the input feature sequences using several Transformer basic blocks; wherein, the Transformer basic blocks include a multi-head attention layer and a position feedforward network.

[0118] As a preferred embodiment, the text region detection module 11 is used to perform text region detection on the fused feature map based on a number of preset prior boxes with the same width but different heights corresponding to each feature point in the fused feature map, to obtain a number of text blocks and non-text blocks, specifically including:

[0119] Based on several prior boxes corresponding to each feature point, the RPN layer in the target CTPN model is used to perform position regression and foreground or background classification on the prior boxes to obtain the position offset and confidence score of the prior boxes.

[0120] The prior boxes are adjusted based on the position offset to obtain the adjusted prior boxes;

[0121] Based on the adjusted prior boxes and the confidence scores, several text regions are generated, and several text blocks and non-text blocks are determined based on the text regions.

[0122] As a preferred embodiment, the text region detection module 11 is used to generate several text regions based on each of the adjusted prior boxes and the confidence scores, specifically including:

[0123] Non-maximum suppression is applied to each of the adjusted prior boxes to obtain several candidate prior boxes;

[0124] Based on each of the candidate prior boxes and the confidence score, the candidate prior boxes are filtered using a preset confidence score threshold to obtain a number of candidate text boxes.

[0125] Based on the position information and confidence score of each candidate text box, several candidate text boxes are merged to generate several text regions.

[0126] As a preferred embodiment, the encoding module 12 encodes each of the text blocks through the following steps:

[0127] Based on the color information of each pixel in the text block, the text block is divided into a quadtree to obtain several leaf nodes; wherein, the color information of each pixel contained in the leaf node is the same or the size of the pixel region corresponding to the leaf node is a preset minimum segmentation size;

[0128] Based on the position, size, and color information of the pixel region corresponding to each leaf node, run-length encoding is used to generate run-length block information corresponding to each leaf node.

[0129] Based on the run-length block information, each text block is encoded to obtain the encoded data of each text block.

[0130] As a preferred embodiment, the encoding module 12 encodes each of the non-text blocks through the following steps:

[0131] Each of the non-text blocks is encoded using the H.264 encoding format to obtain the encoded data of each non-text block.

[0132] The cloud desktop text scene encoding device 100 provided in this embodiment of the invention extracts spatial and sequence features from text scene images and performs text region detection by using a target CTPN model. Compared with the text scene image division method based on a fixed size, it significantly improves the detection efficiency and accuracy of text regions, effectively avoids the problem of missing text regions or including too many non-text parts in text blocks, and can meet the encoding requirements of cloud desktop text scenes.

[0133] Please see Figure 4 The third aspect of the present invention provides an electronic device 200, including a memory 22, a processor 21, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the cloud desktop text scene encoding method described in any embodiment of the first aspect.

[0134] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 200.

[0135] The electronic device 200 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 200 and does not constitute a limitation on the electronic device 200. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device 200 may also include input / output devices, network access devices, buses, etc.

[0136] The processor 21 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 21 can be any conventional processor 21. The processor 21 is the control center of the electronic device 200, connecting various parts of the electronic device 200 via various interfaces and lines.

[0137] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the electronic device 200 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0138] A fourth aspect of the present invention provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the cloud desktop text scene encoding method described in any embodiment of the first aspect.

[0139] A fifth aspect of the present invention provides a computer program product, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, it implements the steps of the cloud desktop text scene encoding method described in any embodiment of the first aspect.

[0140] Wherein, if the modules / units integrated in the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0141] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for encoding text scenes on a cloud desktop, characterized in that, include: The Target Connectivity Text Proposal Network (CTPN) model is used to detect text regions in the current text scene image to be processed, and obtain several text blocks and non-text blocks. Each of the text blocks and the non-text blocks is encoded separately to obtain the encoded data of the text scene image; The target CTPN model uses a lightweight neural network as the spatial feature extraction network and a Transformer encoder as the sequence feature extraction network. The method specifically encodes each of the text blocks through the following steps: Based on the color information of each pixel in the text block, the text block is divided into a quadtree to obtain several leaf nodes; wherein, the color information of each pixel contained in the leaf node is the same or the size of the pixel region corresponding to the leaf node is a preset minimum segmentation size; Based on the position, size, and color information of the pixel region corresponding to each leaf node, run-length encoding is used to generate run-length block information corresponding to each leaf node. Based on the run-length block information, each text block is encoded to obtain the encoded data of each text block.

2. The cloud desktop text scene encoding method as described in claim 1, characterized in that, The method of using the Target Connectivity Text Proposal Network (CTPN) model to detect text regions in a text scene image and obtain several text blocks and non-text blocks specifically includes: The lightweight neural network is used to sequentially perform high-level feature extraction and spatial feature extraction on the text scene image to obtain a high-level feature map and a first feature map with spatial feature information, respectively. The Transformer encoder is used to extract sequence features from the high-level feature map to obtain a second feature map with sequence feature information. The first feature map and the second feature map are fused to obtain a fused feature map; Based on a number of predefined prior boxes with the same width but different heights corresponding to each feature point in the fused feature map, text region detection is performed on the fused feature map to obtain a number of text blocks and non-text blocks; wherein, the feature points are determined by splitting the fused feature map based on the feature size of the fused feature map.

3. The cloud desktop text scene encoding method as described in claim 2, characterized in that, The step of sequentially performing high-level feature extraction and spatial feature extraction on the text scene image using the lightweight neural network to obtain a high-level feature map and a first feature map with spatial feature information, specifically includes: Low-level feature extraction is performed on the text scene image using convolutional layers to obtain a low-level feature map; The high-level feature map is obtained by using several inverted residual blocks to extract high-level features from the low-level feature map. Spatial features are extracted from the high-level feature map using a sliding window of a preset size to obtain the first feature map.

4. The cloud desktop text scene encoding method as described in claim 2, characterized in that, The step of extracting sequence features from the high-level feature map using the Transformer encoder to obtain a second feature map with sequence feature information specifically includes: Based on each row of feature vectors in the high-level feature map after dimensional deformation processing, several feature sequences in the high-level feature map are obtained. Add a position embedding vector to each of the aforementioned feature sequences to obtain an input feature sequence with positional information; The second feature map is obtained by extracting sequence features from each of the input feature sequences using several Transformer basic blocks; wherein, the Transformer basic blocks include a multi-head attention layer and a position feedforward network.

5. The cloud desktop text scene encoding method as described in claim 2, characterized in that, The step of detecting text regions in the fused feature map based on a set of pre-defined bounding boxes of the same width but different heights corresponding to each feature point in the fused feature map, thereby obtaining a set of text blocks and non-text blocks, specifically includes: Based on several prior boxes corresponding to each feature point, the RPN layer in the target CTPN model is used to perform position regression and foreground or background classification on the prior boxes to obtain the position offset and confidence score of the prior boxes. The prior boxes are adjusted based on the position offset to obtain the adjusted prior boxes; Based on the adjusted prior boxes and the confidence scores, several text regions are generated, and several text blocks and non-text blocks are determined based on the text regions.

6. The cloud desktop text scene encoding method as described in claim 5, characterized in that, The generation of several text regions based on the adjusted prior boxes and the confidence scores specifically includes: Non-maximum suppression is applied to each of the adjusted prior boxes to obtain several candidate prior boxes; Based on each of the candidate prior boxes and the confidence score, the candidate prior boxes are filtered using a preset confidence score threshold to obtain a number of candidate text boxes. Based on the position information and confidence score of each candidate text box, several candidate text boxes are merged to generate several text regions.

7. The cloud desktop text scene encoding method as described in claim 1, characterized in that, The method specifically encodes each of the non-text blocks through the following steps: Each of the non-text blocks is encoded using the H.264 encoding format to obtain the encoded data of each non-text block.

8. A cloud desktop text scene encoding device, characterized in that, include: The text region detection module is used to detect text regions in the current text scene image to be processed using the Target Connected Text Proposal Network (CTPN) model, and obtain several text blocks and non-text blocks. The encoding module is used to encode each of the text blocks and the non-text blocks respectively to obtain the encoded data of the text scene image; The target CTPN model uses a lightweight neural network as the spatial feature extraction network and a Transformer encoder as the sequence feature extraction network. The encoding module encodes each text block through the following steps: Based on the color information of each pixel in the text block, the text block is divided into a quadtree to obtain several leaf nodes; wherein, the color information of each pixel contained in the leaf node is the same or the size of the pixel region corresponding to the leaf node is a preset minimum segmentation size; Based on the position, size, and color information of the pixel region corresponding to each leaf node, run-length encoding is used to generate run-length block information corresponding to each leaf node. Based on the run-length block information, each text block is encoded to obtain the encoded data of each text block.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cloud desktop text scene encoding method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the cloud desktop text scene encoding method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the steps of the cloud desktop text scene encoding method according to any one of claims 1 to 7.

Images