Image processing method and electronic equipment

Abstract

The invention provides an image processing method and electronic equipment, and the method comprises the steps: carrying out the coding of a target image through an encoding module of an image segmentation model, and obtaining a target image feature; processing the target image by using the visual feature extraction model to obtain a target visual feature; the target visual features represent visual attributes of multi-scale visual information in the target image; decoding based on the target image features and the target visual features by using a decoding module of the image segmentation model to obtain a text segmentation result of the target image; the decoding module is trained to output a result of text segmentation from the image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method and electronic device. Background Technology

[0002] In the field of image text segmentation, it is necessary to accurately distinguish text pixels from complex backgrounds. Existing technologies lack sufficient accuracy for segmenting text with diverse shapes and complex backgrounds, making it difficult to balance detail preservation and semantic understanding, resulting in blurred segmentation boundaries. Summary of the Invention

[0003] This disclosure provides an image processing method and an electronic device.

[0004] According to one aspect of this disclosure, an image processing method is provided, comprising: encoding a target image using an encoding module of an image segmentation model to obtain target image features; processing the target image using a visual feature extraction model to obtain target visual features; the target visual features characterizing the visual attributes of multi-scale visual information in the target image; decoding the target image based on the target image features and the target visual features using a decoding module of an image segmentation model to obtain a text segmentation result of the target image; the decoding module is trained to output the result of text segmentation from the image.

[0005] According to embodiments of this disclosure, the method further includes: acquiring text guidance features; using the text guidance features to guide the model to focus on text features; fusing the text guidance features and target image features to obtain target fusion features; and using the decoding module of the image segmentation model to decode based on the target image features and target visual features to obtain the text segmentation result of the target image, including: using the decoding module to decode based on the target fusion features and target visual features to obtain the text segmentation result of the target image.

[0006] According to embodiments of this disclosure, obtaining text guidance features includes: obtaining initial descriptive text; the initial descriptive text representing the visual attributes of text-type objects; obtaining a target query vector; the target query vector being used to guide the model to focus on text features related to the target image; fusing the initial descriptive text and the target query vector to obtain combined semantic information; and determining text guidance features based on the combined semantic information.

[0007] According to embodiments of this disclosure, obtaining initial description text includes: inputting text attribute description instructions into a large language model; and obtaining the initial description text output by the large language model.

[0008] According to embodiments of this disclosure, determining text guidance features based on combined semantic information includes: inputting the combined semantic information into a pre-trained text encoder for semantic encoding to obtain a text feature vector; and performing linear projection processing on the text feature vector to obtain text guidance features.

[0009] According to embodiments of this disclosure, text guidance features and target image features are fused to obtain target fusion features, including: fusing text guidance features and target image features based on a cross-attention mechanism to obtain cross-modal alignment features; and adding the cross-modal alignment features to the target image features to obtain target fusion features.

[0010] According to embodiments of this disclosure, a target image is processed using a visual feature extraction model to obtain target visual features, including: extracting multi-scale visual information from the target image using the visual feature extraction model; fusing the multi-scale visual information using the visual feature extraction model to obtain fused visual features of a unified scale; and performing feature optimization processing on the fused visual features to obtain target visual features.

[0011] According to embodiments of this disclosure, the training process of the image segmentation model and the visual feature extraction model includes: inputting training data into an initial model and obtaining the training results output by the initial model; adjusting the model parameters of the initial model according to the training results to obtain the trained model; the parameters of the encoding module of the trained model are the same as the parameters of the encoding module of the initial model, and the parameters of the decoding module of the trained model are different from the parameters of the decoding module of the initial model.

[0012] According to embodiments of this disclosure, the text segmentation result of the target image is obtained by decoding the target image features and target visual features using the decoding module of the image segmentation model. This includes: decoding the target image features and target visual features using the decoding module to generate an initial segmentation mask; and performing resolution enhancement and edge optimization processing on the initial segmentation mask to obtain the text segmentation result of the target image.

[0013] Another aspect of this disclosure provides an electronic device comprising: an image acquisition module for acquiring a target image; at least one processor; configured to: encode the target image using an encoding module of an image segmentation model to obtain target image features; process the target image using a visual feature extraction model to obtain target visual features; the target visual features characterizing the visual attributes of multi-scale visual information in the target image; and decode the target image based on the target image features and the target visual features using a decoding module of an image segmentation model to obtain a text segmentation result of the target image; the decoding module is trained to output the result of text segmentation from the image.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0016] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0017] Figure 2 This is a flowchart of an image processing method according to another embodiment of the present disclosure;

[0018] Figure 3 This is a flowchart of an image processing method according to yet another embodiment of the present disclosure;

[0019] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure; and

[0020] Figure 5 This is a schematic block diagram of an example electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0023] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0024] like Figure 1 As shown, the image processing method of this embodiment includes operations S110-S130.

[0025] In operation S110, the target image is encoded using the encoding module of the image segmentation model to obtain the target image features.

[0026] In the embodiments of this disclosure, the image segmentation model refers to a trained machine learning model capable of dividing an image into different semantic regions. The image segmentation model includes an encoding module and a decoding module. The encoding module is responsible for extracting image features, and the decoding module is responsible for generating the segmentation result. The encoding module performs deep feature extraction on the input image, transforming the image from pixel space to feature space.

[0027] For example, image segmentation models can be Segment Anything Model (SAM), Deep Lab series models, high-resolution network models, and segmentation deformer models, etc.

[0028] In embodiments of this disclosure, the target image refers to the input image to be segmented into text. For example, the target image may include natural scene images such as street view signs, document photos, or posters.

[0029] In the embodiments of this disclosure, target image features refer to the feature representation output by the encoding module after processing the target image, which includes the global semantic information of the image.

[0030] In the embodiments of this disclosure, the target image is encoded using the encoding module of an image segmentation model to obtain target image features. The target image is input into the encoding module of a pre-trained image segmentation model. The encoding module can abstract the image layer by layer through a series of convolutional or transform layers, and output a feature map containing global context information, i.e., the target image features.

[0031] In operation S120, the target image is processed using a visual feature extraction model to obtain target visual features; the target visual features characterize the visual attributes of multi-scale visual information in the target image.

[0032] In embodiments of this disclosure, the visual feature extraction model is a machine learning model for extracting multi-scale visual features from images. The visual feature extraction model can capture visual information of different sizes and levels of detail in an image.

[0033] For example, visual feature extraction models can be SegFormer, visual geometry network, residual network, high-resolution network, etc.

[0034] In embodiments of this disclosure, multi-scale visual information refers to the features presented by the same image at different resolutions or different levels of abstraction. For example, low-resolution features contain stronger semantic information (such as the general category of an object), while high-resolution features contain richer detail information (such as object edges and textures).

[0035] In the embodiments of this disclosure, the target visual feature refers to the feature representation output by the visual feature extraction model after processing the target image. This feature integrates multi-scale visual information and focuses on spatial details.

[0036] In the embodiments of this disclosure, a visual feature extraction model is used to process the target image to obtain the target visual features. This embodiment can obtain text-related detailed features in the target image through the visual feature extraction model. The visual feature extraction model extracts features of different scales from the target image through its internal structure, and fuses these features to generate a feature representation rich in detailed information such as text edges and strokes.

[0037] For example, the target image is a street photograph containing a shop sign. The visual feature extraction model analyzes the target image and generates target visual features. These visual features can clearly highlight the sharp edges of the strokes in the text on the sign, the gaps between characters, and other subtle structures, even if the text is curved or stylized.

[0038] In operation S130, the decoding module of the image segmentation model is used to decode based on the target image features and target visual features to obtain the text segmentation result of the target image; the decoding module is trained to output the result of text segmentation from the image.

[0039] In the embodiments of this disclosure, the decoding module is part of the image segmentation model. The decoding module is used to recover the spatial position and contour of the target object based on the target image features (and possible external auxiliary features) extracted by the encoding module, and generate a segmentation result.

[0040] In the embodiments of this disclosure, the decoding module of the image segmentation model decodes based on target image features and target visual features to obtain text segmentation results of the target image. The decoding module receives target image features and target visual features. The decoding module fuses this information to accurately delineate the pixel-level boundaries of the text region.

[0041] In embodiments of this disclosure, target image features can provide global semantic guidance on "where text might be," while target visual features can provide local detail information on "where the text edges are specifically located."

[0042] In the embodiments of this disclosure, the training of the decoding module is a supervised learning process, the goal of which is to enable the decoding module to correctly associate the input feature information with the text segmentation results. During the training phase, by using a large amount of training image data containing precise annotations (i.e., whether each pixel belongs to text), the internal parameters of the decoding module are continuously adjusted through optimization algorithms (such as gradient descent), enabling it to learn to recognize and reconstruct the unique patterns of text regions (such as stroke continuity, edge sharpness, etc.). When processing the target image, the trained decoding module can generate segmentation results containing only text regions, rather than other irrelevant objects, based on the received image features and visual features.

[0043] Through the embodiments of this disclosure, by collaboratively utilizing the encoding module and visual feature extraction model of the image segmentation model, target image features and target visual features are extracted respectively. These are then fused and decoded by the decoding module to obtain the text segmentation result. This effectively overcomes the problem of blurred text boundary segmentation caused by the lack of detail perception capability in global features in related technologies, as well as the limitation of a single model in simultaneously achieving semantic understanding and detail preservation. Through the complementarity of dual feature sources, high-precision and robust segmentation of text regions can be achieved in complex scenes, significantly improving the accuracy and completeness of the segmentation results.

[0044] In some embodiments of this disclosure, the method further includes: acquiring text guidance features; using the text guidance features to guide the model to focus on text features; fusing the text guidance features and target image features to obtain target fusion features; and using the decoding module of the image segmentation model to decode based on the target image features and target visual features to obtain the text segmentation result of the target image, including: using the decoding module to decode based on the target fusion features and target visual features to obtain the text segmentation result of the target image.

[0045] In embodiments of this disclosure, text-guided features are semantic feature vectors generated through natural language descriptions, used to provide the model with guidance on abstract attributes of text regions (such as "the text has sharp edges and high contrast"). The essence of text-guided features is to embed prior knowledge into the feature space.

[0046] In embodiments of this disclosure, textual guidance features are obtained. The steps of this embodiment aim to supplement the model with semantic priors in the text domain. Semantic representations describing the general properties of text are generated using external technical means (such as language models) and encoded into machine-processable feature vectors.

[0047] For example, for a street view billboard image, text guidance features can encode semantic information such as "text is usually regularly shaped and has a clear contrast with the background color", guiding the model to focus on areas that conform to these characteristics.

[0048] In the embodiments of this disclosure, the target fusion feature is an enhanced feature obtained by fusing textual guidance features and target image features through a specific fusion mechanism. The target fusion feature simultaneously preserves the visual information of the image and the semantic guidance information of the text.

[0049] In the embodiments of this disclosure, textual guidance features and target image features are fused to obtain target fused features. This fusion mechanism combines semantic guidance with visual features, enabling the model to simultaneously perceive image content and text attributes during decoding.

[0050] In the embodiments of this disclosure, a decoding module is used to decode based on target fusion features and target visual features to obtain the text segmentation result of the target image. The decoding module simultaneously receives target fusion features (providing semantically enhanced image representation) and target visual features (providing multi-scale details), and generates a more accurate text segmentation mask through feature complementarity.

[0051] Through embodiments of this disclosure, by introducing text-guided features and fusing them with image features, the model can simultaneously utilize visual details and semantic priors during decoding, significantly improving the recognition accuracy and robustness of text in complex scenes.

[0052] Figure 2 This is a flowchart of an image processing method according to another embodiment of the present disclosure.

[0053] like Figure 2 As shown, in this embodiment, obtaining text guidance features includes operations S210-S240.

[0054] In operation S210, the initial description text is obtained; the initial description text represents the visual attributes of the text class object.

[0055] In the embodiments of this disclosure, the initial description text refers to semantic content described in natural language that represents the common visual attributes of text-type objects. The initial description text provides general textual prior knowledge across scenarios.

[0056] In the embodiments of this disclosure, initial descriptive text is obtained. Natural language representations describing the essential attributes of the text are generated using external technical means, providing a stable semantic foundation for subsequent feature generation.

[0057] For example, a language model can be triggered to automatically generate a description by inputting "summarize the typical visual features of text", and the text description "text usually has continuous strokes and has a color difference with the background" can be obtained.

[0058] In operation S220, the target query vector is obtained; the target query vector is used to guide the model to focus on text features related to the target image.

[0059] In embodiments of this disclosure, the target query vector is a learnable parameter vector that adaptively captures specific textual features (such as decorative strokes of artistic fonts) related to the current task or data distribution during training.

[0060] In embodiments of this disclosure, a target query vector is obtained. Through an optimization process, this vector learns specific patterns of text in the dataset, thus providing a personalized supplement to general text priors.

[0061] For example, when processing handwritten text datasets, the target query vector can learn characteristics such as "uneven stroke thickness and loose structure," which complement the regular description of the initial descriptive text.

[0062] In operation S230, the initial description text and the target query vector are fused to obtain combined semantic information.

[0063] In the embodiments of this disclosure, the combined semantic information is a unified semantic representation that fuses the general attributes of the initial descriptive text with the specific information of the target query vector.

[0064] In the embodiments of this disclosure, the initial description text and the target query vector are fused to obtain combined semantic information. The two types of information are then integrated through methods such as concatenation or weighted fusion to form a semantic carrier that is both universal and adaptable.

[0065] For example, by combining the initial text describing "sharp edges of text" with the target query vector learned that "curved strokes are common", a reinforced semantic guidance can be formed that "texts with sharp curved edges should be paid attention to".

[0066] In operation S240, text guidance features are determined based on combined semantic information.

[0067] In embodiments of this disclosure, combined semantic information is mapped into machine-processable low-dimensional feature vectors, enabling them to participate in model computation.

[0068] For example, the fused semantic information can be converted into a feature vector by an encoder, which serves as a semantic guidance signal during decoding.

[0069] Through the embodiments of this disclosure, text guidance features are generated by fusing general text attributes and scene-adaptive features, enabling semantic guidance to maintain stability while also being scene-specific, significantly improving the model's adaptability to segmenting diverse texts.

[0070] In some embodiments of this disclosure, obtaining the initial description text includes: inputting text attribute description instructions into a large language model; and obtaining the initial description text output by the large language model.

[0071] In the embodiments of this disclosure, a large language model refers to a large-scale artificial intelligence model trained on massive amounts of text data that is capable of understanding and generating natural language text. Large language models possess powerful language understanding and knowledge representation capabilities.

[0072] In the embodiments of this disclosure, text attribute description instructions refer to artificially constructed natural language instructions used to guide a large language model to generate specific content, which explicitly require the model to output descriptive text related to the visual attributes of text. Initial description text refers to the natural language description generated by the large language model according to the instructions, representing the visual attributes of text-type objects.

[0073] In embodiments of this disclosure, text attribute description instructions are input into a large language model. By constructing precise instructions, the large language model is guided to focus on the specific domain of text visual attributes for content generation.

[0074] For example, the system inputs instructions into the large language model, such as "Please describe the typical visual features and attributes of text regions in natural scene images from a computer vision perspective." This instruction clearly defines the technical field, object, and content scope of the generated content.

[0075] In embodiments of this disclosure, initial descriptive text output by a large language model is obtained. Upon receiving instructions, the large language model generates natural language text that meets the requirements and describes the general attributes of the text, based on its internally stored language patterns and visual knowledge. This text will serve as the original semantic input for subsequently generating text-guided features.

[0076] For example, a large language model might output the following initial descriptive text: "Text in natural scenes typically has high color contrast with the background, sharp edge contours, continuous stroke structure, relatively regular character spacing and layout, and a relatively consistent font style." This text accurately summarizes the common visual attributes of text across different scenes.

[0077] Through the embodiments of this disclosure, high-quality text attribute descriptions are generated using a large language model, ensuring the accuracy and richness of the initial description text. This lays a solid foundation for constructing accurate text-guided features, thereby improving the semantic understanding capability of subsequent text segmentation.

[0078] In some embodiments of this disclosure, determining text guidance features based on combined semantic information includes: inputting the combined semantic information into a pre-trained text encoder for semantic encoding to obtain a text feature vector; and performing linear projection processing on the text feature vector to obtain text guidance features.

[0079] In the embodiments of this disclosure, the pre-trained text encoder refers to a deep learning model trained on massive amounts of text-image pairs, capable of converting natural language descriptions into numerical feature vectors. The pre-trained text encoder possesses powerful semantic understanding capabilities.

[0080] In the embodiments of this disclosure, semantic encoding refers to the computational process of converting input natural language text into its internal semantic representation using a text encoder. A text feature vector is a high-dimensional numerical vector containing deep semantic information of the text, output by the text encoder after semantic encoding of the input text.

[0081] In embodiments of this disclosure, linear projection processing refers to a mathematical operation that transforms an input feature vector to a specified dimension through a fully connected layer (linear layer).

[0082] In the embodiments of this disclosure, the combined semantic information is input into a pre-trained text encoder for semantic encoding to obtain a text feature vector. The steps of this embodiment utilize the powerful semantic understanding capability of the pre-trained text encoder to transform the combined semantic information from a discrete symbolic representation into a continuous, semantically rich feature vector representation.

[0083] For example, the combined semantic information "find text regions with sharp edges and high contrast" is input into a text encoder. The encoder transforms it into a high-dimensional vector (e.g., 512-dimensional) that numerically represents the composite semantic concepts of "sharp edges," "high contrast," and "text regions."

[0084] In embodiments of this disclosure, text feature vectors are linearly projected to obtain text guidance features. A linear projection layer projects the text feature vectors onto a specific dimension that matches other modules of the model, enabling effective cross-modal interaction with visual features.

[0085] For example, a 512-dimensional text feature vector can be projected onto a 256-dimensional vector through a linear layer. This operation unifies the feature dimensions and further refines and optimizes the feature representation during the projection process, making it more suitable for guiding subsequent image segmentation tasks.

[0086] Through the embodiments of this disclosure, the natural language description is efficiently transformed into low-dimensional dense features for adaptation model computation by cascading pre-trained text encoder and linear projection, ensuring the fidelity and usability of semantic guidance information and providing high-quality text guidance signals for subsequent feature fusion.

[0087] In some embodiments of this disclosure, text guidance features and target image features are fused to obtain target fusion features, including: fusing text guidance features and target image features based on a cross-attention mechanism to obtain cross-modal alignment features; and adding the cross-modal alignment features to the target image features to obtain target fusion features.

[0088] In embodiments of this disclosure, the cross-attention mechanism is a neural network mechanism that allows features from different modalities (such as text and images) to interact. By calculating the relevance weights of query-key-value pairs, features from one modality can selectively focus on important information from another modality.

[0089] In the embodiments of this disclosure, cross-modal alignment features refer to features obtained after processing by a cross-attention mechanism, wherein visual features have been reweighted according to text semantic guidance to achieve spatial alignment between visual and semantic information.

[0090] In the embodiments of this disclosure, the addition process refers to the element-wise addition of two feature vectors of the same dimension, which is used to fuse feature information from different sources.

[0091] In the embodiments of this disclosure, text-guided features and target image features are fused based on a cross-attention mechanism to obtain cross-modal aligned features. The steps of this embodiment use target image features as queries and text-guided features as keys and values. Through attention calculation, the image features actively "query" regions related to the text semantics, thereby generating semantically enhanced visual features.

[0092] For example, in street view images, text-guided features contain the semantics of "sharp text edges". The cross-attention mechanism calculates the correlation between each region of the image and this semantics, which enhances the visual feature response of the text edge region and weakens the response of the background texture region.

[0093] In the embodiments of this disclosure, cross-modal alignment features are added to target image features to obtain target fusion features. The original target image features are then added to the aligned features using a residual connection method, incorporating semantic guidance while preserving the original visual information, thus forming complementary and enhanced fusion features.

[0094] For example, by adding cross-modal alignment features that highlight text edges to the original image features that contain complete scene information, a target fusion feature is obtained that both preserves the global context and enhances the response of text regions.

[0095] Through the embodiments of this disclosure, the cross-attention mechanism enables precise guidance of image features by text semantics, allowing the model to adaptively focus on text-related regions, significantly improving the accuracy and robustness of text segmentation in complex backgrounds.

[0096] Figure 3 This is a flowchart of an image processing method according to yet another embodiment of the present disclosure.

[0097] like Figure 3As shown, in this embodiment, the target image is processed using a visual feature extraction model to obtain the target visual features, including operations S310-S330.

[0098] When operating S310, a visual feature extraction model is used to extract multi-scale visual information from the target image.

[0099] In the embodiments of this disclosure, fusion processing refers to integrating visual information at different scales to achieve complementary enhancement of detailed features and semantic features. Unified-scale fused visual features refer to multi-scale fusion results that maintain spatial consistency after scale alignment and feature concatenation. Feature optimization processing refers to reducing the dimensionality and refining the fused features through operations such as convolution to improve their feature representation quality.

[0100] In the embodiments of this disclosure, a visual feature extraction model is used to extract multi-scale visual information from a target image. The visual feature extraction model analyzes the input image through its hierarchical architecture. The shallow layers of the model have a smaller receptive field, focusing more on local details and capturing fine-grained features such as edges, corners, and textures in the image. The deeper layers have a larger receptive field, integrating more contextual information and understanding a wider range of regional relationships and semantic content. This generates feature map sets at different scales (i.e., different resolutions and different levels of abstraction), resulting in a multi-scale visual information representation of the target image.

[0101] For example, taking an image containing a street sign as an example, when the image is input into a visual feature extraction model, the shallow layer of the model outputs a high-resolution feature map, which clearly activates the sharp edges of each stroke of the sign text, the corners of the strokes, and the contrast boundaries between the characters and the background. The deeper layer of the model outputs a lower-resolution feature map, which focuses more on the overall semantics and may have a high response across the entire text area, indicating that it recognizes this as a coherent text block, but the specific stroke details are blurred.

[0102] When operating S320, the visual feature extraction model fuses multi-scale visual information to obtain fused visual features of a unified scale.

[0103] In the embodiments of this disclosure, the visual feature extraction model fuses multi-scale visual information to obtain fused visual features of a unified scale. Upsampling operations can be used to enlarge low-resolution, semantically strong feature maps to the same size as high-resolution feature maps, and then splicing or weighted fusion can be performed along the channel dimension. The precise spatial details provided by high-resolution features and the rich semantic context implied by low-resolution features can be effectively combined to form a more comprehensive and scale-unified fused feature representation.

[0104] For example, the model upsamples the low-resolution semantic feature maps from the deep layer output that identify "text regions," making their size similar to the high-resolution detail features from the shallow layer output. Figure 1 Then, these two feature maps are concatenated along the channel dimension. The resulting fused feature map contains detailed information from the shallow layer indicating "text edges exist here," as well as semantic information from the deep layer indicating "this region belongs to a whole text."

[0105] In operation S330, feature optimization processing is performed on the fused visual features to obtain the target visual features.

[0106] In the embodiments of this disclosure, the fused visual features are subjected to feature optimization processing to obtain the target visual features. This optimization can be achieved through operations such as convolutional layers. Convolutional layers can learn how to filter and combine useful information from the fused features, suppress noise, and output a more compact and discriminative feature map, i.e., the final target visual features.

[0107] For example, the fused feature map obtained by stitching is processed using convolutional layers. The convolutional layer acts like a "smart filter," learning that for text segmentation tasks, it should strengthen the responses of locations that are both labeled as "text regions" by deep semantic features and detected as "edges" by shallow detail features, while weakening the responses of regions that respond to detail features (such as background texture) but are not supported by semantic features. After this optimization, the resulting target visual features are spatially rich and highly optimized for text regions.

[0108] Through the embodiments of this disclosure, multi-scale feature extraction and fusion are used to fully leverage the complementary advantages of visual information at different levels. After feature optimization, target visual features with rich details and clear semantics are finally obtained, providing high-quality spatial detail information for text segmentation.

[0109] In some embodiments of this disclosure, the training process of the image segmentation model and the visual feature extraction model includes: inputting training data into an initial model and obtaining the training results output by the initial model; adjusting the model parameters of the initial model according to the training results to obtain the trained model; the parameters of the encoding module of the trained model are the same as the parameters of the encoding module of the initial model, and the parameters of the decoding module of the trained model are different from the parameters of the decoding module of the initial model.

[0110] In the embodiments of this disclosure, training data refers to a dataset containing input images and corresponding ground truth segmentation labels, used for supervised learning of the model. The initial model refers to a neural network model with initial parameter values ​​set up before the training process begins.

[0111] In the embodiments of this disclosure, training data is input into an initial model to obtain the training results output by the initial model. The steps in this embodiment constitute the forward propagation process of model training. The training data includes the input image and its corresponding ground truth segmentation annotations. After the training data is input into the model, the data undergoes forward computation sequentially through the model's encoding and decoding modules. The encoding module extracts features from the image to generate feature representations, and the decoding module performs upsampling and pixel classification prediction based on these features, outputting a probability map of each pixel belonging to a text region, i.e., the training result. This process represents the model's inference process based on the current parameters and the input data.

[0112] For example, the training data might consist of an image containing a street view sign and a binary mask of precisely labeled text regions. After the image is input into the initial model, the model outputs a probability map of the same size as the input image. Each pixel value in the map represents the probability that the model currently classifies that pixel as text. In the early stages of training, due to unoptimized parameters, this probability map may be highly inaccurate, for example, misclassifying background as text.

[0113] In the embodiments of this disclosure, the model parameters of the initial model are adjusted based on the training results to obtain the trained model. By comparing the difference between the model output and the ground truth labels, the gradient is calculated using the backpropagation algorithm, and the model parameters are selectively updated and optimized. First, the loss function value is calculated. Then, using the backpropagation algorithm, the gradient of the loss value with respect to each model parameter is calculated. This gradient indicates the direction and magnitude of adjustment for each parameter to reduce the loss. The optimizer updates the model parameters based on the gradient information.

[0114] In the embodiments of this disclosure, only the parameters of the decoding module are adjusted, while the parameters of the encoding module remain unchanged. Gradients are blocked during backpropagation to the encoding module; only the parameters of the decoding module are optimized and adjusted based on the training data.

[0115] For example, the difference between the probability map output by the model and the real text mask is calculated. Through backpropagation, the gradient of the loss with respect to the parameters of each layer of the decoder is calculated. Based on these gradients, the optimizer updates the decoder parameters so that it can output a more accurate probability map next time. Meanwhile, the encoder parameters remain unchanged throughout the process. This results in a trained model where the encoder retains its original weights, while the decoder has been specifically optimized.

[0116] Through the embodiments of this disclosure, by freezing the parameters of the encoding module and training only the decoding module, the model's feature extraction capability is maintained while significantly reducing the number of training parameters. This achieves efficient parameter fine-tuning, significantly improves training efficiency, and reduces computational resource requirements.

[0117] In some embodiments of this disclosure, the decoding module of the image segmentation model decodes the target image features and target visual features to obtain the text segmentation result of the target image, including: using the decoding module to decode the target image features and target visual features to generate an initial segmentation mask; and performing resolution enhancement and edge optimization processing on the initial segmentation mask to obtain the text segmentation result of the target image.

[0118] In the embodiments of this disclosure, the initial segmentation mask refers to the low-resolution binary segmentation result directly output by the decoding module, where each pixel value represents the probability that the position belongs to a text region.

[0119] In the embodiments of this disclosure, resolution enhancement refers to the process of restoring the low-resolution segmentation result to the original image size through upsampling techniques. Edge optimization refers to refining the boundary regions of the segmentation mask to make the text edges smoother and more accurate.

[0120] In embodiments of this disclosure, the text segmentation result refers to the final segmentation mask with the same resolution as the input image obtained after complete processing.

[0121] In the embodiments of this disclosure, a decoding module is used to decode the target image features and target visual features to generate an initial segmentation mask. The decoding module receives target image features (containing global semantic information of the image) from the image segmentation model encoding module and target visual features (rich in spatial details of text regions) from the visual feature extraction model. Through feature fusion and pixel-level classification, these two complementary features are mapped into a preliminary, low-resolution binary segmentation map, i.e., the initial segmentation mask. Each pixel value of the initial segmentation mask represents the probability that the model initially determines that the location belongs to text.

[0122] For example, the target image is a 1024x1024 pixel street view photo containing a shop sign. After the image is encoded, it may yield a 32x32 or 64x64 deep feature map. The decoding module fuses the target image features containing the semantics of the "sign area" with the target visual features containing the details of the strokes of the text on the sign. Through a series of convolution and upsampling operations, the decoding module outputs an initial 256x256 segmentation mask. On this mask, the text area of ​​the sign will be displayed as a bright color (high probability value), but the edges of the text may be relatively rough, and due to the low resolution, the details of small characters may be blurry.

[0123] In the embodiments of this disclosure, the initial segmentation mask undergoes resolution upscaling and edge optimization to obtain the text segmentation result of the target image. Because the initial segmentation mask has low resolution and coarse edges, it cannot be directly used as a high-quality segmentation result. The purpose of resolution upscaling is to enlarge the low-resolution initial mask to the same size as the original input image, for example, upsampling from 256x256 to 1024x1024. This can be achieved through transposed convolution.

[0124] In the embodiments of this disclosure, upsampling can produce blurred and jagged edges. Therefore, edge optimization processing is required. This can be achieved by filtering the upsampled mask using one or more standard convolutional layers. These convolutional layers can learn to smooth edges, sharpen boundaries, and eliminate noise or artifacts introduced by upsampling, thereby generating a final text segmentation result with clear boundaries and pixel-level alignment with the input image.

[0125] For example, an initial 256x256 mask is upsampled to 1024x1024 using a transposed convolution. The general shape of the sign text emerges, but the edges are blurry and jagged. Passing the upsampled mask through a lightweight convolutional neural network (e.g., containing three to four 3x3 convolutions) smooths the jagged edges, making the outlines of the text strokes sharp and continuous, and filling in any gaps in the strokes that might appear broken due to low resolution. The output is a high-quality 1024x1024 binary segmentation mask where the text on the sign is accurately segmented with smooth edges and complete details.

[0126] Through the embodiments of this disclosure, a hierarchical processing strategy is adopted to first generate low-resolution initial results to ensure processing efficiency, and then to ensure final accuracy through upsampling and edge optimization. This significantly improves the segmentation quality and detail restoration of text boundaries while maintaining computational efficiency.

[0127] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure.

[0128] In this embodiment of the disclosure, the electronic device 400 includes: an image acquisition module 410 for acquiring a target image; at least one processor 420 for: encoding the target image using an encoding module of an image segmentation model to obtain target image features; processing the target image using a visual feature extraction model to obtain target visual features; the target visual features characterize the visual attributes of multi-scale visual information in the target image; decoding the target image based on the target image features and the target visual features using a decoding module of an image segmentation model to obtain a text segmentation result of the target image; the decoding module is trained to output the result of text segmentation from the image.

[0129] In this embodiment of the disclosure, processor 420 may execute the image processing method described above.

[0130] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0131] Figure 5 This is a schematic block diagram of an example electronic device used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0132] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0133] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0134] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform image processing methods by any other suitable means (e.g., by means of firmware).

[0135] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: electronic devices for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0139] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0140] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0141] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0142] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image processing method, comprising: The target image is encoded using the encoding module of the image segmentation model to obtain the target image features; The target image is processed using a visual feature extraction model to obtain the target visual features; The target visual features characterize the visual attributes of multi-scale visual information in the target image; The decoding module of the image segmentation model decodes the target image based on the target image features and the target visual features to obtain the text segmentation result of the target image; the decoding module is trained to output the result of text segmentation from the image.

2. The method according to claim 1, further comprising: Obtain text-guided features; these text-guided features are used to guide the model to focus on text features. The text guidance features and the target image features are fused together to obtain the target fusion features; The decoding module utilizing the image segmentation model decodes the target image based on the target image features and the target visual features to obtain the text segmentation result of the target image, including: The decoding module is used to decode the target image based on the target fusion features and the target visual features to obtain the text segmentation result of the target image.

3. The method according to claim 2, wherein obtaining the text guidance feature includes: Get the initial description text; The initial descriptive text represents the visual attributes of the text class object; Obtain the target query vector; The target query vector is used to guide the model to focus on text features related to the target image; The initial description text and the target query vector are fused together to obtain combined semantic information; Based on the combined semantic information, the text guidance features are determined.

4. The method according to claim 3, wherein obtaining the initial description text includes: Input text attribute description instructions into the large language model; Obtain the initial description text output by the large language model.

5. The method according to claim 3, wherein determining the text guidance feature based on the combined semantic information comprises: The combined semantic information is input into a pre-trained text encoder for semantic encoding to obtain a text feature vector; The text feature vector is linearly projected to obtain the text guidance feature.

6. The method according to claim 2, wherein fusing the text guidance features and the target image features to obtain target fusion features includes: The text guidance features and the target image features are fused based on the cross-attention mechanism to obtain cross-modal alignment features; The cross-modal alignment features are added to the target image features to obtain the target fusion features.

7. The method according to claim 1, wherein processing the target image using a visual feature extraction model to obtain target visual features includes: The visual feature extraction model is used to extract multi-scale visual information from the target image; The visual feature extraction model fuses the multi-scale visual information to obtain fused visual features of a unified scale. The fused visual features are subjected to feature optimization processing to obtain the target visual features.

8. The method according to claim 1, wherein the training process of the image segmentation model and the visual feature extraction model includes: Input the training data into the initial model and obtain the training results output by the initial model; Based on the training results, the model parameters of the initial model are adjusted to obtain the trained model; The parameters of the encoding module of the trained model are the same as those of the encoding module of the initial model, while the parameters of the decoding module of the trained model are different from those of the decoding module of the initial model.

9. The method according to claim 1, wherein the decoding module of the image segmentation model decodes based on the target image features and the target visual features to obtain the text segmentation result of the target image, comprising: The decoding module is used to decode the target image features and the target visual features to generate an initial segmentation mask; The initial segmentation mask is then subjected to resolution enhancement and edge optimization processing to obtain the text segmentation result of the target image.

10. An electronic device, comprising: The image acquisition module is used to acquire the target image; At least one processor; Used for: The target image is encoded using the encoding module of the image segmentation model to obtain the target image features; The target image is processed using a visual feature extraction model to obtain target visual features; the target visual features characterize the visual attributes of multi-scale visual information in the target image. The decoding module of the image segmentation model decodes the target image based on the target image features and the target visual features to obtain the text segmentation result of the target image; the decoding module is trained to output the result of text segmentation from the image.

Images