Intelligent processing method, device and equipment for AIPC screen content and storage medium
By constructing a lightweight convolutional neural network model based on depthwise separable convolution and channel attention mechanisms, and combining edge detection and region growing algorithms, the problem of insufficient feature extraction in screen content processing by lightweight neural network models is solved, and real-time, efficient processing and structured reorganization are achieved on the AIPC.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-22
- Publication Date
- 2026-03-27
AI Technical Summary
Existing lightweight neural network models suffer from insufficient feature extraction when processing screen content, making it difficult to run in real time on resource-constrained AIPCs.
A lightweight convolutional neural network model based on depthwise separable convolution and channel attention mechanism is adopted to preprocess, extract features, segment and classify screen content. The segmentation boundary is optimized by combining edge detection algorithm and region growing algorithm, and a scene adaptive classification mechanism is introduced.
While reducing computational complexity and the number of parameters, it improves feature extraction capabilities, enabling real-time, efficient processing and structured recombination of screen content, and significantly improving segmentation accuracy and classification accuracy.
Smart Images
Figure CN121747142A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of screen content processing, specifically to an AIPC screen content intelligent processing method, apparatus, device, and storage medium. Background Technology
[0002] With the development of artificial intelligence and computer vision technologies, intelligent processing of screen content has become an important research direction for human-computer interaction and information extraction. Traditional screen content processing methods mainly rely on cloud servers for large-scale computing; however, large-scale computing via cloud servers suffers from problems such as network latency, privacy and security risks, and dependence on network connectivity. With the rise of AIPCs (AI Personal Computers), an increasing number of computing tasks need to be completed on local devices when performing intelligent processing of screen content, which places higher demands on the computing power of local devices.
[0003] Intelligent processing of screen content first requires extracting the screen content. Existing methods mainly include those based on traditional image processing techniques and those based on deep learning. Traditional image processing methods, such as OCR (Optical Character Recognition) and template matching, are relatively simple to implement, but they have poor adaptability to complex scenes and struggle to handle diverse screen content. While deep learning-based extraction methods offer high accuracy, they typically involve large models and high computational complexity, making them difficult to run in real-time on resource-constrained AIPCs.
[0004] While existing lightweight neural network models can reduce the computational complexity of deep learning to some extent, they still suffer from insufficient feature extraction when dealing with scenarios with complex structures and diversity, such as screen content. Summary of the Invention
[0005] The purpose of this application is to overcome the shortcomings of existing lightweight neural network models, which, although they can reduce the computational complexity of deep learning to a certain extent, still suffer from insufficient feature extraction when dealing with complex and diverse scenarios such as screen content.
[0006] Therefore, this application provides an intelligent processing method for AIPC screen content, including:
[0007] Acquire screen content and preprocess it;
[0008] A lightweight convolutional neural network model is used to extract features from the preprocessed screen content and output a feature map. The lightweight convolutional neural network model is a lightweight network model based on depthwise separable convolution and channel attention mechanism.
[0009] Multi-element segmentation and multi-element extraction of screen content based on feature maps;
[0010] The element classification output includes the classification results and the extracted element data.
[0011] Furthermore, the lightweight convolutional neural network model includes an input layer, a feature extraction module, a feature fusion module, and an output layer, wherein the feature extraction module includes several lightweight convolutional blocks.
[0012] Furthermore, the lightweight convolutional block contains, in sequence, a depth-separable convolutional layer, a batch normalization layer, an activation function, and a channel attention module.
[0013] Furthermore, the depth-separable convolutional layer fuses channel information through pointwise convolution.
[0014] Furthermore, the multi-element segmentation and multi-element extraction of screen content based on feature maps includes:
[0015] A feature map-based segmentation network generates an initial segmentation mask for coarse segmentation;
[0016] The region growing algorithm is used to optimize the segmentation boundary based on the edge detection algorithm.
[0017] Furthermore, the region growing algorithm uses edge pixels as seed points and determines whether to merge adjacent pixels by calculating the similarity between pixels.
[0018] Furthermore, after the step of performing multi-element segmentation and multi-element extraction of screen content based on feature maps, the method further includes:
[0019] The extracted elements are classified in an adaptive scenario manner to determine the content type and contextual relationship of each element.
[0020] This application also provides an AIPC screen content intelligent processing device, including:
[0021] The image acquisition module is used to acquire screen content;
[0022] A preprocessing module is used to preprocess the screen content;
[0023] The feature extraction module is used to extract features from the preprocessed screen content using a lightweight convolutional neural network model and output a feature map. The lightweight convolutional neural network model is a lightweight network model based on depthwise separable convolution and channel attention mechanism.
[0024] A segmentation module, which is used to perform multi-element segmentation and multi-element extraction of screen content based on feature maps;
[0025] The classification output module is used to classify elements and output the classification results and extracted element data.
[0026] This application also provides an AIPC screen content intelligent processing device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the aforementioned AIPC screen content intelligent processing method.
[0027] This application also provides a storage medium storing a program, which, when executed by a processor, is used to implement the AIPC screen content intelligent processing method as described above.
[0028] The technical solution of this application has at least the following advantages:
[0029] The AIPC screen content intelligent processing method of this application constructs a lightweight convolutional neural network model based on deep separable convolution and channel attention mechanism. While significantly reducing computational complexity and the number of parameters, it effectively maintains a high level of feature extraction capability. When processing screen content, which has a complex structure and diversity, it can fully extract features, which helps to process and restructure screen content in real time and efficiently on the AIPC with the highest resource availability. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0031] Figure 1 A flowchart illustrating the AIPC screen content intelligent processing method provided in this application embodiment;
[0032] Figure 2 This is a structural block diagram of the AIPC screen content intelligent processing device provided in the embodiments of this application;
[0033] Figure 3 This is a structural block diagram of the AIPC screen content intelligent processing device provided in an embodiment of this application. Detailed Implementation
[0034] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0035] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0036] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two components; and they can refer to a wireless connection or a wired connection. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.
[0037] Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.
[0038] Example 1
[0039] Example 1 provides an intelligent processing method for AIPC screen content. Figure 1 This is a flowchart illustrating an exemplary embodiment of the AIPC screen content intelligent processing method provided in this application. (Refer to...) Figure 1 AIPC screen content processing methods include:
[0040] S1. Obtain screen content and perform preprocessing;
[0041] In some embodiments, acquiring and preprocessing screen content includes:
[0042] The current screen content image is obtained through the screen capture API (Application Programming Interface), and then the current screen content image is preprocessed.
[0043] Specifically, the preprocessing of the current screen content image involves: scaling the image to 224×224 pixels using bilinear interpolation; normalizing pixel values to the [0,1] range using a normalization function; and converting the color space from RGB to YCrCb using a color space transformation matrix. This preprocessing enhances the robustness to brightness variations in the screen content image.
[0044] Specifically, the normalization function is:
[0045] Among them I norm (x,y) represents the normalized pixel value at coordinates (x,y), I scaled (x,y) represents the original pixel value at coordinates (x,y), I max I is the maximum value of all pixels in the entire image. min This represents the smallest value among all pixels in the entire image.
[0046] S2. A lightweight convolutional neural network model is used to extract features from the preprocessed screen content and output a feature map. The lightweight convolutional neural network model is a lightweight network model based on depthwise separable convolution and channel attention mechanism.
[0047] In some embodiments, the lightweight convolutional neural network model includes an input layer, a feature extraction module, a feature fusion module, and an output layer. The feature extraction module includes several lightweight convolutional blocks. Each lightweight convolutional block contains, in sequence, a depthwise separable convolutional layer, a batch normalization layer, an activation function, and a channel attention module.
[0048] Specifically, the feature extraction module consists of six lightweight convolutional blocks stacked together, and the activation function contained in the lightweight convolutional blocks is the ReLU activation function.
[0049] In some embodiments, depthwise separable convolutional layers fuse channel information through pointwise convolution, and their computational logic satisfies equation (1), which is:
[0050]
[0051] Among them, F point (i,j,l) represents the value of the l-th channel at position (i,j) in the pointwise convolution output feature map; C in Indicates the total number of input channels; W point (1,1,k,l) represents the pointwise convolution weights with a kernel size of 1×1; F depth (i,j,k) represents the feature map of the k-th channel output by the previous depthwise convolution; b l Let l represent the bias term of the l-th channel. The pointwise convolutional fusion of channel information achieved by equation (1) can significantly reduce the number of parameters in the convolutional neural network model.
[0052] Furthermore, the channel attention module is used to reweight the feature channels, and its calculation logic satisfies equation (2), which is:
[0053]
[0054] Among them, Fatt (i,j,c) represents the final output feature after attention weighting; σ represents the Sigmoid activation function; δ represents the ReLU activation function; W1 and W2 represent the weight matrices of the two fully connected layers, used to capture the dependencies between channels; F avg (c) represents the global average pooling eigenvalue of the c-th channel; F point (i,j,c) represents the numerical values of the input feature map.
[0055] The lightweight convolutional neural network model proposed in this application extracts features from the preprocessed screen content, thereby outputting a 256-channel high-bit feature map. Compared with the traditional convolutional neural network model, its computational cost is reduced by about 65%, effectively meeting the real-time requirements of smart terminal devices.
[0056] S3. Perform multi-element segmentation and multi-element extraction on screen content based on feature maps; specifically, multi-element refers to identifying text, images, tables, and code blocks in the screen content image. Multi-element segmentation and multi-element extraction on screen content based on feature maps means identifying and extracting text, images, tables, and code blocks from the screen image content.
[0057] Multi-element segmentation and extraction of screen content based on feature maps, including:
[0058] A segmentation network based on feature maps generates an initial segmentation mask for coarse segmentation, which determines the approximate region of each element.
[0059] The segmentation boundary is optimized using a region growing algorithm based on edge detection algorithms. Specifically, the Canny edge detection algorithm is introduced to extract high-frequency edge information, and a region growing algorithm is executed based on the high-frequency edge information to optimize the segmentation boundary.
[0060] In some embodiments, the region growing algorithm uses edge pixels as seed points and determines whether to merge adjacent pixels by calculating the similarity between pixels. The similarity calculation logic between pixels satisfies equation (3), which is:
[0061] Where S(p,q) represents the similarity weight between pixel p and its neighboring pixel q, and the similarity value ranges from (0,1]; I(p) and I(q) represent the image grayscale value (or feature intensity value) of pixel p and q, respectively; σ represents the standard deviation parameter of the Gaussian function, which is used to control the sensitivity of similarity to grayscale differences.
[0062] By optimizing the segmentation boundary using a region growing algorithm based on edge detection, high-precision element extraction can be achieved under complex screen layouts, with text IoU ≥ 93%, table IoU ≥ 91%, image IoU ≥ 95%, and code block IoU ≥ 92%.
[0063] S4. Classify elements and output the classification results and extracted element data;
[0064] Specifically, the element classification includes:
[0065] The extracted elements are classified in an adaptive scenario manner to determine the content type and contextual relationship of each element.
[0066] Specifically, scene-adaptive classification includes two stages: pre-classification and fine-grained classification. In the pre-classification stage, a lightweight classifier is used to calculate the initial probability of an element belonging to a major category (such as text, image, table, or code block) based on the visual features of the element. Subsequently, in the fine-grained classification stage, a context association mechanism is introduced to modify the initial results by combining the type semantics and spatial location information of adjacent elements to determine the specific sub-type of the element (such as subdividing text into title, body text, and list items; and subdividing images into charts, screenshots, and icons).
[0067] In some embodiments, the final classification confidence score is calculated using the following scene-adaptive scoring formula, which is shown in equation (4). Equation (4) is:
[0068] Where Score(t|F,C) represents the final score of feature F being classified as type t in context C; α is an adjustment factor balancing the weights of pre-classification and fine-classification; P(t|F) represents the initial probability value obtained based on visual features in the pre-classification stage; N represents the number of neighboring elements; Sim(t,t) n ) represents the relationship between the current element type t and its neighboring elements t n Semantic co-occurrence similarity between them; Dist(p,p) n ) represents the position of the current element p and the positions of its neighboring elements p. n The closer the distance between the two objects (Euclidean distances), the greater their influence on the classification result. Testing showed that this adaptive classification mechanism achieves a classification accuracy of ≥90% in complex screen scenarios, with an average confidence level of ≥0.85 for fine-grained classification results.
[0069] Furthermore, based on the determined content type and contextual relationship of each element, the classification results and extracted element data are output. The classification results (element type, confidence level, contextual relationship) and the extracted element data (coordinates, pixel information) are output in JSON / XML structured format, which can be used for subsequent information processing, content analysis or user interaction.
[0070] The aforementioned AIPC screen content intelligent processing method constructs a lightweight convolutional neural network model based on depthwise separable convolution and channel attention mechanisms. While significantly reducing computational complexity and the number of parameters, it effectively maintains a high level of feature extraction capability. When processing screen content, which has complex structure and diversity, it can fully extract features, which helps to achieve real-time, efficient processing and structured reorganization of screen content on the AIPC with the highest resource availability.
[0071] In addition, the AIPC screen content intelligent processing method of this application combines edge detection and region growing algorithms for multi-element segmentation extraction, which significantly improves the segmentation accuracy of different types of elements in complex screen layouts; and introduces a scene adaptive classification mechanism to further optimize classification accuracy by utilizing contextual association information.
[0072] In some embodiments, in order to ensure that the model can run efficiently on AIPCs with limited computing resources, the AIPC screen content intelligent processing method also includes a model compression step, which significantly reduces the model size and computational complexity through multi-dimensional optimization strategies.
[0073] Specifically, the model compression steps mainly include the following three aspects: First, knowledge distillation technology is used to select the high-performance ResNet50 as the teacher model to guide the training of the lightweight CNN student model, so that the small model can learn and inherit the feature extraction capabilities of the large model; Second, through pruning technology, based on the preset weight threshold judgment condition, redundant connections with low contribution in the convolutional neural network are eliminated to achieve sparsity of the convolutional neural network structure; Finally, an 8-bit quantization strategy is adopted to map the parameters of the convolutional neural model from 32-bit floating-point numbers to 8-bit integers, and its quantization precision conversion logic satisfies equation (5), which is:
[0074]
[0075] Among them, w int8 This represents the quantized 8-bit integer weight value; w float32 This represents the original 32-bit floating-point weight value; w max and w minThese represent the maximum and minimum values in the current weight tensor, used to determine the dynamic range of quantization; `round` represents the rounding function. After model compression, the final deployed lightweight convolutional neural network model size is compressed to ≤5MB, and the single-frame inference latency on mainstream AIPC hardware is less than 50ms, which can fully meet the performance indicators of real-time screen content processing. The above-mentioned intelligent processing method for AIPC screen content deeply optimizes the model size through model compression technology, fundamentally realizing real-time, efficient processing and structured reorganization of screen content on resource-constrained AIPCs.
[0076] Example 2
[0077] Example 2 provides an intelligent processing device for AIPC screen content. Figure 2 This is a structural block diagram of the AIPC screen content intelligent processing device provided in the embodiments of this application, with reference to... Figure 2 The AIPC screen content intelligent processing device includes an acquisition module 201, a preprocessing module 202, a feature extraction module 203, a segmentation module 204, and a classification output module 205. Specifically, the image acquisition module 201 acquires screen content; the preprocessing module 202 preprocesses the screen content; the feature extraction module 203 uses a lightweight convolutional neural network model to extract features from the preprocessed screen content and output a feature map, wherein the lightweight convolutional neural network model is a lightweight network model based on depthwise separable convolution and channel attention mechanisms; the segmentation module 204 performs multi-element segmentation and multi-element extraction on the screen content based on the feature map; and the classification output module 205 classifies the elements and outputs the classification results and extracted element data.
[0078] Furthermore, the classification output module 205 includes a classification module and a classification output module 205. The classification module is used to perform scene-adaptive classification on the extracted elements and determine the content type and contextual relationship of each element. The classification output module 205 is used to output the classification results and the extracted element data.
[0079] Example 3
[0080] Example 3 provides an AIPC screen content intelligent processing device. Figure 3 This is a structural block diagram of the AIPC screen content intelligent processing device provided in an embodiment of this application. (Refer to...) Figure 3 The AIPC screen content intelligent processing device includes a processor 310 and a memory 320.
[0081] Processor 310 may include one or more processing cores. Processor 310 connects to various parts of the device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 320, and by calling data stored in memory 320. Optionally, processor 310 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 310 may integrate one or more of a Central Processing Unit (CPU) and a modem. The CPU primarily handles the operating system and applications; the modem is used for wireless communication. It is understood that the modem may also not be integrated into processor 310 and may be implemented as a separate chip.
[0082] Optionally, when the processor 310 executes the program instructions in the memory 320, it implements the AIPC screen content intelligent processing method provided in the above-described method embodiments.
[0083] The memory 320 may include random access memory (RAM) or read-only memory. Optionally, the memory 320 may include a non-transitory computer-readable storage medium. The memory 320 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 320 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created according to the use of the device, etc.
[0084] Optionally, this application also provides a storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the AIPC screen content intelligent processing method provided in the above method embodiments.
[0085] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this application.
Claims
1. A method for intelligent processing of AIPC screen content, characterized in that, include: Acquire screen content and preprocess it; A lightweight convolutional neural network model is used to extract features from the preprocessed screen content and output a feature map. The lightweight convolutional neural network model is a lightweight network model based on depthwise separable convolution and channel attention mechanism. Multi-element segmentation and multi-element extraction of screen content based on feature maps; Classify elements and output the classification results and extracted element data.
2. The AIPC screen content intelligent processing method according to claim 1, characterized in that, The lightweight convolutional neural network model includes an input layer, a feature extraction module, a feature fusion module, and an output layer. The feature extraction module includes several lightweight convolutional blocks.
3. The AIPC screen content intelligent processing method according to claim 2, characterized in that, The lightweight convolutional block contains, in sequence, a depth-separable convolutional layer, a batch normalization layer, an activation function, and a channel attention module.
4. The AIPC screen content intelligent processing method according to claim 1, characterized in that, The depth-separable convolutional layer fuses channel information through pointwise convolution.
5. The AIPC screen content intelligent processing method according to claim 1, characterized in that, The multi-element segmentation and multi-element extraction of screen content based on feature maps includes: A feature map-based segmentation network generates an initial segmentation mask for coarse segmentation; The region growing algorithm is used to optimize the segmentation boundary based on the edge detection algorithm.
6. The AIPC screen content intelligent processing method according to claim 5, characterized in that, The region growing algorithm uses edge pixels as seed points and determines whether to merge adjacent pixels by calculating the similarity between pixels.
7. The AIPC screen content intelligent processing method according to claim 1, characterized in that, The element classification includes: The extracted elements are classified in an adaptive scenario manner to determine the content type and contextual relationship of each element.
8. An AIPC screen content intelligent processing device, characterized in that, include: The image acquisition module is used to acquire screen content; A preprocessing module is used to preprocess the screen content; The feature extraction module is used to extract features from the preprocessed screen content using a lightweight convolutional neural network model and output a feature map. The lightweight convolutional neural network model is a lightweight network model based on depthwise separable convolution and channel attention mechanism. A segmentation module, which is used to perform multi-element segmentation and multi-element extraction of screen content based on feature maps; The classification output module is used to classify elements and output the classification results and extracted element data.
9. An AIPC screen content intelligent processing device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the AIPC screen content intelligent processing method according to any one of claims 1-7.
10. A storage medium storing a program, which, when executed by a processor, is used to implement the AIPC screen content intelligent processing method according to any one of claims 1-7.