Image processing methods, systems, electronic devices, and storage media

Abstract

This invention relates to image processing methods, systems, electronic devices, and storage media, belonging to the field of computer technology. The method includes: extracting features from an input target image through multi-scale branches to obtain first information, wherein the first information includes feature information at multiple scales; calculating the feature information at each scale and the inter-scale correlation information in the first information using an attention mechanism to obtain second information; and performing prediction output operations on the second information to obtain category data and an embedded representation of the target image. This invention improves the ability to express fine-grained features in images during image processing, thereby enhancing accuracy and robustness in tasks such as image classification, target detection, recognition, and judgment in complex scenes or with high precision requirements.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more particularly to an image processing method, system, electronic device, and storage medium. Background Technology

[0002] With the development of computer vision technology, image processing-based intelligent recognition and analysis technologies have been widely applied in fields such as target detection, target recognition, image classification, scene understanding, medical image analysis, and industrial inspection. Most existing image processing technologies rely on neural network models to automatically extract and represent features in images in order to recognize and judge image content.

[0003] Typical tasks or scenarios in the field of digital pathology analysis include the detection and classification of cells / glands in tissue sections, such as the Gleason grading of prostate cancer. Current mainstream methods include object detection methods based on Convolutional Neural Networks (CNNs) (such as Faster R-CNN and YOLO), methods based on U-Net-like segmentation networks, and methods combining traditional methods (edge ​​detection, thresholding, and morphological manipulation, etc.) with machine learning.

[0004] In the field of industrial quality inspection, typical tasks / scenarios include detecting defects in products with complex textures (crack detection in fabrics, steel, etc.). Current mainstream technologies primarily consist of traditional image processing (such as thresholding and filtering) and CNN-based image processing.

[0005] For example, in the field of autonomous driving, typical tasks / scenarios include traffic scene understanding (small object detection, such as pedestrians and signs). Existing mainstream technologies typically include single-stage / two-stage detectors, Feature Pyramid Networks (FPNs), and Transformers.

[0006] As explained above, CNNs are the most commonly used method. CNNs abstract features such as edges, textures, and shapes in images layer by layer through local receptive fields and parameter sharing mechanisms, achieving good results in various visual tasks. However, this type of model mainly extracts features based on fixed convolutional kernels. Its scope of attention and feature response patterns are basically fixed after training, making it difficult to dynamically adjust local features according to the overall semantics or contextual information of the current image content. Especially when it is necessary to distinguish subtle differences or complex backgrounds, its feature representation ability is somewhat limited.

[0007] To enhance the model's ability to handle complex scenes, existing technologies have proposed Multi-Scale Feature Fusion Networks (FPNs) and their variants. These methods fuse feature maps from different levels to balance low-level detail and high-level semantic information in images. However, multi-scale fusion typically employs predefined structures or weighting methods, resulting in relatively static fusion relationships between features at different scales. This makes it difficult to adaptively adjust for semantic differences in different images or regions, leading to feature redundancy or weakening of key information in fine-grained feature recognition tasks.

[0008] Furthermore, in recent years, attention mechanisms have been introduced into image processing models to enhance the network's ability to focus on important regions or channels. For example, channel attention, spatial attention, and their combinations can highlight task-related features to a certain extent. However, existing attention mechanisms typically generate weights based on local feature statistics or simple global pooling operations, which limits their ability to model the overall semantic structure and contextual relationships of images, making it difficult to accurately characterize the importance of different fine-grained features in the global context in complex scenes.

[0009] In recent years, Transformer-based models have seen increasing applications. Replacing traditional convolution with a Transformer encoder-decoder structure, the input image is segmented into fixed-size patches. Each patch is linearly projected as a feature vector, and the final image representation is obtained through global average pooling of these feature vectors, enabling tasks such as object detection, segmentation, or classification in a specified scene. Unlike the local convolution operations of traditional CNNs, each Transformer encoder layer relies on a multi-head self-attention (MSA) mechanism. In this mechanism, each feature token needs to calculate attention weights with all other tokens in the sequence. This means that even when processing the smallest local features in an image, the computation process forcibly associates all distant and potentially unrelated regions in the image. While this "full-image view" helps capture long-range dependencies, it results in significant redundant computation for many tasks that only require local context. Furthermore, when processing high-resolution images or detailed visual tasks, it often faces problems of high computational complexity and insufficient ability to model local details, making it difficult to achieve dynamic feature enhancement for specific image content. Summary of the Invention

[0010] To address the technical problems existing in the prior art, this invention proposes an image processing method, system, electronic device, and storage medium, which improves the accuracy and robustness of tasks such as image classification, target detection, recognition, and judgment by enhancing the expressive power of fine-grained image features and strengthening local features.

[0011] To address the aforementioned technical problems, according to one aspect of the present invention, an image processing method is provided. When a target image is input to a trained image processing system, the image processing method performed by the image processing system on the target image includes:

[0012] The first information is obtained by extracting features from the input target image through multi-scale branches, wherein the first information includes feature information at multiple scales.

[0013] The second information is obtained by dynamically weighting the feature information at each scale and the inter-scale correlation information in the first information using an attention mechanism; and

[0014] The second information is used to perform prediction output operations to obtain the category data and embedded representation of the target image.

[0015] Optionally, when extracting features from the input target image through multi-scale branches to obtain the first information, a preset number of feature iterations are performed to extract the first information.

[0016] Optionally, the step of dynamically weighting the feature information at each scale and the correlation information between scales in the first information using an attention mechanism to obtain the second information includes:

[0017] The first piece of information is divided into multiple batches, and the total number of batches is determined based on the following formula: Where N is the total number of batch messages. To measure the number of branches, To be responsible for the batch size under a single scale branch. The interval distance between scale branches. The scale branch interval distance is The corresponding association weight coefficient at that time, ( Indicates in In each scale branch, the interval distance is exactly... The number of scale branch pairs;

[0018] Attention is calculated for each batch of information to obtain the corresponding attention calculation result;

[0019] The attention calculation results of all batch information are concatenated to obtain the attention calculation result concatenation information;

[0020] The second information is obtained by summing the attention calculation result, the spliced ​​information, and the first information.

[0021] Optionally, the step of performing prediction output operations on the second information to obtain the category data and embedded representation of the target image includes:

[0022] The second information is sequentially processed by fully connected layer computation and first activation function computation to obtain a first embedded representation of a first preset dimension; and

[0023] The second information is sequentially processed by the first fully connected layer calculation, the second activation function calculation, and the second fully connected layer calculation to obtain the second embedded representation of the second preset dimension, where the number of the second preset dimension corresponds to the number of classification categories.

[0024] Wherein, the first embedded representation serves as the embedded representation of the target image; the second embedded representation serves as the category data of the target image.

[0025] According to another aspect of the present invention, the present invention also provides an image processing system based on a neural network, comprising:

[0026] The backbone feature extraction network is configured to extract features from the input target image through multi-scale branches to obtain first information, wherein the first information includes feature information at multiple scales.

[0027] A feature processing network is configured to dynamically weight and calculate the feature information at each scale and the correlation information between scales in the first information using an attention mechanism to obtain the second information; and

[0028] The prediction output network is configured to perform prediction output operations on the second information to obtain the category data and embedded representation of the target image.

[0029] Optionally, the backbone feature extraction network includes a forward path and a retrieval path. In the forward path, features are extracted from the input target image through multi-scale branches. The extracted feature information is fed back to the forward path through the retrieval path for multi-scale feature iteration extraction with a preset number of iterations to obtain the first information. Alternatively, the backbone feature extraction network includes multi-level cascaded feature extraction units, with the last-level feature extraction unit outputting the first information.

[0030] Optionally, the forward path of the backbone feature extraction network includes:

[0031] The initial feature extraction module is configured to perform initial feature extraction on the input target image to convert image information in the spatial domain to image information in the channel domain.

[0032] Multiple convolutional branch modules are configured to use convolutional kernels of corresponding scales to extract features from the target information to obtain feature information at multiple scales;

[0033] The feature stitching module is configured to stitch together feature information from multiple scales to obtain feature stitching information; and

[0034] The average pooling module is configured to perform average pooling on the feature concatenation information to obtain average pooling information;

[0035] In the initial forward path processing, the target information is the image information converted to the channel domain by the initial feature extraction module. In the feature iterative extraction process, the target information is the average pooling information output by the average pooling module passed through the recycling path.

[0036] Optionally, the forward path of the backbone feature extraction network further includes:

[0037] The original feature extraction module is configured to extract features from the target information through residual branching to obtain the original feature information of the target information; correspondingly, the feature concatenation module concatenates the feature information at multiple scales to obtain intermediate concatenation information; and

[0038] The accumulation module is configured to accumulate the intermediate splicing information and the original feature information output by the residual branch original feature extraction module to obtain feature splicing information;

[0039] Correspondingly, the average pooling module performs an average pooling operation on the feature splicing information output by the accumulation module to obtain average pooling information.

[0040] Optionally, the forward path of the backbone feature extraction network further includes:

[0041] The feature enhancement module is configured to sequentially perform normalization, activation function calculation, and max pooling operations on the received target information to enhance its features, and then send the feature-enhanced target information to multiple convolutional branch modules; and / or

[0042] Multiple BR modules, each corresponding to a convolutional branch module, are used to perform normalization and nonlinear mapping operations on the feature information extracted by the convolutional branch modules in sequence; and / or

[0043] The BN module, corresponding to the original feature extraction module, is used to perform batch normalization on the original feature information extracted by the original feature extraction module.

[0044] Optionally, the feature processing network includes:

[0045] The data segmentation module is configured to segment the first information output by the backbone feature extraction network into multiple batches; the total number of batches is determined based on the following formula: Where N is the total number of batch messages. To measure the number of branches, To be responsible for the batch size under a single scale branch. The interval distance between scale branches. The scale branch interval distance is The corresponding correlation weight coefficient, Indicates in In each scale branch, the interval distance is exactly... The number of scale branch pairs;

[0046] Multiple attention calculation modules are used to perform attention calculations on each batch of information to obtain the corresponding calculation results;

[0047] The stitching module is configured to stitch together the calculation results output by each attention calculation module to obtain the attention calculation stitched result; and

[0048] The dynamic weighted calculation module is configured to accumulate the first information and the attention calculation result output by the splicing module to obtain the second information used for predictive output processing.

[0049] Optionally, the feature processing network further includes a dimension mapping module, configured to perform preset dimension mapping processing on the first information, and input the dimension-mapped first information to the data segmentation module; correspondingly, the first information calculated by the dynamic weighted calculation module is the first information output by the dimension mapping module.

[0050] Optionally, the prediction output network includes:

[0051] The vector output module is configured to sequentially compute the second information through a fully connected layer and a first activation function to obtain a first embedded representation of a first preset dimension; and

[0052] The category output module is configured to calculate the second embedded representation of the second information by sequentially passing it through a fully connected layer and a second activation function, wherein the number of the second preset dimensions corresponds to the number of classification categories.

[0053] Optionally, the category output module further performs a second fully connected layer calculation on the second embedded representation of the second preset dimension, and the result of the second fully connected layer calculation is used as the category data of the target image.

[0054] According to another aspect of the present invention, an electronic device is also provided, comprising a processor and a memory, wherein a set of computer program instructions is stored in the memory, and the aforementioned image processing method or the aforementioned image processing system is executed when the processor executes the set of computer program instructions in the memory.

[0055] According to another aspect of the present invention, the present invention also provides a computer-readable storage medium storing a computer program instruction set thereon, which, when executed by a processor, performs the aforementioned image processing method or implements the aforementioned image processing system.

[0056] According to another aspect of the present invention, the present invention also provides a computer program product comprising a computer program instruction set, which is executed by a processor to perform the aforementioned image processing method or to implement the aforementioned image processing system.

[0057] In summary, this invention improves the ability to express fine-grained features in images during image processing, and dynamically adjusts and enhances local features by combining the global context information of the image, unaffected by background interference or scale changes. This improves accuracy and robustness in complex scenes or high-precision tasks such as image classification, target detection, recognition, and judgment. Attached Figure Description

[0058] The preferred embodiments of the present invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0059] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of a neural network-based image processing system according to an embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of the backbone feature extraction network 1 according to an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of the backbone feature extraction network 1 according to another embodiment of the present invention;

[0063] Figure 5 This is a schematic diagram of the feature extraction unit 11 according to an embodiment of the present invention;

[0064] Figure 6 This is a schematic diagram of the feature extraction unit 11 according to another embodiment of the present invention;

[0065] Figure 7 This is a schematic diagram of a feature processing network 2 according to an embodiment of the present invention;

[0066] Figure 8 This is a block diagram of the predictive output network 3 according to an embodiment of the present invention;

[0067] Figure 9This is a flowchart of a training method for a neural network-based image processing system according to an embodiment of the present invention;

[0068] Figure 10 This is a statistical data matrix heatmap applied when determining positive and negative samples according to an embodiment of the present invention;

[0069] Figure 11 This is a block diagram illustrating the principle of a training system for a neural network-based image processing system according to an embodiment of the present invention; and

[0070] Figure 12 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] In the following detailed description, reference can be made to the accompanying drawings, which form part of this application and illustrate specific embodiments of the present application. In the drawings, similar reference numerals describe substantially similar components in different figures. Specific embodiments of the present application are described in sufficient detail below to enable those skilled in the art to implement the technical solutions of the present application. It should be understood that other embodiments may also be utilized, or structural, logical, or electrical changes may be made to the embodiments of the present application.

[0073] This invention provides an image processing method and system, wherein the image processing system processes an input image using a neural network model. When a target image is input to the trained image processing system, the image processing method performed by the system on the target image is described below. Figure 1 , Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention. The method includes the following steps:

[0074] Step S1: Extract features from the input target image through multi-scale branches to obtain first information, wherein the first information includes feature information at multiple scales.

[0075] Step S2: The second information is obtained by calculating the feature information of each scale and the correlation information between scales in the first information through an attention mechanism.

[0076] Step S3: Perform prediction output operations on the second information to obtain the category data and embedded representation of the target image.

[0077] In step S1, when feature extraction is performed on the input target image through multiple scale branches to obtain the first information, a preset number of feature iterations are performed to obtain the first information, thereby enhancing the extracted features in the first information. One way to achieve the preset number of feature iterations is to set a preset number of cascaded feature extraction units. Each feature extraction unit includes multiple scale branches, and different feature extraction units extract features at different levels. The feature information output by the last feature extraction unit is used as the first information. In another embodiment, a feature extraction unit and a retrieval path are set. The feature information extracted by the feature extraction unit is returned to the input of the feature extraction unit through the retrieval path for a second feature extraction. After a preset number of iterative feature extractions, the feature information output by the feature extraction unit is the first information. Regardless of the method, during each feature extraction, features are extracted from the input image through multiple parallel branches of different scales, and then the feature information extracted from each scale branch is concatenated. Therefore, this invention not only extracts feature information at different levels through a multi-level feature extraction process, but also captures and extracts image details at different scales through feature extraction by multiple parallel branches of different scales at each level.

[0078] In one embodiment, to prevent feature loss due to multiple feature extractions and abstractions, in a better embodiment, after each feature extraction of the input image through multiple branches at different scales, the original features are extracted from the input image through a residual branch with a 1×1 convolution kernel, and then accumulated with the concatenation result of the feature information extracted by the multi-scale branches. Let the input features of the original image be... The aforementioned process can be represented by the following equation (1-1):

[0079] (1-1)

[0080] in, This represents a sequential splicing operation performed within a specified dimension. This represents the computation process of the convolution kernel, where F is the input target information. These represent convolution operations at different scales. This represents the residual branch that ensures the stability of the network's gradient learning. These are the learnable parameters in the network. After multiple iterative operations, the features are fused. This process condenses the image features at different scales under each branch.

[0081] In step S2, when calculating the feature information at each scale and the correlation information between scales in the first information using an attention mechanism to obtain the second information, the first information is first divided into multiple batches of information ( ), wherein the scale is determined based on the number of feature information in the first information. The total quantity. For example, determined by the following formula (1-2). Total number N:

[0082] (1-2)

[0083] Where N is Total quantity To measure the number of branches, To be responsible for the batch size under a single scale branch. The interval distance between scale branches. The scale branch interval distance is The corresponding association weight coefficient is used to control whether the distance is within a certain range. Establish cross-scale correlation information between scale branches. Indicates in In each scale branch, the interval distance is exactly... The number of scale branch pairs.

[0084] Then, for each one... Attention calculations are performed to obtain the corresponding attention calculation results. Specifically, the following formulas (1-3) are used for each... Perform attention calculations:

[0085] (1-3)

[0086] in, Represents the i-th Attention calculation results They are respectively passed through 1×1 convolution kernels The result obtained after performing the calculation. For the channel dimension within a single patch. Each Each has an independent self-attention head to perform the above calculations to obtain independent calculation results.

[0087] Then, the attention calculation results of all patches are concatenated. Specifically, the attention calculation results of all batch information are concatenated using the following equation (1-4):

[0088] (1-4)

[0089] The representative will have all of Merge sequentially along the specified dimensions.

[0090] In one embodiment, assuming , That is, when using three scale branches, each scale branch is assigned two batches; let If we only focus on the association of continuous scale branches, then the batch size N = 3×2 + 0×(3-0) + 1×(3-1) + 0×(3-2) = 8.

[0091] When performing calculations using an attention mechanism, eight independent attention heads are set up for attention calculations. The attention calculation result represents the first-scale branch (such as a 1×1 convolution kernel). The attention calculation results represent the second-scale branch (such as a 3×3 convolution kernel). The attention calculation results represent the third-scale branch (such as a 5×5 convolution kernel). The results of the correlation attention calculation for the first and second scale branches are shown. The results of the correlation attention calculation represent the second-scale branch and the third-scale branch.

[0092] Assuming the first information After dividing it into 8 patches, each ,but, Attention statistics after merging .

[0093] In one embodiment, the merged attention statistics This can be used as the second piece of information, D2, for predicting the output. Even better, the first piece of information, D1, can be combined with the attention statistics using formula (1-5). The two are combined to generate a second piece of information, D2, for prediction.

[0094] (1-5)

[0095] In step S3, the second information D2 is sequentially processed by fully connected layer calculation and first activation function calculation to obtain the first embedded representation of the first preset dimension. The specific process can be represented by the following formulas (1-6):

[0096] (1-6)

[0097] in, This represents a first embedded representation, which serves as an embedded representation of the target image. In one embodiment, the embedded representation of the target image is called embedding. , This indicates the number of the first preset dimensions. This indicates fully connected layer computation, primarily used for dimensional mapping and transformation. As an activation function, it can retain more detailed information and is therefore used in obtaining high-dimensional embeddings.

[0098] The second information D2 is sequentially processed through a first fully connected layer calculation, a second activation function calculation, and a second fully connected layer calculation to obtain a second embedded representation with a second preset dimension. The number of the second preset dimensions corresponds to the number of classification categories. The specific process can be represented by the following formula (1-7):

[0099] (1-7)

[0100] The second embedded representation is a vector with a second preset dimension, where each dimension corresponds to a classification category, serving as the category data for the target image. In one embodiment, . This indicates the number of the second preset dimension. This is the activation function. In this embodiment, it is used to calculate the classification category of the image. At that time, two fully connected layer calculations were performed through two fully connected layers, thereby mitigating the problem of detail loss caused by large dimensional transformations.

[0101] In another aspect, the present invention also provides an image processing system based on a neural network, see [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of a neural network-based image processing system according to an embodiment of the present invention. The image processing system includes a backbone feature extraction network 1, a feature processing network 2, and a prediction output network 3. The backbone feature extraction network 1 is configured to extract features from the input target image through parallel feature extraction branches at multiple scales (hereinafter referred to as scale branches) to obtain first information D1. The feature processing network 2 is configured to calculate the feature information at each scale and the inter-scale correlation information in the first information through an attention mechanism to obtain second information D2. The prediction output network 3 is configured to perform prediction output operations on the second information to obtain category data of the target image. Embedded means embedded.

[0102] See Figure 3 , Figure 3This is a schematic diagram of the backbone feature extraction network 1 according to an embodiment of the present invention. In this embodiment, the backbone feature extraction network 1 includes multiple cascaded feature extraction units, such as the first feature extraction unit 111, the second feature extraction unit 112, and the Mth feature extraction unit 11M shown in the figure. The final Mth feature extraction unit 11M outputs first information D1. Each feature extraction unit includes multiple parallel scale branches. By performing multi-level feature extraction operations on the input image D, the image processing system can gradually construct high-level abstract semantic features from low-level local features, thereby improving the ability to express complex patterns and the accuracy of target task recognition.

[0103] See Figure 4 , Figure 4 This is a schematic diagram of the backbone feature extraction network 1 according to another embodiment of the present invention. In this embodiment, the backbone feature extraction network 1 includes a forward path (solid line path in the figure) and a return path (dashed line path in the figure). The forward path includes a feature extraction unit 11, which includes multiple parallel scale branches. The forward path extracts features from the input target image through multiple scale branches respectively. The extracted feature information is fed back to the iterative input of the feature extraction unit 11 in the forward path through the return path, and multi-scale feature iterative extraction is performed for a preset number of iterations to obtain the first information D1.

[0104] See Figure 5 , Figure 5 This is a block diagram illustrating the principle of a feature extraction unit 11 according to an embodiment of the present invention. The feature extraction unit 11 in this embodiment includes an initial feature extraction module 110, a convolutional branch module 120, a feature concatenation module 130, and an average pooling module 140. The initial feature extraction module 110 performs initial feature extraction on the input target image to convert spatial domain image information to channel domain image information, and sends this information to the convolutional branch module 120. The convolutional branch module 120 includes multiple parallel convolutional branches of different scales. Each convolutional branch uses a convolutional kernel of the corresponding scale to extract features from the image information sent by the initial feature extraction module 110 to obtain feature information of the corresponding scale, and sends this information to the feature concatenation module 130. The feature concatenation module 130 receives the feature information of the corresponding scale sent by each convolutional branch and concatenates the feature information of the multiple scales to obtain feature concatenation information. The average pooling module 140 performs an average pooling operation on the feature concatenation information to obtain average pooling information.

[0105] In this invention, the convolutional branch module 120 serves as the input end of the consolidation path in the forward path. The information input to the convolutional branch module 120 is referred to as target information. In the initial forward path processing, the target information is the image information converted to the channel domain by the initial feature extraction module 110. In the feature iterative extraction process, the target information is the average pooling information output by the average pooling module passed through the consolidation path.

[0106] In one embodiment, Figure 3 The structure of the first feature extraction unit 111 in the middle can be the same as Figure 5 The structure of the feature extraction unit 11 is the same as that of the second feature extraction unit 112 to the Mth feature extraction unit 11M, and their structures respectively include: Figure 5 The convolutional branch module 120, feature concatenation module 130, and average pooling module 140 are included.

[0107] See Figure 6 , Figure 6 This is a schematic diagram of the feature extraction unit 11 according to another embodiment of the present invention. This embodiment... Figure 5 In the illustrated structure, a feature enhancement module 150 is added between the initial feature extraction module 110 and the convolutional branch module 120 to enhance the initial features extracted by the initial feature extraction module 110. For example, in one embodiment, the feature enhancement module 150 sequentially performs normalization, activation function calculation, and max pooling. Normalization eliminates dimensional differences, stabilizes data distribution, mitigates gradient vanishing / exploding, accelerates training, and allows for larger learning rates. Activation function calculation, such as ReLU, increases non-linear expressive power, enabling the network to fit complex functions. Finally, max pooling achieves dimensionality reduction and feature enhancement.

[0108] Optionally, in this embodiment, each convolutional branch in the convolutional branch module 120 further includes a corresponding BR module 160, which is used to perform normalization and nonlinear mapping operations on the feature information extracted by the convolutional branch module in sequence.

[0109] In neural network structures, gradient vanishing may occur as the number of network layers increases. Therefore, the convolution branch module 120 of this invention also includes a residual branch. The residual branch is used to extract features from the target information to obtain the original feature information of the target information, thereby preserving the original features of the original image. Then, the operation result is normalized by the BN module 170.

[0110] Then, the intermediate splicing information and the original feature information of the residual branch calculated by the feature splicing module 130 are accumulated by the accumulation module 180 to obtain the feature splicing information.

[0111] The calculation process of the main module in the aforementioned feature extraction unit 11 can be represented by formula (1-1). For details, please refer to the aforementioned method description section, which will not be repeated here.

[0112] See Figure 7 , Figure 7 This is a schematic diagram of a feature processing network 2 according to an embodiment of the present invention. The feature processing network 2 includes a data segmentation module 21, multiple attention calculation modules 22, a concatenation module 23, and a dynamic weighted calculation module 24. The data segmentation module 21 is used to segment the first information D1 output by the backbone feature extraction network 1 into multiple batch information; wherein, the data segmentation module 21 can use formula (1-2) to determine the total number N of batch information, for details please refer to the aforementioned method description section, which will not be repeated here.

[0113] Each attention calculation module 22 performs attention calculations on each batch of information to obtain the corresponding calculation results. Attention is calculated using formula (1-3), which can be found in the method description section above and will not be repeated here.

[0114] The splicing module 23 will combine the calculation results output by each attention calculation module. The splicing is performed to obtain the attention calculation splicing result.

[0115] The dynamic weighted calculation module 24 accumulates the first information D1 and the attention calculation splicing result output by the splicing module 23 to obtain the second information for predicting output processing, as shown in formula (1-5) in the method section.

[0116] In addition, the feature processing network 2 also includes a dimension mapping module, which performs preset dimension mapping processing on the first information D1 and inputs the first information D1 after dimension mapping processing to the data segmentation module; correspondingly, the first information D1 applied by the dynamic weighted calculation module is the first information output by the dimension mapping module.

[0117] See Figure 8 , Figure 8 This is a block diagram of a prediction output network 3 according to an embodiment of the present invention. The prediction output network includes a vector output module 31 and a category output module 32. The vector output module 31 calculates the first embedded representation of the first preset dimension for the second information D2 through a fully connected layer and a first activation function, for example, by using formula (1-6) to calculate the first embedded representation of the first preset dimension. It is output as an embedded representation of the target image.

[0118] The category output module 32 calculates the second embedded representation of the second information D2 sequentially through a fully connected layer and a second activation function to obtain a second embedded representation of the second preset dimension. The number of the second preset dimensions corresponds to the number of classification categories. In a better embodiment, the category output module 32 further performs a second fully connected layer calculation on the second embedded representation of the second preset dimension, and the result of the second fully connected layer calculation is used as the category data of the target image. Specifically, it can be calculated using the following formulas (1-7).

[0119] In a more specific embodiment Figure 2 The image processing system described herein can be implemented as a neural network model. After training driven by training data, the image processing system can improve the expressive power of fine-grained image features and enhance local features, thereby improving the accuracy and robustness of tasks such as image classification, object detection, recognition, and judgment. Compared with existing technologies, this invention achieves a shift from local multi-scale feature fusion to adaptive scale feature fusion based on global semantic context. This enables the neural network model to have stronger discriminative power and robustness when dealing with complex visual tasks involving object variability, scale diversity, and contextual dependence.

[0120] See Figure 9 , Figure 9 This is a flowchart of a training method for a neural network-based image processing system according to an embodiment of the present invention, including:

[0121] Step S11: Construct a learning sample set, which includes multiple category sample subsets. Each category sample subset includes anchor samples, positive anchor samples, and negative anchor samples. The positive anchor samples of each category are samples that are easily identified as belonging to one or more of the remaining categories within the same category. The negative anchor samples of each category are samples that are easily identified as belonging to the category of the anchor sample within one or more of the remaining categories. Each sample includes a corresponding label category.

[0122] In order to identify subtle differences in images, this invention employs a contrastive learning method to train a neural network model. The strategy for determining positive samples is to identify samples in the same category that are easily misclassified as other categories as positive samples of that category, with the aim of reducing the embedding distance between easily misclassified samples. The strategy for determining negative samples is to identify samples in different categories that are easily misclassified as anchor samples as negative samples of each other, with the aim of increasing the distance between easily confused samples in different categories.

[0123] For example, in one embodiment of the present invention, the image processing task includes five categories, denoted as tC, aC, tP, aP, and NS. For each of these five categories, an anchor sample set, a positive sample set, and a negative sample set are constructed. Each anchor sample, along with one positive sample and one negative sample, forms a sample triplet. Then, the positive and negative sample sets are determined. To determine the corresponding positive and negative sample sets for each category's anchor sample set, each anchor sample of each category is first input into a neural grid model trained without using a contrastive learning strategy to obtain the corresponding predicted category. Then, a statistical data matrix heatmap is constructed based on the actual labeled categories and the predicted categories (see [reference]). Figure 10 , Figure 10 This is a statistical data matrix heatmap applied to determine positive and negative samples according to an embodiment of the present invention. Different colors in the graph represent different sample numbers, as shown in the legend on the right side of the matrix graph. The vertical axis of the matrix heatmap is the actual labeled category of the sample, and the horizontal axis is the predicted category obtained by the neural network model without using the contrastive learning strategy. As can be seen from the graph, a sample corresponds to one labeled category and one predicted category. The boxes in the diagonal represent samples whose predicted category is the same as the actual labeled category, that is, samples accurately predicted by the current conventional neural grid model, and are also easy to predict. When determining the set of positive and negative samples for a category, taking category tP as an example, according to the principle that samples in its category that are easily judged as other categories are its positive samples, the samples in the four squares in the first to fourth columns of the bottom row can all be considered as its positive samples. According to the principle that samples in the other four categories that are easily judged as category tP are its negative samples, the samples in the four squares from the second row from the bottom to the top row of the fifth column can all be considered as its negative samples. Furthermore, based on experience, the two most difficult-to-distinguish or most easily confused categories can be used as comparison categories to determine positive and negative samples. For example, in this embodiment, the tP and aP categories are comparison categories; therefore, these two categories are each other's designated categories for determining positive and negative samples. (Refer to...) Figure 9 The samples in the red squares of the second column of the bottom row (true label is category tP, predicted result is category aP) are taken as positive samples of category tP and negative samples of category aP, respectively. Similarly, the samples in the blue squares of the fifth column of the second row from the top (true label is category aP, predicted result is category tP) are taken as negative samples of category tP and positive samples of category aP, respectively. This method is used to obtain the positive and negative sample sets for other categories.

[0124] Step S12: Divide the learning samples into multiple batches. Each batch includes a certain number of anchor samples, positive samples, and negative samples.

[0125] Step S13: Initialize the trainable parameters in the neural network model. The initialization method can be random initialization or initialization based on a preset distribution.

[0126] Step S14, forward propagation computation. Specifically, a batch of training samples is input into the neural network-based image processing system (hereinafter referred to as the neural network model). The neural network model performs a forward propagation operation through layer-by-layer computation to obtain the prediction output for each sample. The prediction output includes the embedding of the training sample and the predicted category. .

[0127] Step S15: Calculate the contrastive learning loss based on the loss function. Specifically, for each learning sample in the training batch, construct a triplet sample with each anchor sample, and calculate the contrastive learning loss value. The contrastive learning loss value is a weighted sum of the class loss value and the embedded representation distance loss value. The learning loss function includes a class loss calculation term and an embedded representation distance calculation term. The class loss calculation term is used to calculate the cross-entropy loss between the sample label class and the predicted class, and the embedded representation distance calculation term is used to calculate the sum of the embedded representation distances of the anchor sample and its positive samples and the embedded representation distances of the anchor sample and its negative samples.

[0128] In one embodiment, the contrastive learning loss is calculated according to formulas (2-1) to (2-4). .

[0129] (2-1)

[0130] in, To compare the learning loss values; This represents the category loss value. To embed the distance loss value, and They are respectively and The weight parameters.

[0131] The category loss value This is the average of the cross-entropy loss values ​​for the label class and the predicted class of all samples in the batch. For example, it can be calculated using the following equation (2-2):

[0132] (2-2)

[0133] in, For the first The first sample image Types of tags, For the neural network model to the first The predicted category obtained from the prediction of the sample image is the _th The probability of each category, where N is the total number of sample images in the batch, and E is the total number of preset categories.

[0134] The embedded representation represents the distance loss value. This is the average of the sum of the embedded representation distances of all anchor samples and their positive samples in the batch, and the embedded representation distances of the anchor samples and their negative samples. For example, the embedded representation distance loss value... Calculate using the following formula (2-3):

[0135] (2-3)

[0136] in, This represents the process of obtaining an embedded representation of a sample image through a neural network model. For sample image dimensions, For the dimensions of the embedded representation, Represents the input number Anchor sample images, Represents the input number One positive sample image, Represents the input number One negative sample image, , It is the set of all triples in the learning sample set, and N is the total number of anchor sample images. It is an edge constant. For distance metric functions, distance metric functions The calculation formula is shown in equation (2-4) below:

[0137] (2-4)

[0138] in, denoted by , X represents the root mean square error, and X and Y represent the embedded representations of the two sample images, respectively. It is a constant. and The embedding representations of the two sample images are subjected to center normalization operations, respectively. This represents the singular value decomposition process.

[0139] Step S16, backpropagation and parameter update. Based on the contrastive learning loss... The backpropagation algorithm is used to calculate the gradient information of each trainable parameter in the neural network model relative to the loss value; based on the gradient information, the trainable parameters are updated using a preset parameter update strategy to reduce the loss value.

[0140] Step S17: Determine whether a preset stopping condition is met. If met, the model training process ends. The stopping condition includes one of the following: loss convergence, the number of training iterations reaching a preset threshold, or the model performance metrics meeting expected requirements. If the preset stopping condition is not met, return to step S14 and continue iterative training.

[0141] See Figure 11 , Figure 11 This is a block diagram illustrating the principle of a training system for a neural network-based image processing system according to an embodiment of the present invention. The training system includes a sample set construction module 51, a forward propagation module 52, a loss calculation module 53, a parameter tuning module 54, and an evaluation module 55. The sample set construction module 51 constructs a learning sample set, which includes multiple category sample subsets. Each category sample subset includes anchor samples, positive samples of the anchor samples, and negative samples. The forward propagation module 52 inputs batch samples into the neural network-based image processing system (or neural network model), obtaining the predicted category and embedded representation of each sample through forward propagation. Each batch of samples includes a preset number of anchor samples, positive samples of each anchor sample, and negative samples. The loss calculation module 53 calculates a contrastive learning loss value based on the predicted data and sample labels of each sample image in each batch of sample images. The contrastive learning loss value is a weighted sum of the category loss value and the embedded representation distance loss value. The parameter tuning module 54 calculates the gradient information of the trainable parameters in the neural network system relative to the loss value based on the contrastive learning loss value; and calculates and updates the trainable parameters based on the gradient information. The evaluation module 55 determines whether the training stopping conditions are met, such as whether the loss value has converged, whether the number of training iterations has reached a preset threshold, or whether the model performance metrics meet the expected requirements. If the training stopping conditions are met, training is stopped; otherwise, the forward propagation module 52 is notified to re-input a new batch of samples into the model.

[0142] This invention improves the strategy for using learning samples and the loss function, guiding the network to retain and mine fine-grained semantic features in images. This enhances the model's ability to recognize and express complex and subtle image details, fundamentally improving the accuracy and robustness of tasks such as image classification, object detection, recognition, and judgment.

[0143] The neural network-based image processing system of this invention can be implemented as a neural network model. It constructs a general visual feature learning paradigm of "multi-scale perception-global context calibration" based on a multi-scale convolutional branch and Transformer channel weighting structure. This paradigm can be applied to general visual tasks in various fields, such as object detection, segmentation, and classification. It can also be applied to specific tasks, demonstrating advantages in areas requiring rich understanding of image content, diverse object morphologies, and complex contextual relationships. For example, in digital pathology analysis, facing tissue slices with varying cell morphologies and dense distribution, existing CNN detection models, due to their fixed receptive fields, struggle to adaptively capture cross-scale features ranging from nuclear atypia details to the overall glandular structure. This model, however, can explicitly extract visual features at different levels through parallel multi-scale branches and utilize the Transformer module to dynamically weight and identify the most relevant scale channels based on key features, thereby achieving more accurate cancer grading, antibody classification, and so on. Furthermore, in industrial texture surface defect detection and autonomous driving scene understanding, this invention can effectively distinguish between real defects and normal texture variations, or dynamically increase attention to high-risk small targets (such as pedestrians) in traffic scenes.

[0144] Figure 12 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. The electronic device can be implemented as a server or other various terminal devices, such as desktop personal computers, tablet computers, laptop computers, etc., including a processor 601 and a memory 602. The memory 602 stores a program instruction set. When the processor 601 executes the program instruction set in the memory 602, it executes any of the aforementioned image processing methods or implements the aforementioned neural network-based image processing system.

[0145] Specifically, the processor 601 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0146] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.

[0147] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform any of the aforementioned image processing methods or implement the aforementioned neural network-based image processing system.

[0148] In one example, the electronic device may also include a communication interface 603 and a bus 610. The processor 601, memory 602, and communication interface 603 are connected via the bus 610 and communicate with each other.

[0149] The communication interface 603 is mainly used to realize communication between various modules, systems, units and / or devices in the embodiments of the present invention.

[0150] Bus 610 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. While specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0151] The present invention also provides a computer-readable storage medium storing computer program instructions thereon, which can be executed by a processor to perform any of the aforementioned image processing methods or to implement the aforementioned neural network-based image processing system. The computer-readable storage medium can be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. The storage medium can be a transient computer-readable storage medium or a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include, for example, magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state storage such as flash memory and solid-state drives.

[0152] This invention also provides a computer program product comprising a set of computer program instructions, which are executed by a processor to perform any of the aforementioned image processing methods or to implement the aforementioned neural network-based image processing system. The computer program product includes, but is not limited to, application installation packages, application plugins, and mini-programs that can run within certain applications, all published on websites or in app stores.

[0153] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0154] The above embodiments are for illustrative purposes only and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the scope of the invention. Therefore, all equivalent technical solutions should also fall within the scope of the invention.

Claims

1. An image processing method, characterized by, When a target image is input into a trained image processing system, the image processing system performs image processing on the target image, the method including: The first information is obtained by extracting features from the input target image through multi-scale branches, wherein the first information includes feature information at multiple scales. The second information is obtained by dynamically weighting the feature information at each scale and the inter-scale correlation information in the first information using an attention mechanism; and The second information is used to perform prediction output operations to obtain the category data and embedded representation of the target image; The step of dynamically weighting the feature information at each scale and the inter-scale correlation information in the first information using an attention mechanism to obtain the second information includes: The first information is divided into a plurality of batch information, wherein the total number of batch information is determined based on the following formula: Wherein N is the total number of batch information, is the number of scale branches, is the number of batches under a single scale branch, is the interval distance of the scale branch, is the interval distance of the scale branch is the corresponding correlation weight coefficient when is the number of scale branches is the interval distance of the scale branch is the number of scale branches Attention is calculated for each batch of information to obtain the corresponding attention calculation result; The attention calculation results of all batch information are concatenated to obtain the attention calculation result concatenation information; The second information is obtained by summing the attention calculation result, the spliced ​​information, and the first information.

2. The image processing method according to claim 1, characterized in that, When extracting features from the input target image using multi-scale branches to obtain the first information, a preset number of feature iterations are performed to extract the first information.

3. The image processing method according to claim 1, characterized in that, The steps of performing prediction output operations on the second information to obtain the category data and embedded representation of the target image include: The second information is sequentially processed by fully connected layer computation and first activation function computation to obtain a first embedded representation of a first preset dimension; and The second information is sequentially processed by the first fully connected layer calculation, the second activation function calculation, and the second fully connected layer calculation to obtain the second embedded representation of the second preset dimension, where the number of the second preset dimension corresponds to the number of classification categories. Wherein, the first embedded representation serves as the embedded representation of the target image; the second embedded representation serves as the category data of the target image.

4. An image processing system based on a neural network, characterized in that, include: The backbone feature extraction network is configured to extract features from the input target image through multi-scale branches to obtain first information, wherein the first information includes feature information at multiple scales. A feature processing network is configured to dynamically weight and calculate the feature information at each scale and the correlation information between scales in the first information using an attention mechanism to obtain the second information; and The prediction output network is configured to perform prediction output operations on the second information to obtain the category data and embedded representation of the target image; The feature processing network includes: The data segmentation module is configured to segment the first information output by the backbone feature extraction network into multiple batches; the total number of batches is determined based on the following formula: N is the total number of batch messages. To measure the number of branches, To be responsible for the batch size under a single scale branch The interval distance between scale branches. The scale branch interval distance is The corresponding correlation weight coefficient, Indicates in In each scale branch, the interval distance is exactly... The number of scale branch pairs; Multiple attention calculation modules are used to perform attention calculations on each batch of information to obtain the corresponding calculation results; The stitching module is configured to stitch together the calculation results output by each attention calculation module to obtain the attention calculation stitched result; and The dynamic weighted calculation module is configured to accumulate the first information and the attention calculation result output by the splicing module to obtain the second information used for predictive output processing.

5. The image processing system according to claim 4, characterized in that, The backbone feature extraction network includes a forward path and a retrieval path. In the forward path, features are extracted from the input target image through multi-scale branches. The extracted feature information is fed back to the forward path through the retrieval path for multi-scale feature iteration extraction with a preset number of iterations to obtain the first information. Alternatively, the backbone feature extraction network includes multi-level cascaded feature extraction units, with the last-level feature extraction unit outputting the first information.

6. The image processing system according to claim 5, characterized in that, The forward path of the backbone feature extraction network includes: The initial feature extraction module is configured to perform initial feature extraction on the input target image to convert image information in the spatial domain to image information in the channel domain. Multiple convolutional branch modules are configured to use convolutional kernels of corresponding scales to extract features from the target information to obtain feature information at multiple scales; The feature stitching module is configured to stitch together feature information from multiple scales to obtain feature stitching information; and The average pooling module is configured to perform average pooling on the feature concatenation information to obtain average pooling information; In the initial forward path processing, the target information is the image information converted to the channel domain by the initial feature extraction module. In the feature iterative extraction process, the target information is the average pooling information output by the average pooling module passed through the recycling path.

7. The image processing system according to claim 6, characterized in that, The forward path of the backbone feature extraction network further includes: The original feature extraction module is configured to extract features from the target information through residual branching to obtain the original feature information of the target information; correspondingly, the feature concatenation module concatenates the feature information at multiple scales to obtain intermediate concatenation information; and The accumulation module is configured to accumulate the intermediate splicing information and the original feature information output by the residual branch original feature extraction module to obtain feature splicing information; Correspondingly, the average pooling module performs an average pooling operation on the feature splicing information output by the accumulation module to obtain average pooling information.

8. The image processing system according to claim 7, characterized in that, The forward path of the backbone feature extraction network further includes: The feature enhancement module is configured to sequentially perform normalization, activation function calculation, and max pooling operations on the received target information to enhance its features, and then send the feature-enhanced target information to multiple convolutional branch modules; and / or Multiple BR modules, each corresponding to a convolutional branch module, are used to perform normalization and nonlinear mapping operations on the feature information extracted by the convolutional branch modules in sequence; and / or The BN module, corresponding to the original feature extraction module, is used to perform batch normalization on the original feature information extracted by the original feature extraction module.

9. The image processing system according to claim 4, characterized in that, The feature processing network further includes a dimension mapping module, which is configured to perform preset dimension mapping processing on the first information and input the first information after dimension mapping processing to the data segmentation module; correspondingly, the first information for the dynamic weighted calculation module to perform cumulative calculation is the first information output by the dimension mapping module.

10. The image processing system according to claim 4, characterized in that, The prediction output network includes: The vector output module is configured to sequentially compute the second information through a fully connected layer and a first activation function to obtain a first embedded representation of a first preset dimension; and The category output module is configured to calculate the second embedded representation of the second information by sequentially passing it through a fully connected layer and a second activation function, wherein the number of the second preset dimensions corresponds to the number of classification categories.

11. The image processing system according to claim 10, characterized in that, The category output module further performs a second fully connected layer calculation on the second embedded representation of the second preset dimension, and the result of the second fully connected layer calculation is used as the category data of the target image.

12. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a set of computer program instructions, which, when executed by the processor, execute the set of computer program instructions in the memory, perform the image processing method according to any one of claims 1-3 or implement the image processing system according to any one of claims 4-11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a set of computer program instructions, which, when executed by a processor, perform the image processing method according to any one of claims 1-3 or implement the image processing system according to any one of claims 4-11.

14. A computer program product, characterized in that, It includes a computer program instruction set, which, when executed by a processor, performs the image processing method according to any one of claims 1-3 or implements the image processing system according to any one of claims 4-11.

Images