Visual model construction method and system based on modular neural network architecture
By decomposing the visual task into multiple lightweight modules for parallel processing and fusing them using an attention mechanism, the problems of high computational resource consumption and poor interpretability of the holistic convolutional neural network model are solved, achieving efficient and robust computer vision processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIAMEI HUITONG TECHNOLOGY (SHENZHEN) CO LTD
- Filing Date
- 2025-09-11
- Publication Date
- 2026-05-08
AI Technical Summary
Existing holistic convolutional neural network models are computationally expensive, resource-intensive, poorly interpretable, dependent on large-scale labeled data, and lack generalization ability. They are difficult to run efficiently on resource-constrained devices, and transfer learning is inefficient in cross-domain scenarios.
The vision task is decomposed into multiple specialized subtasks, lightweight modules are constructed for parallel processing, lightweight convolutional structures and specific loss functions are used to train the modules, the module outputs are fused through an attention mechanism, and an interpretable report is generated.
It reduces computational resource consumption, improves the feasibility and robustness of the model on resource-constrained devices, enhances cross-domain adaptability and interpretability, and improves the reliability and credibility of the system.
Smart Images
Figure CN121997989A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neural network architecture technology, and in particular to a method and system for constructing visual models based on a modular neural network architecture. Background Technology
[0002] In recent years, deep learning-based computer vision technology has made significant progress. Among them, single, holistic convolutional neural network (CNN) models (such as VGG and ResNet) trained end-to-end have become the mainstream implementation. These models are typically pre-trained on large-scale general datasets (such as ImageNet), learning to extract hierarchical features from low to high levels, and then fine-tuning them for transfer to specific tasks.
[0003] However, this traditional monolithic model architecture has several inherent drawbacks. First, it is computationally expensive; training and deploying a large monolithic network requires enormous computing resources and time, making it difficult to run efficiently on resource-constrained edge devices. Second, the model has poor interpretability; its decision-making process is like a "black box," making it difficult to trace which input features caused a specific output, which poses a serious obstacle to its application in high-risk fields such as healthcare and autonomous driving. Third, the model heavily relies on large amounts of high-quality labeled data, making it prone to overfitting and insufficient generalization ability in data-scarce scenarios. Furthermore, such models have poor fault tolerance; failure or performance degradation of components in the network can directly lead to the failure of the entire system.
[0004] To overcome these limitations, existing technologies attempt to employ transfer learning, which involves adapting a pre-trained model on a general dataset to a new domain through fine-tuning. However, this method also faces significant challenges: the high-level features learned by the pre-trained model are highly dependent on the source data domain (such as natural images). When the target domain differs greatly from the source domain (such as medical images or remote sensing images), the features in the deep filters become almost ineffective. The fine-tuning process essentially requires a huge computational overhead to cover the model's original, irrelevant knowledge, resulting in low efficiency and nearly equivalent to retraining.
[0005] Currently, the construction of specialized, modular model systems still heavily relies on algorithm engineers manually performing task decomposition, module design, and joint debugging. This manually driven process is not only inefficient but also difficult to scale, and it cannot dynamically build and optimize the system architecture according to task requirements. Therefore, there is an urgent need in this field for a technical solution that can automatically design, train, integrate, and optimize a series of specialized modules, ultimately forming an efficient, robust, and interpretable vision processing system. Summary of the Invention
[0006] The purpose of this invention is to address the shortcomings of the prior art by providing a visual model construction method and system based on a modular neural network architecture. This method automatically breaks down instructions into several tasks, automatically trains multiple specialized and smaller neural network modules for each task, and integrates their outputs to achieve efficient, interpretable, and robust computer vision processing.
[0007] On the one hand, a method for constructing visual models based on a modular neural network architecture is provided, including the following steps: S1: Receive the task requirements and performance standards input by the client, and parse out the set of sub-tasks to be executed through the semantic understanding module. The sub-tasks include, but are not limited to, edge detection, color analysis, and shape recognition. S2: For each subtask, automatically build and train its corresponding specialized neural network module, where each module uses a lightweight convolutional structure and is independently customized with a specific loss function to generate a customized pre-trained model for each specialized neural network module. S3: Evaluate the performance of each customized pre-trained model and optimize it based on the evaluation results; S4: Integrate the optimized and qualified customized pre-trained models to form the final central fusion model that meets the client's requirements, and output the final processing result according to the input task requirements.
[0008] Further, in step S2, the training process of the specialized neural network module includes: Generative adversarial networks are used to automatically generate synthetic data related to subtasks to enhance the module's generalization ability; Configure the network depth, activation function, and loss function independently for each module based on the characteristics of its sub-tasks; A strategy of dynamically adjusting the learning rate is used for training to ensure that each specialized neural network module achieves optimal performance on its specific feature extraction task.
[0009] Preferably, the configuration of the loss function further includes: For the module corresponding to the edge detection task, a convolutional neural network based on the encoder-decoder structure is adopted, and a hybrid loss function combining cross-entropy loss and Dice loss is used for training. For the module corresponding to the color analysis task, a structure based on a fully convolutional network is adopted, and a loss function based on the difference in color distribution is used for training; For the module corresponding to the shape recognition task, a deep convolutional neural network is used for implementation, and a multi-class cross-entropy loss function is used for training.
[0010] Furthermore, in step S3, the performance evaluation of each customized pre-trained model includes: S31: Evaluate the current performance of each customized prediction model using a set of reserved validation datasets; S31: If the model performance does not meet the standard, automatically adjust its training hyperparameters or supplement it with new training data and retrain it; S33: Repeat steps S31 to S32 until the performance of all customized prediction models meets the predetermined criteria, which are defined by the performance criteria and efficiency indicators proposed by the client.
[0011] Furthermore, in step S4, the integration of the central fusion model specifically includes: The model fusion is performed using a feature fusion method based on an attention mechanism. The attention weights for each customized pre-trained model are calculated as follows: in, Indicates the first The feature vector output by a customized pre-trained model Represents the learning parameter matrix, For bias vectors, For attention weight vectors, Indicates the first The attention weights of a customized pre-trained model are used for weighted summation to obtain fused features.
[0012] Preferably, the integration of the central fusion model further includes: Establish a shared latent space mapping function to map the heterogeneous feature vectors output by each customized pre-trained model to the same metric space; The mapping function is jointly optimized with the central fusion model through an adversarial training process, so that semantically related features from different customized pre-trained models are close in the latent space, while unrelated features are far apart, thereby achieving effective alignment and splicing of cross-modal features.
[0013] Furthermore, in step S4, while outputting the final processing result, a corresponding interpretability report is also output. This report can be traced back to the decision explanation information of each specialized neural network module, specifically including: The contribution weights of each specialized neural network module to the current decision, the visualization heatmap of the basic visual features extracted by each module on the input image, and the visualization labels output by the anomaly submodule.
[0014] On the other hand, a visual model construction system based on a modular neural network architecture is provided, including: The task receiving and parsing module is used to receive the task requirements and performance standards input by the client, and parse out the set of sub-tasks to be executed through the semantic understanding module. The sub-tasks include, but are not limited to, edge detection, color analysis, and shape recognition. The model building and training module automatically builds and trains the corresponding specialized neural network module for each subtask. Each module uses a lightweight convolutional structure and is independently customized with a specific loss function to generate a customized pre-trained model for each specialized neural network module. The performance analysis and optimization module is used to evaluate the performance of each customized pre-trained model and optimize it based on the evaluation results. The model integration and interaction module is used to integrate the optimized and qualified customized pre-trained models to form a central fusion model that meets the client's requirements and outputs the final processing results according to the input task requirements.
[0015] In addition, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the visual model construction method based on the modular neural network architecture described above.
[0016] Meanwhile, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the visual model construction method based on the modular neural network architecture described above.
[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention significantly reduces overall computational overhead and resource consumption by decomposing complex visual tasks into multiple specialized sub-tasks and constructing lightweight modules for parallel processing. This enables high-performance visual models to be deployed on resource-constrained edge devices, exhibiting good scalability. The modules are independent of each other, and the failure of a single module does not affect the overall operation of the system. Combined with performance monitoring and iterative optimization mechanisms, weak links can be located and repaired in real time, forming a continuously evolving robust visual processing model. This invention makes the decision-making process transparent through a modular architecture, allowing for the traceability of the output and contribution weight of each sub-module and generating a visual report, which greatly improves the credibility of the model in various fields with high reliability requirements. This invention employs a customized loss function and attention fusion mechanism, enabling the model to effectively learn cross-domain features, reduce dependence on large-scale labeled data, and improve adaptability and anti-interference performance in unfamiliar scenarios. Attached Figure Description
[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a visual model construction method based on a modular neural network architecture according to the present invention; Figure 2 This is a block diagram of a visual model construction system based on a modular neural network architecture according to the present invention. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] This invention relates to a method and system for constructing a visual model based on a modular neural network architecture. The method automatically constructs, trains, and optimizes multiple specialized lightweight sub-modules by parsing user task requirements. Finally, it fuses the outputs of each module through an attention mechanism to form an efficient, interpretable, and robust central fusion model, achieving high-performance computer vision processing.
[0021] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples.
[0022] Example 1 This embodiment takes an example of a scenario applied to the detection of surface defects in industrial parts.
[0023] A technical solution for constructing visual models based on a modular neural network architecture is provided, such as... Figure 1 As shown, it includes the following steps: S1: Receive the task requirements and performance standards input by the client, and parse out the set of sub-tasks to be executed through the semantic understanding module. The sub-tasks include, but are not limited to, edge detection, color analysis, and shape recognition. S2: For each subtask, automatically build and train its corresponding specialized neural network module, where each module uses a lightweight convolutional structure and is independently customized with a specific loss function to generate a customized pre-trained model for each specialized neural network module. S3: Evaluate the performance of each customized pre-trained model and optimize it based on the evaluation results; S4: Integrate the optimized and qualified customized pre-trained models to form the final central fusion model that meets the client's requirements, and output the final processing result according to the input task requirements.
[0024] The specific operation process of this embodiment is as follows: Figure 2 As shown. First, the user inputs the task requirements on the client. In this embodiment, the user (client) inputs the training instructions, evaluation criteria, and data requirements through the interface as follows: Training instructions: "Inspect the product surface for scratches, dents, and stains"; Evaluation procedure: An evaluation script that includes accuracy, recall, and F1 score; Data requirements / standards: "The most acceptable types of images to delete include: blurry images, backlit images, and distorted images." (That is, the system should automatically filter out these low-quality data to prevent them from affecting training).
[0025] Next, the semantic understanding module parses the instruction and identifies three core basic visual subtasks: edge detection (for identifying the contours of scratches and dents), color analysis (for identifying stains that are different from the substrate color), and texture recognition (for distinguishing scratches from natural textures).
[0026] Based on this, the system automatically constructs independent specialized modules for the three sub-tasks. In this embodiment, the system determines, according to the task instructions, that three specialized filters are needed: edge detection (for scratches and dents), color analysis (for stains), and texture recognition (to distinguish between real and fake defects). Then, each specialized module is trained in step S2. The training process of the specialized neural network module includes: Generative adversarial networks are used to automatically generate synthetic data related to subtasks to enhance the module's generalization ability; Configure the network depth, activation function, and loss function independently for each module based on the characteristics of its sub-tasks; A strategy of dynamically adjusting the learning rate is employed during training to ensure that each specialized neural network module achieves optimal performance on its specific feature extraction task. The configuration of the loss function further includes: For the module corresponding to the edge detection task, a convolutional neural network based on the encoder-decoder structure is adopted, and a hybrid loss function combining cross-entropy loss and Dice loss is used for training. For the module corresponding to the color analysis task, a structure based on a fully convolutional network is adopted, and a loss function based on the difference in color distribution is used for training; For the module corresponding to the shape recognition task, a deep convolutional neural network is used for implementation, and a multi-class cross-entropy loss function is used for training.
[0027] In this embodiment, filter-1 (edge detection) employs a U-Net encoder-decoder network. The loss function is configured as a weighted sum of cross-entropy loss and Dice loss, and training is performed using synthetic scratch and dent defect image data. Filter-2 (color analysis) employs a lightweight fully convolutional network (FCN). The loss function is defined as the difference in color distribution between the predicted and normal regions in the CIELAB color space (Wasserstein distance). It is trained using image data simulating stains under different lighting conditions. Filter-3 (texture recognition) uses a simplified ResNet-18 network. The loss function uses the standard multi-class cross-entropy loss to classify "scratches", "natural textures", and "background".
[0028] In the above embodiments, the training of each sub-module adopts a dynamic learning rate adjustment strategy (such as CosineAnnealing) and uses defect sample data generated by GAN for data augmentation to improve generalization ability.
[0029] Next, in step S3, a performance evaluation is performed on each customized pre-trained model, which further includes: S31: Evaluate the current performance of each customized prediction model using a set of reserved validation datasets; S31: If the model performance does not meet the standard, automatically adjust its training hyperparameters or supplement it with new training data and retrain it; S33: Repeat steps S31 to S32 until the performance of all customized prediction models meets the predetermined criteria, which are defined by the performance criteria and efficiency indicators proposed by the client.
[0030] In this embodiment, we use a validation set containing 1000 labeled images to evaluate the three customized pre-trained models. Evaluation metrics include subtask accuracy, recall, and inference latency for each module. The system automatically loads the evaluation program and evaluates the initial model trained in step S3 on the validation set. If the recall of a filter (such as a color analysis filter) is insufficient, the system automatically performs the following operations: recover the model from the checkpoint, adjust its hyperparameters (such as reducing the learning rate), and supplement it with more difficult example data of the "stain" class. The optimized high-performance filter is saved and ready for integration and deployment.
[0031] The filter is retrained and evaluated again. This cycle continues until all filters meet the performance requirements.
[0032] The evaluation revealed that the color analysis module's accuracy under strong light interference was only 95%, failing to meet the standard. The system automatically triggered an optimization process: 500 additional images of stains under strong light conditions were added to the module, and the weights of the convolutional layers in its network were fine-tuned. After two rounds of iterative optimization, the performance of all modules met the standards.
[0033] Then, the three optimized modules are deployed and executed in parallel. First, the integration of the central fusion model described in step S4 is performed, specifically including: The model fusion is performed using a feature fusion method based on an attention mechanism. The attention weights for each customized pre-trained model are calculated as follows: in, Indicates the first The feature vector output by a customized pre-trained model Represents the learning parameter matrix, For bias vectors, For attention weight vectors, Indicates the first The attention weights of a customized pre-trained model are used for weighted summation to obtain fused features.
[0034] The integration of the central fusion model also includes: Establish a shared latent space mapping function to map the heterogeneous feature vectors output by each customized pre-trained model to the same metric space; The mapping function is jointly optimized with the central fusion model through an adversarial training process, so that semantically related features from different customized pre-trained models are close in the latent space, while unrelated features are far apart, thereby achieving effective alignment and splicing of cross-modal features.
[0035] In step S4, along with the final processing result, a corresponding interpretability report is also output. This report can be traced back to the decision explanation information of each specialized neural network module, specifically including: The contribution weights of each specialized neural network module to the current decision, the visualization heatmap of the basic visual features extracted by each module on the input image, and the visualization labels output by the anomaly submodule.
[0036] In this embodiment, for an input part image, the three modules output: edge heatmap, color anomaly region mask, and texture classification confidence score, respectively.
[0037] These heterogeneous features are input into a central fusion model. This model first uses a mapping network trained adversarially to map the three features to the same latent space for alignment. Subsequently, an attention fusion mechanism is employed to calculate the weights of the features from each module (the calculation formula is as described in claim 5), and the weighted sum is used to obtain the fused feature vector.
[0038] Finally, a fully connected classifier outputs the final defect classification result ("qualified" or "unqualified", and the defect type) based on the fused feature vector. Simultaneously, the system generates an interpretable report, including: statistics on data cleaning (e.g., how many blurry images were deleted), the highest attention weight assigned to the edge module (0.5), a heatmap of abnormal regions output by the color module, and a note indicating that "the color module has low confidence in this image, but due to the high confidence output of the edge module, it is ultimately judged as a scratch," confirming whether the current model state meets the client's requirements and thus clearly tracing the decision-making logic.
[0039] Example 2 This embodiment briefly illustrates its application in medical imaging (retinal OCT image analysis) to demonstrate its domain adaptability.
[0040] The task requirements are to parse out the following sub-tasks: "bleeding point detection" (color analysis), "vascular morphology segmentation" (edge detection), and "macula recognition" (shape recognition).
[0041] A customized loss function suitable for biomedical images was designed for the "bleeding point detection" module, with a focus on recall rate for small target regions.
[0042] During the fusion phase, the central model, considering the high reliability requirements of medical images, adopts a more conservative integration strategy. A positive result is only output when all modules agree on the "lesion"; otherwise, it must be submitted to a doctor for review. This demonstrates the flexibility and configurability of the system.
[0043] Example 3 This embodiment provides a visual model construction system based on a modular neural network architecture, including: The task receiving and parsing module is used to receive the task requirements and performance standards input by the client, and parse out the set of sub-tasks to be executed through the semantic understanding module. The sub-tasks include, but are not limited to, edge detection, color analysis, and shape recognition. The model building and training module automatically builds and trains the corresponding specialized neural network module for each subtask. Each module uses a lightweight convolutional structure and is independently customized with a specific loss function to generate a customized pre-trained model for each specialized neural network module. The performance analysis and optimization module is used to evaluate the performance of each customized pre-trained model and optimize it based on the evaluation results. The model integration and interaction module is used to integrate the optimized and qualified customized pre-trained models to form a central fusion model that meets the client's requirements and outputs the final processing results according to the input task requirements.
[0044] It should be noted that the steps in the visual model construction method based on modular neural network architecture provided in this embodiment can be implemented based on the corresponding modules in the visual model construction system based on modular neural network architecture. Those skilled in the art can refer to the technical solution of the system to implement the steps of the method. That is, the embodiments in the system can be understood as preferred examples of implementing the method, and will not be elaborated here.
[0045] Besides implementing the system and its various devices provided by this invention in purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the system and its various devices of this invention appear as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0046] Finally, it should be noted that the above description is only a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be pointed out that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.
[0047] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
Claims
1. A method for constructing a visual model based on a modular neural network architecture, characterized in that, Includes the following steps: S1: Receive the task requirements and performance standards input by the client, and parse out the set of sub-tasks to be executed through the semantic understanding module. The sub-tasks include, but are not limited to, edge detection, color analysis, and shape recognition. S2: For each subtask, automatically build and train its corresponding specialized neural network module, where each module uses a lightweight convolutional structure and is independently customized with a specific loss function to generate a customized pre-trained model for each specialized neural network module. S3: Evaluate the performance of each customized pre-trained model and optimize it based on the evaluation results; S4: Integrate the optimized and qualified customized pre-trained models to form the final central fusion model that meets the client's requirements, and output the final processing result according to the input task requirements.
2. The visual model construction method based on a modular neural network architecture according to claim 1, characterized in that, In step S2, the training process of the specialized neural network module includes: Generative adversarial networks are used to automatically generate synthetic data related to subtasks to enhance the module's generalization ability; Configure the network depth, activation function, and loss function independently for each module based on the characteristics of its sub-tasks; A strategy of dynamically adjusting the learning rate is used for training to ensure that each specialized neural network module achieves optimal performance on its specific feature extraction task.
3. The visual model construction method based on a modular neural network architecture according to claim 2, characterized in that, The configuration of the loss function further includes: For the module corresponding to the edge detection task, a convolutional neural network based on the encoder-decoder structure is adopted, and a hybrid loss function combining cross-entropy loss and Dice loss is used for training. For the module corresponding to the color analysis task, a structure based on a fully convolutional network is adopted, and a loss function based on the difference in color distribution is used for training; For the module corresponding to the shape recognition task, a deep convolutional neural network is used for implementation, and a multi-class cross-entropy loss function is used for training.
4. The visual model construction method based on a modular neural network architecture according to claim 1, characterized in that, In step S3, the performance evaluation of each customized pre-trained model further includes: S31: Evaluate the current performance of each customized prediction model using a set of reserved validation datasets; S31: If the model performance does not meet the standard, automatically adjust its training hyperparameters or supplement it with new training data and retrain it; S33: Repeat steps S31 to S32 until the performance of all customized prediction models meets the predetermined criteria, which are defined by the performance criteria and efficiency indicators proposed by the client.
5. The visual model construction method based on a modular neural network architecture according to claim 1, characterized in that, In step S4, the integration of the central fusion model specifically includes: The model fusion is performed using a feature fusion method based on an attention mechanism. The attention weights for each customized pre-trained model are calculated as follows: in, Indicates the first The feature vector output by a customized pre-trained model Represents the learning parameter matrix, For bias vectors, For attention weight vectors, Indicates the first The attention weights of a customized pre-trained model are used for weighted summation to obtain fused features.
6. The visual model construction method based on a modular neural network architecture according to claim 5, characterized in that, The integration of the central fusion model also includes: Establish a shared latent space mapping function to map the heterogeneous feature vectors output by each customized pre-trained model to the same metric space; The mapping function is jointly optimized with the central fusion model through an adversarial training process, so that semantically related features from different customized pre-trained models are close in the latent space, while unrelated features are far apart, thereby achieving effective alignment and splicing of cross-modal features.
7. The visual model construction method based on a modular neural network architecture according to claim 1, characterized in that, In step S4, along with the final processing result, a corresponding interpretability report is also output. This report can be traced back to the decision explanation information of each specialized neural network module, specifically including: The contribution weights of each specialized neural network module to the current decision, the visualization heatmap of the basic visual features extracted by each module on the input image, and the visualization labels output by the anomaly submodule.
8. A visual model construction system based on a modular neural network architecture, characterized in that, include: The task receiving and parsing module is used to receive the task requirements and performance standards input by the client, and parse out the set of sub-tasks to be executed through the semantic understanding module. The sub-tasks include, but are not limited to, edge detection, color analysis, and shape recognition. The model building and training module automatically builds and trains the corresponding specialized neural network module for each subtask. Each module uses a lightweight convolutional structure and is independently customized with a specific loss function to generate a customized pre-trained model for each specialized neural network module. The performance analysis and optimization module is used to evaluate the performance of each customized pre-trained model and optimize it based on the evaluation results. The model integration and interaction module is used to integrate the optimized and qualified customized pre-trained models to form a central fusion model that meets the client's requirements and outputs the final processing results according to the input task requirements.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the visual model construction method based on a modular neural network architecture as described in any one of claims 1-8.
10. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the visual model construction method based on a modular neural network architecture as described in any one of claims 1-8.