A multi-scene-oriented visual detection system and method combining semantic guidance and hybrid experts

Abstract

The application discloses a multi-scene visual detection system and method fusing semantic guidance and mixed experts, and belongs to the technical field of computer vision. The system is composed of a multi-modal perception module, a routing decision module, a pluggable expert pool module, an evidence normalization module, a hyperspherical reliability selection module and a result output module. The system receives images, videos and text instructions, realizes on-demand scheduling of experts relying on the routing decision, unifies heterogeneous expert output with the aid of the evidence normalization, and then completes reliability comparison of expert output in unit hyperspherical space. The system supports two working modes of single expert and fixed double expert cooperation, can reduce calculation cost and expansion cost, improve the stability of detection results in complex scenes, and is suitable for visual detection scenes such as industrial inspection and security monitoring.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, computer vision, and intelligent detection technology, and specifically to a visual detection system and method that integrates semantic guidance and hybrid experts for multiple scenarios. Background Technology

[0002] In visual inspection tasks across various scenarios such as industrial inspection, manufacturing quality inspection, security monitoring, and traffic analysis, different inspection targets, objects, and task requirements typically correspond to different types of visual processing models, such as target detection models, image segmentation models, image classification models, or state assessment models.

[0003] In existing technologies, one approach involves training and deploying independent models for different visual detection tasks. While this approach can achieve good results for specific tasks, it often requires repeated model training, deployment, and maintenance when the number of tasks increases, application scenarios change, or detection targets expand, resulting in high system scalability costs. Another approach uses a single model to handle multiple visual detection tasks. While this approach reduces the number of models, it is prone to representation conflicts, performance trade-offs, and the difficulty of directly reusing existing mature models across different tasks.

[0004] In recent years, visual language models have emerged that can jointly model multiple types of information, such as images and text, and utilize natural language descriptions or task instructions to participate in the visual understanding process. However, existing visual language models typically adopt a tightly coupled end-to-end architecture, which makes it difficult to simultaneously meet the needs of low-cost task expansion, expert coordination, and reuse of existing expert models when facing multiple heterogeneous expert models already deployed in multi-scene visual inspection. At the same time, they usually need to compress the capabilities of multiple types of tasks into a single model parameter space, which is not conducive to explicit invocation and modular iteration of different expert models.

[0005] Furthermore, in visual inspection scenarios requiring multiple expert models to participate in the judgment, the output structure, confidence scale, and representation form of different expert models often differ. For example, a detection expert might output the target category, detection box, and confidence score; a segmentation expert might output a pixel-level mask; a classification expert might output the category probability; and a state assessment expert might output a state label or anomaly score. Directly using numerical weighting, voting, or simple concatenation for fusion can easily introduce fusion bias, affecting the stability and consistency of the final result. Simultaneously, existing visual inspection systems typically lack a unified task control interface, making it difficult to controllably select and on-demand invoke expert models based on externally input task descriptions or operational instructions.

[0006] In conclusion, it is necessary to improve the existing technology. Summary of the Invention

[0007] The purpose of this invention is to provide a visual inspection system and method for multiple scenarios that integrates semantic guidance and hybrid experts, so as to realize expert selection based on semantic information, evidence standardization processing of heterogeneous expert outputs, and reliability comparison and selection of dual or multi-expert outputs under a unified processing link.

[0008] To achieve the objectives of this invention, the technical solution provided by this invention is as follows: First aspect This invention provides a visual inspection system that integrates semantic guidance and hybrid experts for multiple scenarios, comprising: The multimodal perception module receives input visual data and optional text instructions, and outputs a shared visual representation, an effective location mask, and optional control vectors. The routing decision module is used to receive the shared visual representation, the effective location mask, and the optional control vector, and generate expert selection results or dual expert coordination results. A pluggable expert pool module is used to call the target expert to perform inference based on the expert selection result and generate the original detection result, i.e., the expert output result; The evidence normalization module is used to convert the original detection results into evidence vectors of a uniform dimension. The hyperspherical reliability selection module is used to normalize the evidence vector and perform reliability comparison based on the normalized evidence vector to determine the final expert output to be adopted, i.e., the comparison selection result. The results output module is used to convert expert output results or comparison and selection results into structured test results in a unified format.

[0009] Second aspect This invention also provides a visual detection method for multiple scenarios that integrates semantic guidance and hybrid experts, implemented using the aforementioned visual detection system for multiple scenarios that integrates semantic guidance and hybrid experts, and includes the following steps: It receives input visual data and optional text instructions, and outputs a shared visual representation, an effective location mask, and an optional control vector. Receive the shared visual representation, effective location mask, and optional control vector, and generate expert selection results or dual-expert coordination results; Based on the expert selection results, the target expert is invoked to perform reasoning and generate the original detection results, i.e., the expert output results; The original detection results are converted into evidence vectors of a uniform dimension. The evidence vector is normalized and its reliability is compared based on the normalized evidence vector to determine the final expert output to be adopted, i.e., the comparison and selection result. Convert expert output results or comparison and selection results into structured detection results in a unified format.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: First, the present invention selects expert models on demand through a routing decision module. In single-expert execution mode, only the target expert is activated to participate in the reasoning of the current sample, thereby reducing the computational resource consumption caused by experts unrelated to the current sample participating in the reasoning. Secondly, through the evidence normalization module and the hypersphere reliability selection module, the heterogeneous outputs of different expert models are converted into a unified evidence representation, and directional consistency comparison is performed in the unit hypersphere space, thereby reducing the impact of differences in different output structures and confidence scales on the comparison results. In addition, by using a fixed dual-expert collaborative comparison mode to jointly judge the outputs of complementary experts, and determining the final expert output based on the reliability comparison results of the evidence vectors, the consistency and stability of detection results in complex scenarios can be improved. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the overall system structure provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the implementation of the routing decision module in this embodiment of the invention. Figure 3 This is the core logic diagram of the hyperspherical reliability selection module in this embodiment of the invention; Figure 4 This is a comparison diagram of the training paradigms of the single-expert execution mode and the multi-expert collaborative comparison mode in the embodiments of the present invention; Figure 5 This is a diagram illustrating the text instruction-guided expert activation mechanism in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be noted that the acquisition of data and collection of information in this application are legal, compliant, or obtained with the consent of the subject of the data collection.

[0014] like Figures 1-5As shown, the present invention provides a visual inspection system for multiple scenarios that integrates semantic guidance and hybrid experts. The system can be deployed on a computing device that includes at least one processor and at least one memory. The computing device can be an edge computing terminal, a server, or other electronic device with visual data processing capabilities.

[0015] At the logical architecture level, the system includes at least: The multimodal perception module receives input visual data and optional text instructions, and outputs a shared visual representation, an effective location mask, and a control vector. The routing decision module is used to receive the shared visual representation, the effective location mask, and the optional control vector, and generate expert selection results or dual expert coordination results. A pluggable expert pool module is used to call the target expert to perform inference based on the expert selection result and generate the original detection result. The evidence normalization module is used to convert the original detection results into evidence vectors of a uniform dimension. The hyperspherical reliability selection module is used to normalize the evidence vector and perform reliability comparison based on the normalized evidence vector to determine the final expert output to be adopted. The results output module is used to convert expert output results or comparison and selection results into structured test results in a unified format.

[0016] The above modules are connected sequentially through data interfaces to form the following processing chain: multimodal perception → routing decision → expert execution → evidence normalization → hyperspherical reliability selection → result output.

[0017] When the multimodal perception module is executed, the visual input data received by the system can be a single frame image, a continuously acquired image sequence, or a video stream.

[0018] When the input is a single-frame image, the system performs spatial feature extraction on the image; When the input is a video stream or a multi-frame image sequence, the video coding network in the multimodal perception module performs spatiotemporal joint coding on the visual data and outputs a video-level shared visual representation sequence containing inter-frame temporal correlation information.

[0019] Visual data is first input into the visual encoding network in the multimodal perception module for feature extraction. The visual encoding network is used to generate shared visual representations that can be aligned with text instructions. In one embodiment, the visual encoding network may employ a network structure based on a self-attention mechanism; alternatively, it may employ other network structures capable of outputting semantic feature representations.

[0020] Specifically, the multimodal perception module processes the input image. and optional text commands Joint encoding is performed to obtain a shared visual representation, control vector, and corresponding mask. The process is as follows: ; in, This represents a multimodal coding network. Indicates shared visual representation, represents the effective position mask corresponding to the shared visual representation, and z represents the control vector derived from the shared visual representation.

[0021] The shared visual representation is used for subsequent expert selection and execution. Specifically, the shared visual representation is passed to the routing decision module to generate expert selection results; it is also passed to the selected expert model to execute the current detection task and generate the original detection results.

[0022] In addition, the system can also receive natural language text commands. These text commands can be task description statements, rule-based operation instructions, or predefined identifier statements.

[0023] The text instructions are converted into semantic embedding vectors by a text encoding network to represent the semantic intent of the current detection task.

[0024] In one implementation, the system can generate a control vector z based on purely visual information, specifically by: processing the shared visual representation. The visual global representation v is obtained by convergence, and then a control vector is generated through a mapping function, represented as: ; In another implementation, the system may also incorporate textual instruction information to generate control vectors. Embed the text semantics into a vector, or that vector with a visual global representation. The fusion result is input into the mapping function to obtain the control vector.

[0025] The control vector does not directly participate in pixel-level prediction, but rather participates in the expert selection process as a control condition of the routing decision module, thereby achieving decoupling between visual content representation and task control signals.

[0026] Preferably, the routing decision module receives the shared visual representation output by the multimodal perception module. Valid location mask and optional control vectors generated during the text control encoding process. The routing decision module maps the above inputs to generate routing scores and expert selection distributions corresponding to each expert in the expert pool.

[0027] Specifically, record Then the routing score and selection distribution are expressed as: ; in, This represents the output of the coordination function after processing the shared visual representation, effective location mask, and control vector. and For learnable parameters, This represents the routing score for each expert. This indicates the distribution chosen by the experts.

[0028] In single-expert execution mode, the system selects the expert with the highest weight as the target expert based on the expert selection distribution: ; The target expert then generates the original detection result for the current sample: ; in, Indicates the first The input to an expert model can be shared visual representations, raw visual data, or expert input data adapted through an interface. Indicates the first An expert model.

[0029] In the fixed dual-expert collaborative comparison mode, the system does not perform arbitrary expert set search, but generates coordination results for two pre-configured experts. The coordination results are used for subsequent evidence normalization and hyperspherical reliability selection processing.

[0030] Furthermore, the system can also represent text commands. In relation to the identities of the experts The matching relationships between them generate text-guided distributions: ; ; in, Indicates temperature parameter, Indicates the first A predefined identity embedding vector for an expert model.

[0031] Based on this, the text-guided distribution and the visual routing distribution are combined to form the final expert selection distribution: ; in, This represents the visual route distribution obtained based on shared visual representations. This represents the fusion coefficient.

[0032] In this way, the system can call up experts in the expert pool as needed based on the current input image and optional text instructions, thereby reducing the computational resource overhead caused by experts unrelated to the current sample participating in reasoning.

[0033] Preferably, the pluggable expert pool module includes multiple expert models with different structures and output formats. These expert models may include, but are not limited to, detection experts, segmentation experts, classification experts, and regression experts. Each expert model has independent parameters and does not share model parameters.

[0034] During the reasoning process, the selected experts receive shared visual representations, raw visual data, or expert input data adapted through an interface, and generate raw detection results; the unselected experts do not participate in the reasoning process of the current sample.

[0035] Furthermore, the pluggable expert pool module supports the independent access, replacement, or expansion of expert models. When new task capabilities are required, expert models trained for the corresponding task can be accessed through the expert pool module without reconstructing existing expert models.

[0036] Preferably, since different expert models may have different output formats, confidence scales, and representation methods, the present invention sets up an evidence standardization module to uniformly represent the expert output.

[0037] Specifically, the evidence normalization process includes: (1) Receive the raw test results output by the selected expert; (2) Extract intermediate features corresponding to the original detection results; (3) Perform scaling transformation, recalibration, or compression on the confidence information in the original detection results; (4) Convert the original outputs of different experts into evidence vectors with a unified dimension; (5) Normalize the evidence vector to obtain a comparable unified evidence representation.

[0038] In one implementation, for the selected experts The original test results The system transforms the evidence into a uniform-dimensional evidence vector using an evidence transformation function: ; in, This represents the intermediate features corresponding to the original detection result. This represents the evidence transformation function.

[0039] For example, regarding the output of testing experts, This can include the target category, detection bounding box, and detection confidence. This may include target region features and evidence transformation functions. The category embedding, bounding box position parameters, detection confidence, and target region features can be mapped into an evidence vector; for the segmentation expert output, Mask region features, boundary features, and segmentation confidence can be mapped to evidence vectors; for the output of state assessment experts, State label embeddings, anomaly scores, and temporal features can be mapped into evidence vectors.

[0040] Subsequently, the evidence vector is normalized: ; in, This represents the normalized evidence vector. It is a stable term.

[0041] Through the above processing, the original outputs generated by different expert models can be uniformly mapped to a comparable representation space, thus providing a basis for subsequent selection of hyperspherical reliability.

[0042] Preferably, after evidence normalization, the evidence vector is mapped to a unit hypersphere space. The mapping method can be vector norm normalization or other unit-length constrained projection methods.

[0043] In a unit hyperspherical space, the system performs reliability comparisons based on normalized evidence vectors to determine the final expert output to be adopted.

[0044] In the fixed dual-expert collaborative comparison mode, the system pre-configures two experts. and Participate in the current sample inference. Two experts generate the original detection results separately. and The normalized evidence vector is obtained through the evidence normalization module. and .

[0045] Specifically, the system is based on control vectors And the evidence vectors corresponding to the two experts generate the dual-expert coordination weights: ; in, This represents the coordination function between two experts. .

[0046] Subsequently, the system generates a reference comparison direction based on the aforementioned coordination weights: ; The reference comparison direction is then normalized. ; in, This represents the normalized reference comparison direction. This reference comparison direction is used to measure the consistency between the normalized evidence vectors of the two experts and the requirements of the current task.

[0047] The system calculates the directional consistency between the normalized evidence vectors of the two experts and the reference comparison direction, and generates the final selection weights: ; in, This indicates the selection of a weight generation function. The selection weight generation function It can be adopted Implementation of functions, normalized mapping functions, or learnable mapping functions. and They represent , The cosine similarity between c and 'c' is such that a larger value indicates a higher degree of directional consistency.

[0048] During the inference phase, the system selects the output of one of the experts as the final result based on the selection weights: ; in, This represents the final structured detection result.

[0049] The comparison selection process is mainly based on the directional relationship between the evidence vector and the reference comparison direction, in order to reduce the impact of differences in the output scale of different expert models on the comparison results.

[0050] Preferably, the result output module is used to convert the final detection result into a structured detection result in a unified format and output it. The structured detection result includes one or more of the following: target category, spatial location, region mask, status label, confidence level, timestamp, source expert identifier, and alarm information.

[0051] When the input is a single-frame image, the result output module outputs the detection result, classification result, segmentation result, recognition result, or state judgment result corresponding to the current image.

[0052] When the input is a multi-frame image sequence or video stream, the result output module performs time-series post-processing on the expert output of each frame or time period. The time-series post-processing includes one or more of the following: cross-frame target matching, trajectory association, state change analysis, abnormal duration judgment, or time-series consistency verification.

[0053] It should be noted that, as a preferred approach, the system employs a decoupled optimization strategy. During the training phase, the expert model parameters in the expert pool module are frozen, and training is only performed on the relevant parameters in the multimodal perception module, routing decision module, evidence normalization module, or hyperspherical reliability selection module. For newly added expert models, they can be trained independently before being added to the expert pool.

[0054] In single-expert execution mode, the routing decision module is trained under supervision based on the target expert labels, and its loss function is expressed as: ; in, represents the target expert label, and CE(·) represents the cross-entropy loss function.

[0055] In the fixed dual-expert collaborative comparison mode, the system trains relevant parameters based on task loss, and its loss function is expressed as: ; in, This indicates the reference label or reference result corresponding to the sample. Indicates mission loss. The entropy regularization term represents the selection weights. This represents the geometric constraint term between the normalized comparison representation and the reference evidence.

[0056] During the deployment phase, the system performs inference according to the working mode. In single-expert execution mode, the system activates a target expert to participate in the inference of the current sample based on the routing decision result; in fixed dual-expert collaborative comparison mode, the system calls two pre-configured experts to jointly participate in the inference of the current sample, and performs evidence normalization and hyperspherical reliability selection on the outputs of the two experts.

[0057] The following describes the task extension implementation method of the technical solution of the present invention in the single expert execution mode.

[0058] This implementation describes how to integrate a new expert model and complete the task execution without refactoring the existing system when adding a new single visual inspection task. The visual inspection task can be a specific task in industrial inspection, manufacturing quality inspection, security monitoring, or other inspection scenarios. The specific steps are as follows: Step S101: Connect a pre-trained task expert model for a specific detection target to the system, and configure the corresponding task identifier or expert identity information for the expert model to establish a mapping relationship between expert functions and detection tasks.

[0059] Step S102: Connect the task expert model to the pluggable expert pool module, while keeping the parameters of other existing expert models in the system unchanged. Specifically, keep the parameters of the multimodal perception module and existing expert models frozen, and only train the routing decision module, interface adaptation parameters, or scheduling parameters corresponding to newly added experts, so that the routing decision module can generate selection results for the task expert model based on input visual data and optional text instructions.

[0060] Step S103: During the inference phase, the routing decision module generates expert selection results based on the current input visual data and optional text instructions. In the single-expert execution mode, only one target expert model is activated to participate in the inference of the current sample, while the other expert models do not participate in the calculation of the current sample. The original detection results generated by the activated expert model are processed by the result output module to form a structured detection result in a unified format.

[0061] By following the steps above, new expert models can be added while keeping the existing system core unchanged, thus supporting the expansion of a single visual detection task.

[0062] The following describes the implementation of the technical solution of the present invention in a composite scenario under a fixed dual-expert collaborative comparison mode.

[0063] This implementation describes how to improve the stability of results on difficult samples by using a fixed dual-expert collaborative comparison mode in complex visual inspection scenarios. These complex visual inspection scenarios may include industrial inspection, manufacturing quality control, security monitoring, or traffic analysis. The specific steps are as follows: Step S201: During the system configuration phase, two expert models with complementary characteristics are pre-selected as a fixed dual-expert combination. For example, a detection expert and a segmentation expert, a detection expert and a state assessment expert, or a classification expert and an anomaly assessment expert can be selected as the dual-expert combination. During the inference phase, the two expert models receive shared visual representations, raw visual data, or expert input data adapted through an interface.

[0064] Step S202: In the fixed dual-expert collaborative comparison mode, the system does not perform arbitrary expert set search. Instead, it generates original detection results for the two pre-configured expert models respectively, and the evidence normalization module extracts the corresponding evidence vectors. The system also generates a reference comparison direction based on the control vector, the evidence vectors of the two experts, and the coordination result output by the routing decision module.

[0065] Step S203: The original detection results of each expert model are normalized and mapped to a unit hypersphere space. The system calculates the directional consistency between the two normalized evidence vectors and the reference comparison direction, and determines the expert output with higher reliability based on the directional consistency result; the result output module outputs a structured detection result in a unified format based on the comparison selection result.

[0066] Through the above steps, joint judgment can be made on detection targets in complex scenarios under a fixed dual-expert collaborative comparison mode, and the impact of differences in the output structure and confidence scale of different experts on the final result can be reduced.

[0067] The following describes the implementation of the technical solution of the present invention in the expert scheduling method guided by text instructions.

[0068] This implementation describes how the system controls the expert selection process based on external natural language instructions. The specific steps are as follows: Step S301: The operator inputs natural language instructions through an external interactive interface. The natural language instructions can be task description statements, detection target descriptions, rule-based operation instructions, or predefined identifier statements. The text encoding network performs semantic encoding on the instructions to obtain semantic embedding vectors representing the task intent.

[0069] Step S302: The system pre-defines expert identity descriptions corresponding to the functions of each expert model in the expert pool and obtains the corresponding expert identity embeddings. The expert identity embeddings can be obtained from expert function descriptions, task types, or predefined expert identifier encodings. Based on the matching relationship between the semantic embedding of the current instruction and the expert identity embeddings, the system generates expert priority results or text guidance distributions for the current instruction.

[0070] Step S303: The system guides the expert selection process of the routing decision module based on the expert priority results or text-guided distribution. In one embodiment, the target expert model is directly determined based on the expert priority results; in another embodiment, the expert priority results are used as additional control information and participate in the processing of the routing decision module together with the visual input to form the final expert selection result.

[0071] Step S304: The selected expert model performs current sample inference based on shared visual representation, raw visual data, or expert input data adapted through the interface, and generates raw detection results; after the raw detection results are processed by the evidence normalization module, they enter the hyperspherical reliability selection module for reliability comparison when needed, and finally the result output module outputs structured detection results that meet the requirements of the current instruction.

[0072] Through the above steps, the expert selection process can be guided by external natural language instructions without modifying the parameters of the existing expert model, thereby achieving controllable scheduling of visual inspection tasks or patrol tasks.

[0073] In addition, the present invention also provides a visual detection method for multiple scenarios that integrates semantic guidance and hybrid experts, implemented using the aforementioned visual detection system for multiple scenarios that integrates semantic guidance and hybrid experts, including the following steps: It receives input visual data and optional text instructions, and outputs a shared visual representation, an effective location mask, and an optional control vector. Receive the shared visual representation, effective location mask, and optional control vector, and generate expert selection results or dual-expert coordination results; Based on the expert selection results, the target expert is invoked to perform reasoning and generate the original detection results, i.e., the expert output results; The original detection results are converted into evidence vectors of a uniform dimension. The evidence vector is normalized and its reliability is compared based on the normalized evidence vector to determine the final expert output to be adopted, i.e., the comparison and selection result. Convert expert output results or comparison and selection results into structured detection results in a unified format.

[0074] Preferably, when visual data is received, the visual data can be a single-frame image, a continuously acquired image sequence, or a video stream; when the input is a single-frame image, spatial feature extraction is performed on the image; when the input is a video stream or a multi-frame image sequence, the video coding network in the multimodal perception module performs spatiotemporal joint coding on the visual data and outputs a video-level shared visual representation sequence containing inter-frame temporal correlation information. Visual data is first input into the visual encoding network in the multimodal perception module for feature extraction; the visual encoding network is used to generate shared visual representations that can be aligned with text instructions; Specifically, the multimodal perception module processes the input image. and optional text instructions Joint encoding is performed to obtain a shared visual representation, control vector, and corresponding mask. The process is as follows: ; in, This represents a multimodal coding network. Indicates shared visual representation, The effective location mask corresponding to the shared visual representation is represented by z, and the optional control vector derived from the shared visual representation is represented by z. The shared visual representation is used for subsequent expert selection and expert execution; specifically, the shared visual representation is passed to the routing decision module to generate expert selection results; at the same time, it is passed to the selected expert model to execute the current detection task and generate the original detection results.

[0075] Finally, it should be noted that the above embodiments are merely illustrative and explanatory of the present invention, and are not intended to limit the present invention to the scope of the described embodiments. Furthermore, those skilled in the art will understand that the present invention is not limited to the above embodiments, and many more variations and modifications can be made based on the teachings of the present invention, all of which fall within the scope of protection claimed by the present invention.

Claims

1. A visual inspection system integrating semantic guidance and hybrid experts for multiple scenarios, characterized in that, include: The multimodal perception module receives input visual data and optional text instructions, and outputs a shared visual representation, an effective location mask, and optional control vectors. The routing decision module is used to receive the shared visual representation, the effective location mask, and the optional control vector, and generate expert selection results or dual expert coordination results. A pluggable expert pool module is used to call the target expert to perform inference based on the expert selection result and generate the original detection result, i.e., the expert output result; The evidence normalization module is used to convert the original detection results into evidence vectors of a uniform dimension. The hyperspherical reliability selection module is used to normalize the evidence vector and perform reliability comparison based on the normalized evidence vector to determine the final expert output to be adopted, i.e., the comparison selection result. The results output module is used to convert expert output results or comparison and selection results into structured test results in a unified format.

2. The visual inspection system for multiple scenarios, integrating semantic guidance and hybrid experts, as described in claim 1, is characterized in that... include: During the execution of the multimodal perception module, when visual data is received, the visual data can be a single-frame image, a continuously acquired image sequence, or a video stream. When the input is a single-frame image, spatial feature extraction is performed on the image. When the input is a video stream or a multi-frame image sequence, the video coding network in the multimodal perception module performs spatiotemporal joint coding on the visual data and outputs a video-level shared visual representation sequence containing inter-frame temporal correlation information. Visual data is first input into the visual encoding network in the multimodal perception module for feature extraction; the visual encoding network is used to generate shared visual representations that can be aligned with text instructions; Specifically, the multimodal perception module processes the input image. and optional text commands Joint encoding is performed to obtain a shared visual representation, control vector, and corresponding mask. The process is as follows: ； in, This represents a multimodal coding network. Indicates shared visual representation, The effective location mask corresponding to the shared visual representation is represented by z, and the optional control vector derived from the shared visual representation is represented by z. The shared visual representation is used for subsequent expert selection and expert execution; specifically, the shared visual representation is passed to the routing decision module to generate expert selection results; at the same time, it is passed to the selected expert model to execute the current detection task and generate the original detection results.

3. The visual inspection system for multiple scenarios, incorporating semantic guidance and hybrid experts, as described in claim 2, is characterized in that... During the execution of the routing decision module, the routing decision module receives the shared visual representation output by the multimodal perception module. Valid location mask and optional control vectors generated during the text control encoding process. The routing decision module maps the above inputs to generate routing scores and expert selection distributions corresponding to each expert in the expert pool. Specifically, record Then the routing score and selection distribution are expressed as: ； in, Representative Order Equivalent to sharing visual representation , This represents the output of the coordination function after processing the shared visual representation, effective location mask, and control vector. and For learnable parameters, This represents the routing score for each expert. This indicates the distribution of expert choices; In single-expert execution mode, the expert with the highest weight is selected as the target expert according to the expert selection distribution: ； in, For experts to select distribution Middle The selection probability corresponding to each expert, with a value range of [0,1], and all... The sum is 1; The target expert then generates the original detection results for the current sample. : ； in, Indicates the first The input to an expert model can be shared visual representations, raw visual data, or expert input data adapted through an interface. Indicates the first An expert model; In the fixed dual-expert collaborative comparison mode, instead of searching for arbitrary expert sets, a coordination result is generated for two pre-configured experts. This coordination result is used for subsequent evidence normalization and hyperspherical reliability selection processing.

4. The visual inspection system for multiple scenarios, incorporating semantic guidance and hybrid experts, as described in claim 3, is characterized in that... During the execution of the routing decision module, according to the text instructions... In relation to the identities of the experts The matching relationship between them generates text-guided distribution : ； ； in, Indicates temperature parameter, Indicates the first A predefined identity embedding vector for an expert model; For the first The cosine similarity score of each expert with the current text instruction; To sum the subscripts; The text-guided distribution and the visual routing distribution are combined to form the final expert selection distribution. : ； in, This represents the visual route distribution obtained based on shared visual representations. Represents the fusion coefficient. The first term is given solely by the semantics of the text. The probability of expert selection.

5. A visual inspection system for multiple scenarios, incorporating semantic guidance and hybrid experts, as described in claim 4, is characterized in that... The pluggable expert pool module includes multiple expert models with different structures and output formats; During the reasoning process, the selected expert receives shared visual representations, raw visual data, or expert input data adapted through an interface, and generates raw detection results. Experts who are not selected do not participate in the reasoning process for the current sample.

6. A visual inspection system for multiple scenarios, incorporating semantic guidance and hybrid experts, as described in claim 5, is characterized in that... The evidence normalization module's evidence normalization process includes: (1) Receive the raw test results output by the selected expert; (2) Extract intermediate features corresponding to the original detection results; (3) Perform scaling transformation, recalibration, or compression on the confidence information in the original detection results; (4) Convert the original outputs of different experts into evidence vectors with a unified dimension; (5) Normalize the evidence vector to obtain a comparable unified evidence representation; For the selected experts The original test results The evidence is transformed into a uniform-dimensional evidence vector using an evidence transformation function. The formula is as follows: ； in, This represents the intermediate features corresponding to the original detection result. This represents the evidence transformation function. Subsequently, the evidence vector is normalized: ； in, This represents the normalized evidence vector. It is a stable term.

7. A visual inspection system for multiple scenarios, incorporating semantic guidance and hybrid experts, as described in claim 5, is characterized in that... The hyperspherical reliability selection module is used to map the evidence vector to a unit hyperspherical space after evidence normalization; the mapping method can be vector norm normalization or other unit length constrained projection methods. In a unit hyperspherical space, reliability comparison is performed based on normalized evidence vectors to determine the final expert output to be adopted; In the fixed dual-expert collaborative comparison mode, two experts are pre-configured. and Participate in the current sample inference; two experts generate the original detection results respectively. and The normalized evidence vector is obtained through the evidence normalization module. and ; Specifically, based on control vectors And the evidence vectors corresponding to the two experts generate the dual-expert coordination weights: ； in, This represents the coordination function between two experts. ; , Preset experts ,expert Specific feature representation; Subsequently, a reference comparison direction is formed based on the aforementioned coordination weights: ； The reference comparison direction is then normalized. ； in, Indicates the reference comparison direction after normalization; Calculate the directional consistency between the normalized evidence vectors of the two experts and the reference comparison direction, and generate the final selection weights: ； in, This indicates the selection of a weight generation function. ; To select a weight generation function; and They represent , The cosine similarity between c and , the larger the value, the higher the directional consistency; During the inference phase, the output of one of the experts is selected as the final result based on the selection weights: ； in, This represents the final structured detection result.

8. A visual inspection system for multiple scenarios, incorporating semantic guidance and hybrid experts, as described in claim 6, is characterized in that... The results output module is used to convert the final detection results into structured detection results in a unified format and output them; the structured detection results include one or more of the following: target category, spatial location, region mask, status label, confidence level, timestamp, source expert identifier, and alarm information; When the input is a single-frame image, the result output module outputs the detection result, classification result, segmentation result, recognition result or state judgment result corresponding to the current image; When the input is a multi-frame image sequence or video stream, the result output module performs time-series post-processing on the expert output of each frame or time period. The time-series post-processing includes one or more of the following: cross-frame target matching, trajectory association, state change analysis, abnormal duration judgment, or time-series consistency verification.

9. A visual inspection method for multiple scenarios that integrates semantic guidance and hybrid experts, implemented using the visual inspection system for multiple scenarios that integrates semantic guidance and hybrid experts as described in any one of claims 1-8, characterized in that, Includes the following steps: It receives input visual data and optional text instructions, and outputs a shared visual representation, an effective location mask, and an optional control vector. Receive the shared visual representation, effective location mask, and optional control vector, and generate expert selection results or dual-expert coordination results; Based on the expert selection results, the target expert is invoked to perform reasoning and generate the original detection results, i.e., the expert output results; The original detection results are converted into evidence vectors of a uniform dimension. The evidence vector is normalized and its reliability is compared based on the normalized evidence vector to determine the final expert output to be adopted, i.e., the comparison and selection result. Convert expert output results or comparison and selection results into structured detection results in a unified format.

10. A visual inspection method for multi-scenario fusion semantic guidance and hybrid experts as described in claim 9, characterized in that, When visual data is received, the visual data can be a single frame image, a continuously acquired image sequence, or a video stream. When the input is a single frame image, spatial feature extraction is performed on the image. When the input is a video stream or a multi-frame image sequence, the video coding network in the multimodal perception module performs spatiotemporal joint coding on the visual data and outputs a video-level shared visual representation sequence containing inter-frame temporal correlation information. Visual data is first input into the visual encoding network in the multimodal perception module for feature extraction; the visual encoding network is used to generate shared visual representations that can be aligned with text instructions; Specifically, the multimodal perception module processes the input image. and optional text commands Joint encoding is performed to obtain a shared visual representation, control vector, and corresponding mask. The process is as follows: ； in, This represents a multimodal coding network. Indicates shared visual representation, The effective location mask corresponding to the shared visual representation is represented by z, and the optional control vector derived from the shared visual representation is represented by z. The shared visual representation is used for subsequent expert selection and expert execution; specifically, the shared visual representation is passed to the routing decision module to generate expert selection results; at the same time, it is passed to the selected expert model to execute the current detection task and generate the original detection results.

Images