Inspection and detection visual data management method and system

Abstract

The invention discloses an inspection and detection visual data management method and system. The method comprises: receiving a query request; when the request contains an attribute filtering condition, submitting the request to a retrieval engine for retrieval to obtain a first result ID set; when the request contains a to-be-queried image, extracting a mixed feature vector of the to-be-queried image, and submitting the mixed feature vector to a special vector database for approximate nearest neighbor search to obtain a second result ID set; executing set operation on the first result ID set and the second result ID set to obtain a target ID set; and obtaining detection data from the transactional relational database according to the target ID set. According to the method, the retrieval efficiency is improved through a ternary hybrid storage architecture and a joint retrieval mechanism, image depth structuring and image search by images are realized through two-way feature extraction, the cost is reduced through Merkel tree batch evidence storage, predictive quality control is realized in combination with a knowledge graph and a graph neural network, the problems of data islands and retrieval bottlenecks are solved, and the retrieval efficiency is improved. And the intelligent level of data management is improved.

Description

Technical Field

[0001] This invention relates to the field of quality inspection, and in particular to a method and system for managing visual data in inspection and testing. Background Technology

[0002] In the testing and inspection industry, visual data (such as images and videos) is the core evidence carrier for recording sample status, defect characteristics, and operational processes. Its quality and management level directly affect the accuracy and reliability of test results. With the surge in testing tasks, the amount of visual data is growing exponentially, and the performance bottlenecks of existing technologies in data retrieval are becoming increasingly prominent.

[0003] In the process of developing the existing technology, the inventors discovered that: Traditional relational databases offer ACID transaction guarantees for storing structured business data, but their retrieval capabilities are inherently limited when dealing with unstructured data, primarily images. When performing fuzzy searches based on text fields such as image descriptions, defect labels, and measurement results, or queries involving multiple conditions such as time ranges, numerical intervals, and geographic coordinates, relational databases need to scan the entire table and perform complex join calculations, leading to a sharp decline in query performance. With tens of millions of data points, the response time for complex queries often reaches tens of seconds, failing to meet the real-world business requirements for sub-second response times.

[0004] Furthermore, existing systems can only perform searches based on file metadata, failing to conduct in-depth analysis of image content. Users cannot find historical similar defect cases through "image search," nor can they perform similarity matching based on defect features within images. When users need to perform joint searches based on both attribute conditions and image content, existing systems typically require executing queries separately in different systems and then manually integrating the results, resulting in low efficiency.

[0005] Therefore, this application provides a visual data management technology solution for inspection and testing with better relational database performance and higher joint retrieval efficiency to solve the problems of performance bottlenecks in data retrieval, lack of image content retrieval, and low efficiency of multi-dimensional joint retrieval in the prior art. Summary of the Invention

[0006] Therefore, this application provides a visual data management technology solution for inspection and testing with better relational database performance and higher joint retrieval efficiency to solve the problems of performance bottlenecks in data retrieval, lack of image content retrieval, and low efficiency of multi-dimensional joint retrieval in the prior art.

[0007] Specifically, the method for managing visual data in inspection and testing includes the following steps: Receive query requests; When the query request contains attribute filtering conditions, the attribute filtering conditions are submitted to the search engine for retrieval to obtain the first result ID set; When the query request contains the image to be queried, the mixed feature vector of the image to be queried is extracted, and the mixed feature vector is submitted to a dedicated vector database for approximate nearest neighbor search to obtain a second result ID set; Perform set operations on the first result ID set and the second result ID set to obtain the target ID set; Based on the target ID set, obtain detection data from a transactional relational database; The search engine is pre-indexed with text descriptions, defect tags, numerical measurement results, and geographic coordinates for full-text search and aggregated queries. The dedicated vector database pre-stores hybrid feature vectors extracted from historical images by a cloud-based multimodal analysis pipeline for approximate nearest neighbor search; The transactional relational database pre-stores detection business data and the relationships between the detection business data. Data change events in the transactional relational database are synchronized in real time to the retrieval engine and the dedicated vector database through change data capture and event-driven architecture.

[0008] Furthermore, the retrieval API gateway receives the query request, parses the query request into a structured query request body, sends the attribute filtering conditions to the retrieval engine, and sends the image to be queried to the dedicated vector database.

[0009] Furthermore, the search engine is Elasticsearch, and the attribute filtering conditions include one or more combinations of time range, defect type, numerical measurement range, and geographic coordinates.

[0010] Furthermore, extracting the blended feature vector of the image to be queried includes: Use the Visual Transformer to extract the global contextual semantic embedding vector of the image; Use a manual feature extractor to extract interpretable engineering feature vectors; The semantic embedding vector and the engineering feature vector are adaptively and weightedly fused through an attention mechanism fusion layer to generate a hybrid feature vector.

[0011] Furthermore, the set operations include one or more combinations of intersection, union, and difference operations.

[0012] Furthermore, the transactional relational database uses JSONB fields to store dynamic metadata; the change data capture is implemented using the Debezium tool.

[0013] Furthermore, before receiving a query request, data acquisition and data preprocessing are also included. The data acquisition includes: creating a detection task on the interactive management platform and binding it to a digital detection procedure; sending the detection task to the terminal; after the terminal loads the detection task, it starts the visual servo guidance engine to identify the sample to be inspected and the reference object in the image in real time, and extracts the contours of the sample to be inspected and the reference object; dynamically generating an ideal posture model according to the digital detection procedure, calculating the transformation matrix between the extracted contour and the ideal posture model, and rendering a guidance animation to guide the user to move to the target shooting pose; after the user triggers the shooting command, images, spatial information, process traceability data, and environmental context data are collected simultaneously to form a reliable data packet. The data preprocessing includes: parsing the trusted data packet, extracting images, spatial information, process traceability data and environmental context data from the trusted data packet, and storing them as detection business data and the association between the detection business data in the transactional relational database; simultaneously storing the hybrid feature vector generated by feature extraction of the images in the trusted data packet in the dedicated vector database; and storing the text description, defect label and numerical measurement results in the trusted data packet in the retrieval engine. The data in the transactional relational database, the search engine, and the dedicated vector database constitute the retrieval objects of the query request.

[0014] Furthermore, the process between data acquisition and data preprocessing also includes: Deploy convolutional neural networks on the terminal; After the image in the trusted data packet is captured, the image is input into the convolutional neural network on the terminal to perform composition compliance detection, image quality assessment and defect area detection in parallel, and provide real-time feedback and decision based on the rule engine; The quality inspection data is anonymized locally on the terminal, and the model weight gradient is uploaded to the federated learning server when preset conditions are met. The federated learning server performs gradient aggregation, and the aggregated global model is periodically distributed to each terminal.

[0015] Furthermore, after obtaining the detection data from the transactional relational database based on the target ID set, the process also includes: Acquire several trusted data packets formed during the data collection, calculate the hash value of each trusted data packet, and use the hash value as a leaf node to construct a Merkle tree to generate a unique Merkle root hash; The Merkle root hash is written into the blockchain via a smart contract to achieve batch aggregation and storage of evidence.

[0016] Furthermore, after obtaining the detection data from the transactional relational database based on the target ID set, the process also includes: Natural language processing is used to extract detection business data and the relationships between detection business data from the detection report data in the transactional relational database, and the data is stored in a graph database to construct a quality knowledge graph. A graph neural network algorithm is run on the quality knowledge graph to perform root cause analysis and quality prediction.

[0017] This application also provides a visual data management system for inspection and testing, including: The terminal is configured as follows: after creating a detection task and binding a digital detection procedure on the interactive management platform, it loads the detection task, starts the visual servo guidance engine, identifies the sample to be inspected and the reference object in the image in real time, and extracts the contours of the sample to be inspected and the reference object; it dynamically generates an ideal posture model according to the digital detection procedure, calculates the transformation matrix between the extracted contour and the ideal posture model, and renders a guidance animation to guide the user to move to the target shooting pose; after the user triggers the shooting command, it synchronously collects images, spatial information, process traceability data and environmental context data to form a reliable data packet. An edge quality inspection module, deployed on the terminal, is configured to: after capturing an image in the trusted data packet, input the image into a convolutional neural network locally on the terminal, and perform parallel operations on composition compliance detection, image quality assessment, and defect region detection, providing real-time feedback and decision-making based on a rule engine; the edge quality inspection module is also configured to: anonymize the quality inspection data locally on the terminal, upload the model weight gradient to the federated learning server when preset conditions are met, and receive the global model issued by the federated learning server after aggregation and updating; The cloud-based analytics system adopts a cloud-native microservice architecture, including: The data preprocessing module is configured to parse the trusted data packet and extract the image, spatial information, process traceability data and environmental context data from the trusted data packet; The hybrid storage engine includes: a transactional relational database for storing the extracted spatial information, process traceability data, and environmental context data as detection business data and the relationships between the detection business data; a retrieval engine for storing the extracted text descriptions, defect labels, and numerical measurement results corresponding to the images; a dedicated vector database for storing the hybrid feature vectors generated by feature extraction from the extracted images; and object storage. The multimodal analysis pipeline is configured to quantify defects in received trusted data packets and extract hybrid feature vectors of images through a dual-path feature extraction network. The data synchronization module is configured to adopt a change data capture and event-driven architecture to synchronize data change events in the transactional relational database to the retrieval engine and the dedicated vector database in real time. The knowledge discovery engine is configured to: receive query requests; when the query request contains attribute filtering conditions, submit the attribute filtering conditions to the retrieval engine for retrieval to obtain a first result ID set; when the query request contains a query image, extract the mixed feature vector of the query image, submit the mixed feature vector to the dedicated vector database for approximate nearest neighbor search to obtain a second result ID set; perform set operations on the first result ID set and / or the second result ID set to obtain a target ID set; and obtain detection detail data from the transactional relational database based on the target ID set. The knowledge discovery engine is further configured to: extract detection business data and the relationships between the detection business data from the detection report data in the transactional relational database using natural language processing, store them in a graph database, construct a quality knowledge graph, and run a graph neural network algorithm on the quality knowledge graph to perform root cause analysis and quality prediction; The blockchain evidence storage module is configured to: acquire several trusted data packets parsed by the data preprocessing module, calculate the hash value of each trusted data packet, and use the hash value as a leaf node to construct a Merkle tree to generate a unique Merkle root hash; and write the Merkle root hash into the blockchain through a smart contract to achieve batch aggregation evidence storage.

[0018] This application addresses the source data quality issue through guided data collection; it achieves sub-second response times for complex queries on datasets with tens of millions of images using a ternary hybrid storage architecture (in this embodiment, on a dataset containing 10 million images, for the complex query "defect type = scratch AND length > 0.5mm AND shooting time = past 30 days", the average response time of a pure MySQL database is approximately 15 seconds, while the average response time using the method of this invention is approximately 600 milliseconds, representing a 25-fold improvement in retrieval efficiency); it achieves deep structuring and similarity search of image content through dual-path feature extraction; it solves the cost problem of reliable evidence storage for massive amounts of data through Merkle tree batch evidence storage; and it achieves predictive quality control through knowledge graphs and GNNs. The combined implementation of the above technical effects demonstrates the significant progress of this invention in solving the technical problems of visual data management in inspection and testing. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1A flowchart of the inspection and testing visual data management method provided in the embodiments of this application.

[0020] Figure 2 This is a schematic diagram of the inspection and testing visual data management system provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] This application provides a method for managing visual data in inspection and testing, which is based on... Figure 2 The China Inspection and Testing Visual Data Management System is implemented, comprising: a terminal, an edge quality inspection module, and a cloud-based analysis system. The cloud-based analysis system adopts a cloud-native microservice architecture, including a data preprocessing module, a hybrid storage engine, a multimodal analysis pipeline, a data synchronization module, a knowledge discovery engine, and a blockchain-based evidence storage module.

[0023] Specifically, to better understand this method, this application describes the invention in further detail as "incoming inspection and testing of precision bearings," but the scope of protection of the invention is not limited to this embodiment.

[0024] For details, please refer to Figure 1 The visual data management method for inspection and testing provided in this application specifically includes the following steps: S101: Receive query request.

[0025] Specifically, users initiate query requests through the interactive management platform. For example, when a testing engineer reports a quality issue on the precision bearing assembly line, they can enter "bearing scratch length > 0.1mm, Q3 2026" into the platform and upload a picture of a suspected defect.

[0026] S102: When the query request contains attribute filtering conditions, the attribute filtering conditions are submitted to the search engine for retrieval to obtain the first result ID set.

[0027] When a query request includes attribute filtering conditions, these conditions are submitted to the search engine for retrieval, yielding a first set of result IDs. In this embodiment, the search engine uses Elasticsearch, which is pre-indexed with text descriptions (e.g., "bearing surface scratches"), defect tags (e.g., "scratches" and "dents"), numerical measurement results (e.g., "length: 0.12mm"), and geographic coordinates (e.g., GPS coordinates of the collection point). When the attribute filtering condition "defect type = scratch AND length > 0.1mm AND shooting time = Q3 2026" is received, the search engine utilizes inverted indexes and numerical range indexes to complete the retrieval within milliseconds, returning a set of image IDs that meet the conditions.

[0028] S103: When the query request contains an image to be queried, extract the mixed feature vector of the image to be queried, submit the mixed feature vector to a dedicated vector database for approximate nearest neighbor search, and obtain a second result ID set.

[0029] When a query request contains the image to be queried, the system extracts the mixed feature vector of the image and submits it to a dedicated vector database for approximate nearest neighbor search to obtain a second set of result IDs. In this embodiment, the dedicated vector database is Milvus, which pre-stores mixed feature vectors extracted from historical images by a cloud-based multimodal analysis pipeline. When a user uploads a suspected defective image, the system extracts the mixed feature vector (1024 dimensions) of the image, submits it to Milvus for approximate nearest neighbor search, and returns a set of IDs for the 100 historical cases most similar to the image to be queried.

[0030] It should be noted that S102 and S103 are executed in parallel, not sequentially.

[0031] S104: Perform set operations on the first result ID set and the second result ID set to obtain the target ID set.

[0032] Set operations are performed on the first result ID set and the second result ID set to obtain the target ID set. In this embodiment, the user needs to satisfy both attribute conditions and image similarity, so an intersection operation is used to obtain a set of image IDs that both meet the attribute filtering conditions and are similar to the image to be queried.

[0033] S105: Obtain detection data from the transactional relational database based on the target ID set.

[0034] Based on the target ID set, the system retrieves detection data from a transactional relational database. In this embodiment, the transactional relational database is PostgreSQL, which pre-stores detection business data and the relationships between detection business data, including user information, detection tasks, detection results, operation records, reports, etc. Based on the target ID set, the system retrieves complete detection details data (such as original reports, operation logs, environmental parameters, etc.) from PostgreSQL and returns it to the user.

[0035] The above method solves the problem of poor retrieval performance of a single relational database when dealing with massive amounts of unstructured data by separating attribute retrieval from image retrieval. Attribute retrieval is handled by the retrieval engine, while image retrieval is handled by a dedicated vector database. Finally, the results are fused through set operations. This solves the problem of low efficiency in joint retrieval of multi-dimensional queries by converting image content into hybrid feature vectors and storing them in a vector database.

[0036] Furthermore, the retrieval API gateway receives the query request, parses the query request into a structured query request body, sends the attribute filtering conditions to the retrieval engine, and sends the image to be queried to the dedicated vector database.

[0037] Specifically, the retrieval API gateway serves as a unified query entry point, receiving HTTP requests initiated by users. The gateway parses the request body, identifying attribute filtering conditions (such as JSON-formatted query parameters) and the image to be queried (such as an image file or Base64 encoded image). For attribute filtering conditions, the gateway forwards them to the Elasticsearch cluster's query interface; for the image to be queried, the gateway forwards them to the Milvus vector database's search interface. This unified access and separate routing mechanism avoids the cumbersome operation of users having to perform queries on different systems separately.

[0038] Furthermore, the search engine is Elasticsearch, and the attribute filtering conditions include one or more combinations of time range, defect type, numerical measurement range, and geographic coordinates.

[0039] In this embodiment, the Elasticsearch cluster is deployed in the cloud, and an index named "inspection_index" is created. The mapping for this index includes the following fields: Text description field: Employs a standard analyzer and supports full-text fuzzy search; Defect tag field: Uses keyword type, supports exact matching and aggregate statistics; Numerical measurement result field: uses float type, supports range query (e.g., "length > 0.1mm"); Geographic coordinates field: uses the geo_point type and supports geospatial queries (e.g., "within 10km of a certain point"); Timestamp field: uses date type and supports time range queries; When a user queries "Defect type = scratch AND length > 0.1mm AND shooting time = Q3 2026", Elasticsearch uses an inverted index to quickly locate the "scratch" tag and a BKD tree index to quickly locate the numerical and date ranges, achieving a millisecond-level response. Compared to the full table scan of traditional relational databases, the retrieval efficiency is improved by tens of times.

[0040] Furthermore, extracting the hybrid feature vector of the image to be queried includes: extracting the global context semantic embedding vector of the image using a visual Transformer; extracting interpretable engineered feature vectors using a hand-crafted feature extractor; and adaptively weighting and fusing the semantic embedding vector and the engineered feature vector through an attention mechanism fusion layer to generate a hybrid feature vector.

[0041] Specifically, in this embodiment, the cloud-based multimodal analysis pipeline runs a dual-path feature extraction network. The semantic feature path uses a pre-trained ViT-B / 16 model to segment the image into 16×16 patches, extracts global context semantic embedding vectors through a Transformer encoder, and outputs a 768-dimensional feature vector. This vector can capture the semantic information of the image, such as "this is the end face of a bearing" or "there is a scratch here".

[0042] The engineering feature path employs a handcrafted feature extractor, computing the following features in parallel: Improved Rotation Invariant Uniform Layer Backpropagation (RIU-LBP): extracts local texture patterns from the image, outputting a 256-dimensional feature vector; Histogram of Oriented Gradients (PHOG): extracts multi-scale edge orientation distributions from the image, outputting a 128-dimensional feature vector; Gabor filter-based texture energy: extracts texture energy responses through Gabor filters of different orientations and scales, outputting a 96-dimensional feature vector; The attention mechanism fusion layer adaptively weights and fuses the semantic embedding vector (768-dimensional) and the engineering feature vector (480-dimensional) using learnable weight parameters, generating a unified hybrid feature vector (1024-dimensional). This hybrid feature vector captures both the global semantic information of the image and preserves interpretable features such as local texture and shape, thereby improving the accuracy of similarity search.

[0043] Furthermore, the set operations include one or more combinations of intersection, union, and difference operations.

[0044] In this embodiment, the knowledge discovery engine supports multiple set operation modes: When users need to satisfy both attribute conditions and image similarity (such as "find all cases where the scratch length is >0.1mm and is similar to the uploaded image"), the intersection operation is used: Result = Set1∩ Set2; When users need to meet attribute conditions or image similarity (such as "find all cases with scratch length > 0.1mm, or cases similar to the uploaded image"), the union operation is used: Result = Set1∪ Set2; When users need to exclude certain conditions (such as "finding cases where the scratch length is >0.1mm but is not similar to the uploaded image"), the difference operation is used: Result = Set1 - Set2; this flexible set operation mechanism allows users to customize search strategies according to their actual needs.

[0045] Furthermore, the transactional relational database uses JSONB fields to store dynamic metadata; change data capture is achieved through the Debezium tool.

[0046] In this embodiment, the PostgreSQL database uses JSONB fields to store dynamic metadata in the detection task. For example, different detection procedures may generate different additional fields (such as "temperature", "humidity", "operator"). These fields do not require a predefined table structure and are stored directly in JSONB format, supporting dynamic expansion and efficient querying.

[0047] The data synchronization module uses Debezium as the change data capture tool. Debezium monitors the PostgreSQL WAL log, captures data change events (INSERT, UPDATE, DELETE), and publishes these events to a Kafka message queue. A separate index building service consumes Kafka messages, performs data enrichment and format conversion, and then writes the data to Elasticsearch and Milvus respectively. This asynchronous event-driven synchronization mechanism ensures eventual consistency between the transactional relational database and the search engine and vector database, while avoiding the impact of synchronous writes on business performance.

[0048] Furthermore, before receiving a query request, data acquisition and data preprocessing are also included. The data acquisition includes: creating a detection task on the interactive management platform and binding it to a digital detection procedure; sending the detection task to the terminal; after the terminal loads the detection task, it starts the visual servo guidance engine to identify the sample to be inspected and the reference object in the image in real time, and extracts the contours of the sample to be inspected and the reference object; dynamically generating an ideal posture model according to the digital detection procedure, calculating the transformation matrix between the extracted contour and the ideal posture model, and rendering a guidance animation to guide the user to move to the target shooting pose; after the user triggers the shooting command, images, spatial information, process traceability data, and environmental context data are collected simultaneously to form a reliable data packet. The data preprocessing includes: parsing the trusted data packet, extracting images, spatial information, process traceability data and environmental context data from the trusted data packet, and storing them as detection business data and the association between the detection business data in the transactional relational database; simultaneously storing the hybrid feature vector generated by feature extraction of the images in the trusted data packet in the dedicated vector database; and storing the text description, defect label and numerical measurement results in the trusted data packet in the retrieval engine. The data in the transactional relational database, the search engine, and the dedicated vector database constitute the retrieval objects of the query request.

[0049] Specifically, the inspection engineer creates an inspection task on the interactive management platform, selecting the "Precision Bearing Incoming Inspection" digital inspection procedure. This procedure includes state machine logic: the "local macro" shooting step is only unlocked after the "overall appearance" photo is taken and passes quality inspection. The task package is encrypted and sent to the terminal via the network. The terminal can be an augmented reality terminal, such as an AR tablet.

[0050] After the terminal loads the detection task, it starts the visual servoing guidance engine. This engine runs an instance segmentation model, such as the improved MobileNetV3-DeepLabV3+. It identifies the sample to be inspected (precision bearing) and standard reference objects (such as rulers and color charts) in the image in real time and extracts the contours of the sample and the reference objects. The visual servoing guidance engine can be understood as an adaptive visual servoing guidance engine.

[0051] Based on predefined shooting parameters (such as shooting distance, angle, and reference object position) in the digital inspection procedure, the engine dynamically generates an ideal pose model. This model describes the ideal position, size, and orientation of the sample in the image. The engine calculates the transformation matrix (including translation, rotation, and scaling) between the extracted contour and the ideal pose model, and dynamically renders guiding animations (such as arrows, semi-transparent contours, and distance cues) using the visual inertial odometry capability of the AR SDK to guide the user's mobile terminal to the target shooting pose.

[0052] After the user triggers the shooting command, the system uses hardware-level synchronization signals or high-precision software timestamps (such as the PTP protocol) to ensure that data from the image sensor, GPS / BeiDou module, IMU, and external Bluetooth sensors (temperature, humidity, illuminance meter) are collected and labeled under the same absolute time reference, forming a trusted data packet. In addition to the image, this trusted data packet also contains spatial information (local high-precision coordinates generated by fusing GPS, BeiDou, Wi-Fi fingerprint, and visual SLAM), process traceability data (each decision state of the guidance engine, user interaction events), environmental context (temperature: -20℃~60℃, humidity: 0%RH~100%RH, illuminance: 0~100,000 Lux), and device attitude.

[0053] In the data preprocessing, defects in the image are quantified through a multimodal analysis pipeline. For example, a subpixel-level edge detection algorithm such as Canny-Zernike moments combined with scale recognition is used to achieve super-resolution measurement of defect size. A dual-path feature extraction network is used to extract the mixed feature vector of the image, and the mixed feature vector is stored in a dedicated vector database. The text description, defect label, and numerical measurement results in the trusted data packet are stored in the retrieval engine.

[0054] By using AR guidance and visual servo control, the problem of uncontrolled quality at the source of data acquisition is solved, ensuring the standardization and normalization of image acquisition; by using multi-sensor spatiotemporal synchronous acquisition, the problem of the inability to correlate metadata and image data is solved; by using data preprocessing steps to transform unstructured images into structured data and store them in three major storage components, the problem of data silos is solved.

[0055] Furthermore, between data acquisition and data preprocessing, the process includes: deploying a convolutional neural network on the terminal; after capturing images from the trusted data packet, inputting the images into the convolutional neural network locally on the terminal, and performing parallel operations such as composition compliance detection, image quality assessment, and defect region detection, providing real-time feedback and decision-making based on a rule engine; anonymizing the quality inspection data locally on the terminal, and uploading the model weight gradient to the federated learning server when preset conditions are met; the federated learning server performs gradient aggregation, and the aggregated global model is periodically distributed to each terminal.

[0056] In this embodiment, a convolutional neural network (Tiny-YOLOv5) optimized using knowledge distillation and quantization techniques is deployed on the terminal. After the image is captured from the trusted data packet, it is input into the convolutional neural network locally on the terminal to perform the following tasks in parallel: Composition compliance check: The integrity is judged by calculating the IoU (Intersection over Union) between the reference object segmentation mask and the preset region (threshold default >90%). Image quality assessment: Calculate no-reference image sharpness metrics (such as Brenner gradient, Laplacian variance) and noise estimation; Defect region detection: Outputs the bounding box and preliminary classification of defects, along with an uncertainty score; The failure level is determined based on a rule engine (such as Drools), and corresponding prompts are triggered. For critical items (such as missing references), the process is forcibly locked; for minor items (such as slight blur), a reshoot is strongly recommended.

[0057] The quality inspection data is anonymized locally on the terminal, including removing sensitive information (such as bearing serial numbers and QR codes) from images and performing image desensitization transformations. When preset conditions are met (such as device idle time, Wi-Fi connection, and sufficient battery power >30%), the model weight gradients are uploaded to the federated learning server. The federated learning server uses a secure aggregation protocol for gradient aggregation and integrates anomaly detection mechanisms (such as calculating gradient norm and cosine similarity) to reduce the aggregation weight of gradients that deviate from historical distributions or to exclude them. The aggregated global model is periodically distributed to each terminal, enabling continuous evolution of the terminal model.

[0058] By using edge quality inspection, the detection results are fed back in real time at the collection end, solving the problem of low efficiency and the need for manual interpretation in the traditional mode; by using the federated learning framework, the continuous evolution of the edge model is realized, solving the problem of insufficient data from a single terminal and the inability of the model to adapt to new defect patterns; by using anonymization processing and gradient uploading instead of raw data uploading, the problem of data privacy protection is solved.

[0059] Furthermore, after obtaining detection data from the transactional relational database based on the target ID set, the method further includes: obtaining several trusted data packets formed during the data collection, calculating the hash value of each trusted data packet, and using the hash value as a leaf node to construct a Merkle tree to generate a unique Merkle root hash; and writing the Merkle root hash into the blockchain through a smart contract to achieve batch aggregation and evidence storage.

[0060] Specifically, several trusted data packets generated during the data collection process are acquired (e.g., packaged every hour, with approximately 100 trusted data packets each time). The hash value of each trusted data packet is calculated (using the SHA-256 algorithm), and the hash values ​​are used as leaf nodes to construct a Merkle tree, generating a unique Merkle root hash. The minimum information, such as the Merkle root hash, batch time range, and brief description, is written into a permissioned blockchain (such as Hyperledger Fabric) via a smart contract to achieve batch aggregation and notarization.

[0061] When verifying the authenticity of a test report, the verifier provides the data packet to be verified and its Merkel proof path (the hashes of all sibling nodes along the root hash path from the data hash). This is then recombined with the root hash stored on the blockchain to complete the verification, eliminating the need to download the entire batch of data. The Merkel root hashes corresponding to test records of the same sample at different time points are linked chronologically on the blockchain, forming a complete chain of evidence.

[0062] By constructing a Merkle tree from the hash values ​​of a batch of data packets and only putting the root hash on the chain, the problem of high cost and low efficiency of putting massive amounts of data on the chain one by one is solved; the problem of fast verification of a single piece of data is solved by the Merkle proof path; and the legal validity of the credible evidence of the detection data is solved by the immutability of the blockchain.

[0063] Furthermore, after obtaining the detection data from the transactional relational database based on the target ID set, the method further includes: using natural language processing to extract the detection business data and the correlation between the detection business data from the detection report data in the transactional relational database, storing them in a graph database, and constructing a quality knowledge graph; and running a graph neural network algorithm on the quality knowledge graph to perform root cause analysis and quality prediction.

[0064] In this embodiment, the knowledge discovery engine utilizes Natural Language Processing (NLP) technology to extract inspection business data and the relationships between inspection business data from inspection report data in a transactional relational database. For example, from the report text "Precision bearing type A, surface scratch found, length 0.12mm, manufactured by supplier B, machined using lathe C," the entities "bearing type A," "scratch," "supplier B," and "lathe C," as well as the relationships "bearing type A has a scratch," "bearing type A is provided by supplier B," and "bearing type A is machined using lathe C," are extracted. These entities and relationships are stored in a graph database (such as Neo4j) to construct a quality knowledge graph.

[0065] The system runs a graph neural network algorithm on the quality knowledge graph to learn complex relationship patterns between entities. When a new quality problem arises, the system automatically navigates and mines within the graph to find historical nodes that are closely related to the current problem node in terms of topological structure. For example, when a batch of bearings shows scratch defects again, the GNN algorithm analysis finds that the defect is highly correlated with nodes such as "supplier B," "lathe C," "temperature > 25℃," and "humidity > 60%," indicating potential root causes in a probabilistic form (e.g., "the scratch defect rate of bearings supplied by supplier B under high temperature and high humidity conditions has increased significantly"), thus achieving predictive quality control.

[0066] By constructing a quality knowledge graph, scattered detection data is transformed into a structured knowledge network, solving the problem of historical cases being difficult to reuse. Root cause analysis is performed on the graph using graph neural network algorithms, solving the problems of tracing and predicting quality issues, and realizing an intelligent upgrade from "post-event detection" to "in-event prevention" and "pre-event prediction".

[0067] Please refer to Figure 2 This application also provides an inspection and testing visual data management system for implementing the above method, wherein the system specifically includes: The terminal is configured as follows: after creating a detection task and binding a digital detection procedure on the interactive management platform, it loads the detection task, starts the visual servo guidance engine, identifies the sample to be inspected and the reference object in the image in real time, and extracts the contours of the sample to be inspected and the reference object; it dynamically generates an ideal posture model according to the digital detection procedure, calculates the transformation matrix between the extracted contour and the ideal posture model, and renders a guidance animation to guide the user to move to the target shooting pose; after the user triggers the shooting command, it synchronously collects images, spatial information, process traceability data and environmental context data to form a reliable data packet.

[0068] An edge quality inspection module, deployed on the terminal, is configured to: after capturing an image in the trusted data packet, input the image into a convolutional neural network locally on the terminal, and perform parallel operations on composition compliance detection, image quality assessment, and defect region detection, providing real-time feedback and decision-making based on a rule engine; the edge quality inspection module is also configured to: anonymize the quality inspection data locally on the terminal, upload the model weight gradient to the federated learning server when preset conditions are met, and receive the global model issued by the federated learning server after aggregation and updating.

[0069] The cloud-based analytics system adopts a cloud-native microservice architecture, including: The data preprocessing module is configured to parse the trusted data packet and extract the image, spatial information, process traceability data and environmental context data from the trusted data packet; The hybrid storage engine includes: a transactional relational database (PostgreSQL) for storing the extracted spatial information, process traceability data, and environmental context data as detection business data and the relationships between detection business data; a retrieval engine (Elasticsearch) for storing the extracted text descriptions, defect labels, and numerical measurement results corresponding to the images; a dedicated vector database (Milvus) for storing the hybrid feature vectors generated after feature extraction from the extracted images; and object storage (MinIO). The multimodal analysis pipeline is configured to quantify defects in received trusted data packets and extract hybrid feature vectors of images through a dual-path feature extraction network. The data synchronization module is configured to use change data capture (Debezium) and an event-driven architecture (Kafka) to synchronize data change events in the transactional relational database to the retrieval engine and the dedicated vector database in real time. The knowledge discovery engine is configured to: receive query requests; when the query request contains attribute filtering conditions, submit the attribute filtering conditions to the retrieval engine for retrieval to obtain a first result ID set; when the query request contains a query image, extract the mixed feature vector of the query image, submit the mixed feature vector to the dedicated vector database for approximate nearest neighbor search to obtain a second result ID set; perform set operations on the first result ID set and / or the second result ID set to obtain a target ID set; and obtain detection detail data from the transactional relational database based on the target ID set. The knowledge discovery engine is also configured to: extract detection business data and the relationships between detection business data from the detection report data in the transactional relational database using natural language processing, store them in a graph database (Neo4j), construct a quality knowledge graph, and run a graph neural network algorithm on the quality knowledge graph to perform root cause analysis and quality prediction; The blockchain evidence storage module is configured to: acquire several trusted data packets parsed by the data preprocessing module, calculate the hash value of each trusted data packet, and use the hash value as a leaf node to construct a Merkle tree to generate a unique Merkle root hash; and write the Merkle root hash into the blockchain (Hyperledger Fabric) through a smart contract to achieve batch aggregation evidence storage.

[0070] This embodiment solves the source data quality problem through guided acquisition; it achieves sub-second response times for complex queries on datasets with tens of millions of images through a ternary hybrid storage architecture (in this embodiment, on a dataset containing 10 million images, for the complex query "defect type = scratch AND length > 0.5mm AND shooting time = past 30 days", the average response time of a pure MySQL database is about 15 seconds, while the average response time using the method of this invention is about 600 milliseconds, improving retrieval efficiency by 25 times); it achieves deep structuring and similarity search of image content through dual-path feature extraction; it solves the cost problem of reliable evidence storage for massive amounts of data through Merkle tree batch evidence storage; and it achieves predictive quality control through knowledge graphs and GNNs. The combined effect of the above technologies demonstrates the significant progress of this invention in solving the technical problems of visual data management in inspection and testing.

[0071] This application provides a computer device, characterized in that it includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the inspection and detection visual data management method.

[0072] This application also provides a computer-readable storage medium storing computer-readable instructions thereon, which, when executed by a computer's processor, cause the computer to perform the method in the inspection and detection visual data management method.

[0073] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0074] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for managing visual data in inspection and testing, characterized in that, Includes the following steps: Receive query requests; When the query request contains attribute filtering conditions, the attribute filtering conditions are submitted to the search engine for retrieval to obtain the first result ID set; When the query request contains the image to be queried, the mixed feature vector of the image to be queried is extracted, and the mixed feature vector is submitted to a dedicated vector database for approximate nearest neighbor search to obtain a second result ID set; Perform set operations on the first result ID set and the second result ID set to obtain the target ID set; Based on the target ID set, obtain detection data from a transactional relational database; The search engine is pre-indexed with text descriptions, defect tags, numerical measurement results, and geographic coordinates for full-text search and aggregated queries. The dedicated vector database pre-stores hybrid feature vectors extracted from historical images by a cloud-based multimodal analysis pipeline for approximate nearest neighbor search; The transactional relational database pre-stores detection business data and the relationships between the detection business data. Data change events in the transactional relational database are synchronized in real time to the retrieval engine and the dedicated vector database through change data capture and event-driven architecture.

2. The inspection and testing visual data management method according to claim 1, characterized in that, The retrieval API gateway receives the query request, parses the query request into a structured query request body, sends the attribute filtering conditions to the retrieval engine, and sends the image to be queried to the dedicated vector database.

3. The inspection and testing visual data management method according to claim 1, characterized in that, The search engine is Elasticsearch, and the attribute filtering conditions include one or more combinations of time range, defect type, numerical measurement range, and geographic coordinates.

4. The inspection and testing visual data management method according to claim 1, characterized in that, Extracting the blended feature vector of the image to be queried includes: Use the Visual Transformer to extract the global contextual semantic embedding vector of the image; Use a manual feature extractor to extract interpretable engineering feature vectors; The semantic embedding vector and the engineering feature vector are adaptively and weightedly fused through an attention mechanism fusion layer to generate a hybrid feature vector.

5. The inspection and testing visual data management method according to claim 1, characterized in that, The set operations include one or more combinations of intersection, union, and difference operations.

6. The inspection and testing visual data management method according to claim 1, characterized in that, The transactional relational database uses JSONB fields to store dynamic metadata; the change data capture is achieved through the Debezium tool.

7. The inspection and testing visual data management method according to claim 1, characterized in that, Before receiving a query request, data collection and data preprocessing are also included; The data collection includes: Create testing tasks on the interactive management platform and bind them to digital testing procedures; The detection task is sent to the terminal; After the terminal loads the detection task, it starts the visual servo guidance engine to identify the sample to be inspected and the reference object in the picture in real time and extract the outline of the sample to be inspected and the reference object. The ideal pose model is dynamically generated according to the digital detection procedure, the transformation matrix between the extracted contour and the ideal pose model is calculated, and a guiding animation is rendered to guide the user to move to the target shooting pose. After the user triggers the shooting command, images, spatial information, process traceability data and environmental context data are collected simultaneously to form a reliable data packet; The data preprocessing includes: parsing the trusted data packet, extracting images, spatial information, process traceability data and environmental context data from the trusted data packet, and storing them as detection business data and the association between the detection business data in the transactional relational database; simultaneously storing the hybrid feature vector generated by feature extraction of the images in the trusted data packet in the dedicated vector database; and storing the text description, defect label and numerical measurement results in the trusted data packet in the retrieval engine. The data in the transactional relational database, the search engine, and the dedicated vector database constitute the retrieval objects of the query request.

8. The inspection and testing visual data management method according to claim 7, characterized in that, Between data acquisition and data preprocessing, there is also: Deploy convolutional neural networks on the terminal; After the image in the trusted data packet is captured, the image is input into the convolutional neural network on the terminal to perform composition compliance detection, image quality assessment and defect area detection in parallel, and provide real-time feedback and decision based on the rule engine; The quality inspection data is anonymized locally on the terminal, and the model weight gradient is uploaded to the federated learning server when the preset conditions are met. The federated learning server performs gradient aggregation, and the aggregated global model is periodically distributed to each terminal.

9. The inspection and testing visual data management method according to claim 7, characterized in that, After obtaining detection data from the transactional relational database based on the target ID set, the process further includes: Acquire several trusted data packets formed during the data collection, calculate the hash value of each trusted data packet, and use the hash value as a leaf node to construct a Merkle tree to generate a unique Merkle root hash; The Merkle root hash is written into the blockchain via a smart contract to achieve batch aggregation and storage of evidence.

10. An inspection and testing visual data management system, characterized in that, include: The terminal is configured as follows: after creating a detection task and binding a digital detection procedure on the interactive management platform, the detection task is loaded, the visual servo guidance engine is started, the sample to be inspected and the reference object in the picture are identified in real time, and the contours of the sample to be inspected and the reference object are extracted; an ideal posture model is dynamically generated according to the digital detection procedure, the transformation matrix between the extracted contour and the ideal posture model is calculated, and a guidance animation is rendered to guide the user to move to the target shooting posture. After the user triggers the shooting command, images, spatial information, process traceability data and environmental context data are collected simultaneously to form a reliable data packet; The edge quality inspection module, deployed on the terminal, is configured to: after capturing the image in the trusted data packet, input the image into the convolutional neural network locally on the terminal, and perform composition compliance detection, image quality assessment and defect area detection in parallel, and provide real-time feedback and decision based on the rule engine; The edge quality inspection module is also configured to: anonymize the quality inspection data locally on the terminal, upload the model weight gradient to the federated learning server when the preset conditions are met, and receive the global model issued by the federated learning server after aggregation and update. The cloud-based analytics system adopts a cloud-native microservice architecture, including: The data preprocessing module is configured to parse the trusted data packet and extract the image, spatial information, process traceability data and environmental context data from the trusted data packet; The hybrid storage engine includes: a transactional relational database for storing the extracted spatial information, process traceability data, and environmental context data as detection business data and the relationships between the detection business data; a retrieval engine for storing the extracted text descriptions, defect labels, and numerical measurement results corresponding to the images; a dedicated vector database for storing the hybrid feature vectors generated by feature extraction from the extracted images; and object storage. The multimodal analysis pipeline is configured to quantify defects in received trusted data packets and extract hybrid feature vectors of images through a dual-path feature extraction network. The data synchronization module is configured to adopt a change data capture and event-driven architecture to synchronize data change events in the transactional relational database to the retrieval engine and the dedicated vector database in real time. The knowledge discovery engine is configured to: receive query requests; when the query request contains attribute filtering conditions, submit the attribute filtering conditions to the retrieval engine for retrieval to obtain a first result ID set; when the query request contains a query image, extract the mixed feature vector of the query image, submit the mixed feature vector to the dedicated vector database for approximate nearest neighbor search to obtain a second result ID set; perform set operations on the first result ID set and / or the second result ID set to obtain a target ID set; and obtain detection detail data from the transactional relational database based on the target ID set. The knowledge discovery engine is further configured to: extract detection business data and the relationships between the detection business data from the detection report data in the transactional relational database using natural language processing, store them in a graph database, construct a quality knowledge graph, and run a graph neural network algorithm on the quality knowledge graph to perform root cause analysis and quality prediction; The blockchain evidence storage module is configured to: acquire several trusted data packets parsed by the data preprocessing module, calculate the hash value of each trusted data packet, and use the hash value as a leaf node to construct a Merkle tree to generate a unique Merkle root hash; and write the Merkle root hash into the blockchain through a smart contract to achieve batch aggregation evidence storage.

Images