Intelligent inspection method and system based on data acquisition and multi-modal AI, and terminal equipment

Abstract

The invention relates to the technical field of data acquisition and analysis of inspection equipment, in particular to an intelligent inspection method and system based on data acquisition and multi-modal AI and terminal equipment. Performing data consistency judgment analysis on each voxel block in the voxel space to obtain each multi-modal data set of the inspection target in a preset time; obtaining modal features of each multi-modal data set, and processing the modal features of each multi-modal data set according to a preset mapping and weight mechanism to obtain a multi-modal fusion feature vector fusion model of the inspection target; the inspection target is analyzed according to the multi-modal fusion feature vector fusion model and the multi-modal fusion feature vector reference model, the working state of the inspection target is obtained, and the working state of the inspection target can be accurately judged, so that the accuracy, reliability and comprehensiveness of inspection are improved, and normal operation of equipment is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection and analysis of inspection equipment, and in particular to an intelligent inspection method, system and terminal equipment based on data collection and multimodal AI. Background Art

[0002] With the continuous improvement of intelligence, the demand for real-time monitoring and maintenance of the operating status of equipment (such as engineering pipelines, power facilities, etc.) is also growing. Traditional inspection methods mainly rely on manual regular inspections, which have disadvantages such as low efficiency, strong subjectivity, and susceptibility to environmental influences. In order to solve these problems, in recent years, intelligent inspection methods based on data acquisition and artificial intelligence technology have gradually emerged and achieved remarkable results. However, most of the existing intelligent inspection methods are based on a single sensor or a single modal data, which makes it difficult to fully and accurately reflect the operating status of the equipment. For example, inspections based solely on visual images may not be able to detect faults inside the equipment in a timely manner; analysis based solely on vibration data may not be able to distinguish different types of faults.

[0003] In order to overcome the limitations of existing technologies, there is an urgent need for an inspection method that can fuse multiple modal data and use artificial intelligence technology for intelligent analysis. Multimodal data fusion technology can combine multiple modal data such as images, sounds, vibrations, and temperatures to form a comprehensive perception of the operating status of the equipment. Artificial intelligence technology, especially deep learning algorithms, can automatically extract features from massive multimodal data and identify abnormal conditions of equipment. Therefore, the development of an intelligent inspection method based on data acquisition and multimodal AI can effectively improve inspection efficiency, reduce labor costs, and improve the accuracy of fault diagnosis, which is of great practical significance. Summary of the invention

[0004] The present invention overcomes the deficiencies of the prior art and provides an intelligent inspection method, system and terminal device based on data acquisition and multimodal AI.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: The present invention discloses an intelligent inspection method based on data collection and multimodal AI, comprising the following steps: Acquire multimodal data of the inspection target, construct a voxel space, and process each multimodal data in the voxel space to obtain a plurality of voxel blocks; Perform data consistency judgment and analysis on each voxel block in the voxel space to obtain various multimodal data groups of the inspection target within a preset time; Obtaining the modal features of each multimodal data group, processing the modal features of each multimodal data group according to a preset mapping and weight mechanism, and obtaining a multimodal fusion feature vector fusion model of the inspection target; A multimodal fusion feature vector benchmark model of the inspection target in a standard state is prefabricated in advance, and the inspection target is analyzed according to the multimodal fusion feature vector fusion model and the multimodal fusion feature vector benchmark model to obtain the working state of the inspection target.

[0006] Preferably, multimodal data of the inspection target is obtained, and a voxel space is constructed, and each multimodal data is processed in the voxel space to obtain a plurality of voxel blocks, specifically: Using the sensor array mounted on the intelligent inspection equipment, multimodal data of the inspection target is collected at several preset time points; wherein the multimodal data includes visual image data, infrared image data, sound wave data, vibration data, pressure data and electrical parameter data; Constructing a voxel space, dividing the voxel space into a number of unit voxel blocks of equal size, discretizing the acquired multimodal data, and mapping the discretized multimodal data to each unit voxel block respectively; Acquire data features of multimodal data on each unit voxel block, wherein the data features include text vectors and image feature vectors; calculate cosine similarity between multimodal data on each unit voxel block according to the corresponding data features; Multimodal data with cosine similarity greater than a preset similarity are defined as the same group of multimodal data; unit voxel blocks corresponding to the multimodal data defined as the same group are merged to obtain a plurality of voxel blocks; wherein each voxel block contains multimodal data with the same data features.

[0007] Preferably, data consistency judgment and analysis is performed on each voxel block in the voxel space to obtain various multimodal data groups of the inspection target within a preset time, specifically: Acquire the multimodal data in each voxel block, perform weighted average processing on the multimodal data in each voxel block to obtain the weighted average value of the multimodal data in each voxel block, and use the weighted average value of the multimodal data in each voxel block as the voxel center of each voxel block; Calculate the Mahalanobis distance between each multimodal data and the corresponding voxel center in each voxel block, perform weighted average processing on the Mahalanobis distance between each multimodal data and the corresponding voxel center in each voxel block, and obtain the compactness of each voxel block; and compare the compactness of each voxel block with a preset threshold; If the compactness of a certain voxel block is greater than a preset threshold, indicating that the data consistency of the voxel block meets the standard, the voxel block is segmented in the voxel space, and the multimodal data in the segmented voxel block is sorted based on the acquisition timestamp to obtain one of the multimodal data groups of the inspection target within the preset time; If the compactness of a certain voxel block is not greater than the preset threshold, it means that the data consistency of the voxel block does not meet the standard, and then the voxel block is subjected to redundancy reduction and denoising processing until the compactness of the voxel block is greater than the preset threshold. Then, the voxel block is segmented in the voxel space, and the multimodal data in the segmented voxel block is sorted based on the acquisition timestamp to obtain one of the multimodal data groups of the inspection target within the preset time; This process is repeated in this way until all voxel blocks in the voxel space are analyzed, and various multimodal data sets of the inspection target within a preset time are obtained.

[0008] Preferably, if the compactness of a certain voxel block is not greater than a preset threshold, indicating that the data consistency of the voxel block does not meet the standard, the voxel block is subjected to redundancy reduction and denoising processing, specifically: Marking voxel blocks whose compactness is not greater than a preset threshold value respectively, and obtaining the Mahalanobis distance between each multimodal data and the corresponding voxel center in the marked voxel block; Sorting the obtained Mahalanobis distances in descending order to obtain a sorting result of the Mahalanobis distances from large to small, and generating a sorting table of each multimodal data in the labeled voxel block according to the sorting result; Deleting the first multimodal data in the sorting table from the marked voxel block, and then calculating the compactness of the marked voxel block; comparing the compactness of the marked voxel block with a preset threshold; If the compactness of the marked voxel group is greater than a preset threshold, the iteration is stopped, and the redundancy reduction and denoising processing of the marked voxel group is completed; If the density of the marked voxel block is still not greater than the preset threshold, the second multimodal data in the sorted table is deleted from the marked voxel block, and this process is repeated iteratively to delete the Nth multimodal data in the sorted table from the marked voxel block, until the density of the marked voxel block is greater than the preset threshold, then the iteration is stopped, and the redundancy reduction and denoising processing of the marked voxel block is completed.

[0009] Preferably, the modal features of each multimodal data group are obtained, and the modal features of each multimodal data group are processed according to a preset mapping and weight mechanism to obtain a multimodal fusion feature vector fusion model of the inspection target, specifically: Obtain various multimodal data groups of the inspection target within a preset time, and perform feature classification on each multimodal data group to obtain image-type multimodal data groups and data-type multimodal data groups; For a multimodal data group of an image, extract the texture features and shape features of each multimodal data in the group to obtain the image modality features of the corresponding multimodal data group; For a multimodal data group of a data class, the mean, median, mode, variance, and standard deviation of each multimodal data in the group are calculated to obtain the data modality characteristics of the corresponding multimodal data group; Construct a fusion framework, and map the acquired image modality features and data modality features into feature spaces respectively mapped to the fusion framework according to a preset mapping mechanism; In the fusion framework, the image modality features and the data modality features in each feature space are fused according to a preset weight allocation mechanism; After being processed by the fusion framework, a unified feature vector representation is output, and a multi-modal fusion feature vector fusion model of the inspection target is obtained; The multimodal fusion feature vector fusion model represents information that integrates multimodal data and is used to reflect status information of the inspected target.

[0010] Preferably, a multimodal fusion feature vector reference model of the inspection target in a standard state is prefabricated in advance, and the inspection target is analyzed according to the multimodal fusion feature vector fusion model and the multimodal fusion feature vector reference model to obtain the working state of the inspection target, specifically: Prefabricate a multi-modal fusion feature vector benchmark model of an inspection target in a standard state, and obtain a multi-modal fusion feature vector fusion model of an inspection target; Comparing the multimodal fusion feature vector fusion model of the inspection target with the multimodal fusion feature vector benchmark model dimension by dimension; For each dimension of the modal feature in the multimodal fusion feature vector fusion model of the inspection target, determine whether it falls within the value range corresponding to the multimodal fusion feature vector benchmark model; If the modal feature of a certain dimension in the multimodal fusion feature vector fusion model of the inspection target does not fall within the value range corresponding to the multimodal fusion feature vector benchmark model, it is marked as an abnormal dimension; If the modal feature of a certain dimension in the multimodal fusion feature vector fusion model of the inspection target falls within the value range corresponding to the multimodal fusion feature vector benchmark model, it is marked as a normal dimension.

[0011] The following steps are also included: Determine whether there is an abnormal dimension in the multimodal fusion feature vector fusion model. If there is no abnormal dimension in the multimodal fusion feature vector fusion model, calibrate the inspection target as a normal state; If there are abnormal dimensions in the multimodal fusion feature vector fusion model, obtain the importance weight of each abnormal dimension in the entire multimodal fusion feature vector fusion model; If there is a situation where the importance weight of one or more abnormal dimensions in the entire multimodal fusion feature vector fusion model is greater than the preset weight value, the inspection target is marked as an abnormal state; If the importance weight of each abnormal dimension in the entire multimodal fusion feature vector fusion model is not greater than the preset weight value, the number of abnormal dimensions in the multimodal fusion feature vector fusion model is ratioed to the preset number to obtain the abnormal dimension ratio; If the proportion of the abnormal dimension is greater than the preset proportion, the inspection target is marked as an abnormal state; if the proportion of the abnormal dimension is not greater than the preset proportion, the inspection target is marked as a normal state.

[0012] The present invention also discloses an intelligent inspection system based on data acquisition and multimodal AI, the intelligent inspection system based on data acquisition and multimodal AI comprising a memory and a processor, the memory storing an intelligent inspection method program based on data acquisition and multimodal AI, when the intelligent inspection method program based on data acquisition and multimodal AI is executed by the processor, any one of the steps of the intelligent inspection method based on data acquisition and multimodal AI is implemented.

[0013] The present invention also discloses a terminal device, which is applied to any of the intelligent inspection methods based on data collection and multimodal AI, comprising: Data acquisition module, used to obtain multimodal data of inspection targets; Voxel space construction module, used to construct voxel space; A data consistency judgment and analysis module is used to perform data consistency judgment and analysis on each voxel block in the voxel space; The model comparison and analysis module is used to compare and analyze the multimodal fusion feature vector fusion model with the multimodal fusion feature vector benchmark model.

[0014] The present invention solves the technical defects existing in the background technology, and has the following beneficial effects: the present intelligent inspection method constructs a fusion model through multi-modal data collection, voxel space processing, data consistency analysis, and modal feature fusion, and compares it with a prefabricated benchmark model. It can comprehensively and comprehensively use various types of data to analyze the inspection target, effectively overcome the limitations of single-modal data inspection, accurately judge the working status of the inspection target, thereby improving the accuracy, reliability and comprehensiveness of the inspection, and helping to timely discover equipment failures and abnormal operation conditions, and ensure the normal operation of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, drawings of other embodiments can be obtained based on these drawings without paying creative work.

[0016] Figure 1 This is the overall method flow chart of this inspection method; Figure 2 This is a partial method flow chart of this inspection method; Figure 3 This is the system block diagram of this inspection system. DETAILED DESCRIPTION

[0017] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0018] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0019] The present invention discloses an intelligent inspection method based on data collection and multimodal AI, such as Figure 1 As shown, the following steps are included: S102: acquiring multimodal data of the inspection target, constructing a voxel space, and processing each multimodal data in the voxel space to obtain a plurality of voxel blocks; S104: performing data consistency judgment and analysis on each voxel block in the voxel space to obtain various multimodal data groups of the inspection target within a preset time; S106: acquiring modal features of each multimodal data group, processing the modal features of each multimodal data group according to a preset mapping and weight mechanism, and obtaining a multimodal fusion feature vector fusion model of the inspection target; S108: prefabricate in advance a multimodal fusion feature vector benchmark model of the inspection target in a standard state, analyze the inspection target according to the multimodal fusion feature vector fusion model and the multimodal fusion feature vector benchmark model, and obtain the working state of the inspection target.

[0020] It should be noted that multimodal data covers many types, such as visual image data, which can directly reflect the appearance, structure and other information of the inspection target; infrared image data can detect temperature distribution, which is helpful to find abnormalities such as overheating; acoustic wave data can capture the sound characteristics of equipment operation; vibration data reflects the mechanical state of the equipment; pressure data and electrical parameter data represent the equipment state from the pressure and electrical aspects respectively. Constructing voxel space and processing multimodal data to obtain voxel blocks is a way to spatially organize and preliminarily integrate multimodal data. Voxel space provides a unified processing framework for multimodal data. Through processing in this space, data of different modes are associated in the form of voxel blocks, so that subsequent analysis can be based on these blocks. After obtaining the modal features of each multimodal data group, the multimodal fusion feature vector fusion model is obtained according to the preset mapping and weight mechanism. This step is one of the cores of the whole method. Different modal features have different importance and representation capabilities. They are mapped to a common feature space through the preset mapping mechanism and then fused according to the weight mechanism. For example, for some equipment, the modal features of vibration data may have a higher weight in fault diagnosis, while for other equipment, the features of infrared image data may be more critical. This fusion model can integrate the information of each modality to form a more comprehensive and accurate feature description of the inspection target. It is an important reference standard to pre-fabricate a multimodal fusion feature vector benchmark model in which the inspection target is in a standard state. By comparing and analyzing the actual multimodal fusion feature vector fusion model with the benchmark model, the differences between the two can be found. These differences can reflect whether the inspection target deviates from the standard state, thereby obtaining the working status of the inspection target. For example, if some features in the fusion model have changed significantly compared with the benchmark model, it may mean that the inspection target is faulty or in an abnormal operating state.

[0021] This intelligent inspection method constructs a fusion model through multi-modal data collection, voxel space processing, data consistency analysis, and modal feature fusion, and compares it with a pre-made benchmark model. It can comprehensively and comprehensively use various types of data to analyze the inspection target, effectively overcome the limitations of single-modal data inspection, and accurately judge the working status of the inspection target, thereby improving the accuracy, reliability and comprehensiveness of the inspection, helping to timely discover equipment failures and abnormal operation conditions, and ensuring the normal operation of the equipment.

[0022] Preferably, multimodal data of the inspection target is obtained, and a voxel space is constructed, and each multimodal data is processed in the voxel space to obtain a plurality of voxel blocks, such as Figure 2 As shown, specifically: S202: using a sensor array mounted on an intelligent inspection device to collect multimodal data of an inspection target at a number of preset time points; wherein the multimodal data includes visual image data, infrared image data, sound wave data, vibration data, pressure data, and electrical parameter data; S204: constructing a voxel space, and dividing a number of unit voxel blocks of equal size in the voxel space, performing discretization processing on the acquired multimodal data, and mapping the discretized multimodal data to each unit voxel block respectively; S206: Acquire data features of the multimodal data on each unit voxel block, wherein the data features include a text vector and an image feature vector; and calculate the cosine similarity between the multimodal data on each unit voxel block according to the corresponding data features; S208: defining the multimodal data whose cosine similarity is greater than a preset similarity as the same group of multimodal data; merging the unit voxel blocks corresponding to the multimodal data defined as the same group to obtain a plurality of voxel blocks; wherein each voxel block contains multimodal data with the same data features.

[0023] It should be noted that the voxel space is constructed and divided into unit voxel blocks of equal size, and then the collected multimodal data is discretized and mapped to the unit voxel blocks. The construction of the voxel space provides a unified spatial framework for multimodal data, so that different types of data can be integrated and processed in this space. The discrete mapping process converts continuous multimodal data into a discrete form corresponding to the voxel block, which is convenient for the subsequent analysis and processing of data features based on the voxel block. The data features of the multimodal data on the unit voxel block are obtained, including text vectors and image feature vectors. The text vector is a representation of the semantic description of the data, and the image feature vector can reflect the texture, shape and other features of the image data. The similarity between the data is measured by calculating the cosine similarity between the multimodal data on each unit voxel block. The multimodal data with a cosine similarity greater than the preset similarity is defined as the same group of multimodal data, and the corresponding unit voxel blocks are merged to obtain the voxel group block. The purpose of this is to cluster multimodal data with the same or similar data features together, so that the data in each voxel group block has a high correlation. Within a voxel block, since the data features are the same, it is easier to conduct a comprehensive analysis of the data and explore the internal correlations within the data, thus providing a more targeted data set for subsequent status judgment of the inspection target.

[0024] This method collects multimodal data, constructs voxel space and performs a series of processing to obtain voxel blocks, thereby realizing the effective integration and classification of multi-source and multi-type inspection data. It can aggregate multimodal data with similar characteristics, explore the inherent connections between data, overcome the complexity and heterogeneity of multimodal data, and provide an orderly and targeted data basis for subsequent accurate judgment of the status of the inspection target, thereby improving the accuracy and efficiency of the inspection.

[0025] Preferably, data consistency judgment and analysis is performed on each voxel block in the voxel space to obtain various multimodal data groups of the inspection target within a preset time, specifically: Acquire the multimodal data in each voxel block, perform weighted average processing on the multimodal data in each voxel block to obtain the weighted average value of the multimodal data in each voxel block, and use the weighted average value of the multimodal data in each voxel block as the voxel center of each voxel block; Calculate the Mahalanobis distance between each multimodal data and the corresponding voxel center in each voxel block, perform weighted average processing on the Mahalanobis distance between each multimodal data and the corresponding voxel center in each voxel block, and obtain the compactness of each voxel block; and compare the compactness of each voxel block with a preset threshold; If the compactness of a certain voxel block is greater than a preset threshold, indicating that the data consistency of the voxel block meets the standard, the voxel block is segmented in the voxel space, and the multimodal data in the segmented voxel block is sorted based on the acquisition timestamp to obtain one of the multimodal data groups of the inspection target within the preset time; If the compactness of a certain voxel block is not greater than the preset threshold, it means that the data consistency of the voxel block does not meet the standard, and then the voxel block is subjected to redundancy reduction and denoising processing until the compactness of the voxel block is greater than the preset threshold. Then, the voxel block is segmented in the voxel space, and the multimodal data in the segmented voxel block is sorted based on the acquisition timestamp to obtain one of the multimodal data groups of the inspection target within the preset time; This process is repeated in this way until all voxel blocks in the voxel space are analyzed, and various multimodal data sets of the inspection target within a preset time are obtained.

[0026] It should be noted that the voxel center is obtained by weighted averaging the multimodal data within the voxel block. This voxel center can be regarded as a representative value of the multimodal data within the voxel block, which integrates the information of each data within the block. Then the Mahalanobis distance between each multimodal data and the voxel center is calculated. The Mahalanobis distance takes into account the covariance structure of the data and can more accurately measure the degree of deviation of the data point from the center. Then, these Mahalanobis distances are weighted averaged to obtain the compactness of the voxel block. The compactness reflects the degree of aggregation of the data within the voxel block. The higher the compactness, the more tightly the data in the block surrounds the voxel center.

[0027] If the compactness is greater than the preset threshold, it indicates that the data consistency of the voxel block meets the standard. At this time, the voxel block is segmented in the voxel space, and the multimodal data in the block is sorted according to the acquisition timestamp to obtain a multimodal data group. This sorting process is helpful for the subsequent time series analysis of the data, such as observing the trend of data changes over time. If the compactness is not greater than the preset threshold, that is, the data consistency does not meet the standard, the voxel block is subjected to redundancy reduction and denoising. This step aims to remove redundant data and noise data that may exist in the block to improve the quality and consistency of the data. When the compactness after processing is greater than the preset threshold, it is segmented and sorted based on the timestamp to obtain a multimodal data group. According to the above steps, all voxel blocks in the voxel space are analyzed, and finally the multimodal data groups of the inspection target within the preset time are obtained. This process ensures that the obtained multimodal data group has high data consistency and is arranged in chronological order, providing a high-quality and orderly data basis for subsequent analysis.

[0028] This method can effectively screen and process multimodal data by analyzing the data consistency of voxel blocks, calculating the voxel center and compactness and comparing them with the threshold. It removes redundancy and noise in the data, improves data consistency, and sorts the data according to the acquisition time, thus providing a reliable, orderly, and high-quality multimodal data set for subsequent accurate analysis of the state changes of the inspection target within the preset time, which helps to improve the accuracy and reliability of the inspection.

[0029] Preferably, if the compactness of a certain voxel block is not greater than a preset threshold, indicating that the data consistency of the voxel block does not meet the standard, the voxel block is subjected to redundancy reduction and denoising processing, specifically: Marking voxel blocks whose compactness is not greater than a preset threshold value respectively, and obtaining the Mahalanobis distance between each multimodal data and the corresponding voxel center in the marked voxel block; Sorting the obtained Mahalanobis distances in descending order to obtain a sorting result of the Mahalanobis distances from large to small, and generating a sorting table of each multimodal data in the labeled voxel block according to the sorting result; Deleting the first multimodal data in the sorting table from the marked voxel block, and then calculating the compactness of the marked voxel block; comparing the compactness of the marked voxel block with a preset threshold; If the compactness of the marked voxel group is greater than a preset threshold, the iteration is stopped, and the redundancy reduction and denoising processing of the marked voxel group is completed; If the density of the marked voxel block is still not greater than the preset threshold, the second multimodal data in the sorted table is deleted from the marked voxel block, and this process is repeated iteratively to delete the Nth multimodal data in the sorted table from the marked voxel block, until the density of the marked voxel block is greater than the preset threshold, then the iteration is stopped, and the redundancy reduction and denoising processing of the marked voxel block is completed.

[0030] It should be noted that when the compactness of the voxel block is not greater than the preset threshold, these substandard data blocks must first be marked for identification. At the same time, the Mahalanobis distance between each multimodal data and the corresponding voxel center in the marked voxel block is obtained. The Mahalanobis distance can reflect the degree of deviation of the multimodal data from the voxel center, and is an important basis for judging whether the data is redundant or noisy. The obtained Mahalanobis distance is sorted in descending order to obtain the sorting result of the Mahalanobis distance from large to small, and a sorting table of each multimodal data in the marked voxel block is generated accordingly. This sorting method makes the multimodal data that deviates farthest from the voxel center come first, because these data are more redundant or noisy data. According to the order of the sorting table, starting from the first multimodal data, it is deleted from the marked voxel block, and then the compactness of the voxel block is recalculated. The calculation of compactness reflects the degree of aggregation of data in the voxel block, and as possible redundant or noisy data is deleted, the compactness is expected to increase. After each data is deleted, the new compactness is compared with the preset threshold. If the compactness is greater than the preset threshold, it means that the data consistency of the voxel block has reached the standard, the iteration is stopped, and the redundancy reduction and denoising process is completed. If the compactness is still not greater than the preset threshold, the next multimodal data (such as the second, third...Nth) is deleted in the order of the sorting table, and the compactness calculation and comparison process is repeated until the compactness is greater than the preset threshold.

[0031] This method performs a series of redundancy reduction and denoising processes on voxel blocks that do not meet the compactness standards, gradually removes possible redundant or noisy data according to the sorting of multimodal data and the Mahalanobis distance of the voxel center, and judges the processing effect by comparing the compactness with the threshold, thereby effectively improving the data consistency of the voxel blocks, removing bad data that may affect subsequent analysis, providing a more reliable data basis for accurate judgment of the inspection target status, and improving the accuracy and effectiveness of the entire inspection process.

[0032] Preferably, the modal features of each multimodal data group are obtained, and the modal features of each multimodal data group are processed according to a preset mapping and weight mechanism to obtain a multimodal fusion feature vector fusion model of the inspection target, specifically: Obtain various multimodal data groups of the inspection target within a preset time, and perform feature classification on each multimodal data group to obtain image-type multimodal data groups and data-type multimodal data groups; For a multimodal data group of an image, extract the texture features and shape features of each multimodal data in the group to obtain the image modality features of the corresponding multimodal data group; For a multimodal data group of a data class, the mean, median, mode, variance, and standard deviation of each multimodal data in the group are calculated to obtain the data modality characteristics of the corresponding multimodal data group; Construct a fusion framework, and map the acquired image modality features and data modality features into feature spaces respectively mapped to the fusion framework according to a preset mapping mechanism; In the fusion framework, the image modality features and the data modality features in each feature space are fused according to a preset weight allocation mechanism; Among them, the importance of image modal features and data modal features in reflecting the target state is first determined according to the inspection target characteristics, and their respective weights are set accordingly. Then, the corresponding elements of the two modal features are multiplied and accumulated according to the set weights to achieve fusion; After being processed by the fusion framework, a unified feature vector representation is output, and a multi-modal fusion feature vector fusion model of the inspection target is obtained; The multimodal fusion feature vector fusion model represents information that integrates multimodal data and is used to reflect status information of the inspected target.

[0033] Among them, the preset mapping mechanism is a pre-set rule or method for mapping different types of modal features (such as the image modal features and data modal features mentioned above) to the corresponding feature space in the fusion framework. The purpose of this mechanism is to process multiple features of different natures in a unified framework. For image modal features, according to their texture features, shape features, etc., they are converted into a form that can be understood and processed by the fusion framework through a preset mapping mechanism, and placed in the corresponding feature space. For example, the texture features of the image are mapped to a specific feature space dimension according to a certain mathematical transformation (such as Fourier transform, etc.), so that it can be subsequently operated with other modal features in the same framework. For data modal features, such as mean, median, mode, variance and standard deviation, they also need to be converted through a preset mapping mechanism. According to factors such as the range and data type of the data, these statistical features are mapped to appropriate feature space positions to ensure that they are in a fusionable state with the image modal features in the fusion framework.

[0034] The preset weight allocation mechanism is a method of presetting weights within the fusion framework according to the importance of different modal features in reflecting the state information of the inspection target. Since the image modal features and data modal features may contribute differently in describing the state of the inspection target, different weights need to be assigned to them. For example, in some inspection scenarios, if the appearance shape and texture changes of the inspection target are very critical to judging its state, then when fusing the image modal features and the data modal features, a higher weight will be assigned to the image modal features. The setting of this weight can be based on domain knowledge, analysis of historical data, or experimental results. For example, by analyzing a large amount of inspection target data with known states, it is found that the variance in the data modal features plays a more important role in judging whether the equipment is faulty. Then, in the preset weight allocation mechanism, the variance, a data modal feature, will be assigned a relatively high weight to ensure that during the fusion process, this feature can have a more realistic impact on the final multimodal fusion feature vector fusion model.

[0035] It should be noted that, first, the various multimodal data groups of the inspection target within the preset time are obtained, and then they are classified into image-type and data-type multimodal data groups. For the image-type multimodal data group, texture features and shape features are extracted to obtain image modal features. Texture features can reflect the surface texture pattern of the image, such as smooth, rough, etc., and shape features describe the geometric shape of the object. These features help to identify the appearance state and structural integrity of the object. For the data-type multimodal data group, the data modal features are obtained by calculating the mean, median, mode, variance and standard deviation. The mean represents the average level of the data, the median is the middle value after the data is sorted, the mode is the value with the highest frequency, and the variance and standard deviation reflect the degree of discreteness of the data. These statistical features can describe the central tendency and discrete characteristics of the data-type multimodal data from different angles. After the fusion framework is constructed, the image modal features and data modal features are mapped to the corresponding feature space according to the preset mapping mechanism. This mapping mechanism finds a suitable position for different types of features in the fusion framework, so that they can be processed under the same framework. In the fusion framework, the image modal features and data modal features in each feature space are fused according to the preset weight distribution mechanism. The weight distribution mechanism takes into account the importance of different features in reflecting the state information of the inspection target. For example, in some inspection scenarios, the shape features of the image may be more important than the texture features, or the variance of the data can better reflect the operating status of the equipment than the mean. By reasonably allocating weights, the fused feature vector can more accurately reflect the state of the inspection target. After being processed by the fusion framework, a unified feature vector representation is output, namely the multimodal fusion feature vector fusion model. This model integrates the information of multimodal data. It is not just a simple combination of data features of different modalities, but through feature extraction, mapping and fusion processes, the information about the state of the inspection target in the multimodal data is integrated, so that the state information of the inspected target can be more comprehensive and accurate.

[0036] This method obtains a multimodal fusion feature vector fusion model by performing operations such as classification, feature extraction, mapping and fusion in a fusion framework on multimodal data groups. This method can fully explore the respective features of image and data classes in multimodal data, and fuse them into a comprehensive feature vector through a reasonable mapping and weight distribution mechanism, thereby effectively integrating multimodal data information, more accurately reflecting the status information of the inspection target, and providing a more reliable decision-making basis for inspection work.

[0037] Preferably, a multimodal fusion feature vector reference model of the inspection target in a standard state is prefabricated in advance, and the inspection target is analyzed according to the multimodal fusion feature vector fusion model and the multimodal fusion feature vector reference model to obtain the working state of the inspection target, specifically: Prefabricate a multi-modal fusion feature vector benchmark model of an inspection target in a standard state, and obtain a multi-modal fusion feature vector fusion model of an inspection target; Comparing the multimodal fusion feature vector fusion model of the inspection target with the multimodal fusion feature vector benchmark model dimension by dimension; For each dimension of the modal feature in the multimodal fusion feature vector fusion model of the inspection target, determine whether it falls within the value range corresponding to the multimodal fusion feature vector benchmark model; If the modal feature of a certain dimension in the multimodal fusion feature vector fusion model of the inspection target does not fall within the value range corresponding to the multimodal fusion feature vector benchmark model, it is marked as an abnormal dimension; If the modal feature of a certain dimension in the multimodal fusion feature vector fusion model of the inspection target falls within the value range corresponding to the multimodal fusion feature vector benchmark model, it is marked as a normal dimension.

[0038] The following steps are also included: Determine whether there is an abnormal dimension in the multimodal fusion feature vector fusion model. If there is no abnormal dimension in the multimodal fusion feature vector fusion model, calibrate the inspection target as a normal state; If there are abnormal dimensions in the multimodal fusion feature vector fusion model, obtain the importance weight of each abnormal dimension in the entire multimodal fusion feature vector fusion model; If there is a situation where the importance weight of one or more abnormal dimensions in the entire multimodal fusion feature vector fusion model is greater than the preset weight value, the inspection target is marked as an abnormal state; If the importance weight of each abnormal dimension in the entire multimodal fusion feature vector fusion model is not greater than the preset weight value, the number of abnormal dimensions in the multimodal fusion feature vector fusion model is ratioed to the preset number to obtain the abnormal dimension ratio; If the proportion of the abnormal dimension is greater than the preset proportion, the inspection target is marked as an abnormal state; if the proportion of the abnormal dimension is not greater than the preset proportion, the inspection target is marked as a normal state.

[0039] It should be noted that the relevant technical personnel have pre-made a multimodal fusion feature vector benchmark model of the inspection target in a standard state. This benchmark model is based on the multimodal data of the inspection target in a normal working state, and contains information such as the normal modal feature value range of each dimension. For each dimension of the modal feature in the fusion model, check whether it is within the value range corresponding to the benchmark model. If it is not within the range, it is marked as an abnormal dimension; if it is within the range, it is marked as a normal dimension. This dimension-by-dimensional comparison can accurately find the feature dimensions that differ from the standard state.

[0040] If there is no abnormal dimension in the fusion model, the inspection target is directly calibrated as normal state. This is a more intuitive situation, indicating that all aspects of the current inspection target meet the requirements of the standard state.

[0041] When there are abnormal dimensions, the importance weights of each abnormal dimension in the entire fusion model are further obtained. If the importance weights of one or more abnormal dimensions are greater than the preset weight value, the inspection target is marked as abnormal. This is because the abnormality of these more important dimensions may mean that there are serious problems with the inspection target. If the importance weights of each abnormal dimension are not greater than the preset weight value, the ratio of the number of abnormal dimensions to the preset number is calculated to obtain the proportion of abnormal dimensions. If this proportion is greater than the preset proportion, the inspection target is also marked as abnormal; if not, it is marked as normal. This judgment method takes into account that although the importance of abnormal dimensions is not high, if there are too many abnormal dimensions, it may also indicate that there are certain problems with the inspection target.

[0042] This method prefabricates a benchmark model under standard conditions and compares it with the actual fusion model. From dimension-by-dimension comparison to multiple judgment logics based on abnormal dimensions, it comprehensively considers multiple factors such as the existence or non-existence of abnormal dimensions, the importance weight of abnormal dimensions, and the proportion of the number of abnormal dimensions. It can accurately and comprehensively judge the working status of the inspection target, effectively improve the accuracy and reliability of the inspection results, and reduce the occurrence of misjudgments and missed judgments.

[0043] In actual application, the inspection method may also include the following steps: Obtaining the bit error rate of the collected signals of the sensor array on the intelligent inspection equipment under various preset collection working environment combination conditions through the big data network; constructing a matching library, and dividing the matching library into a plurality of sub-storage spaces; Various preset collection working environment combinations are stored in corresponding sub-storage spaces, and the bit error rate of the collection signal under various preset collection working environment combination conditions is converted into a string of unique digital codes, and the corresponding digital codes are mapped on the digital nodes of the corresponding sub-storage space to form a feature matching library; Before using the sensor array mounted on the intelligent inspection device to collect multimodal data of the inspection target, obtain the real-time collection working environment of the inspection device at the current location node; Importing the real-time acquisition working environment into the feature matching library, and calculating the mutual information value between the real-time acquisition working environment and the preset acquisition working environment in each sub-storage space; Sorting the calculated mutual information values, extracting the maximum mutual information value, marking the sub-storage space to which the preset collection working environment corresponding to the maximum mutual information value belongs, and extracting the digital code of the digital node on the marked sub-storage space; Decode the extracted digital code to obtain the bit error rate of the collected signal when the sensor array on the intelligent inspection device works at the current collection position node; If the bit error rate of the collected signal is greater than the preset bit error rate, the collection position node of the intelligent inspection device is adjusted until the bit error rate is no greater than the preset bit error rate.

[0044] It should be noted that the acquisition signal bit error rate of the sensor array of the intelligent inspection equipment under various preset acquisition working environment combinations is obtained through the big data network. These preset acquisition working environment combinations cover different situations that the inspection equipment may face, such as different combinations of environmental factors such as temperature, humidity, and electromagnetic interference. Then, a matching library is constructed and divided into several sub-storage spaces, and the preset acquisition working environment combinations are stored in the corresponding sub-storage spaces. At the same time, the corresponding acquisition signal bit error rate is converted into a digital code and mapped to the digital node of the sub-storage space to form a feature matching library. This feature matching library is the basis for subsequent judgment and adjustment, and it establishes the relationship between the acquisition working environment and the acquisition signal bit error rate.

[0045] Before collecting multimodal data of the inspection target, obtain the real-time collection working environment of the current location node of the inspection device. This real-time collection working environment contains information about various environmental factors around the inspection device at the current moment. Import it into the feature matching library and calculate the mutual information value between the preset collection working environment in each sub-storage space. The mutual information value can measure the degree of correlation between two variables (here, the real-time collection working environment and the preset collection working environment). Sort the calculated mutual information values ​​and extract the maximum mutual information value. Mark the sub-storage space to which the corresponding preset collection working environment belongs, and then extract and interpret the digital code of the digital node of the sub-storage space to obtain the collection signal bit error rate when the sensor array works at the current collection position node. This process is actually to find the most matching situation based on the correlation between the current collection working environment and the preset environment to estimate the current collection signal bit error rate. If the obtained collection signal bit error rate is greater than the preset bit error rate, adjust the collection position node of the intelligent inspection device until the bit error rate is no greater than the preset bit error rate. The purpose of this step is to ensure the quality of the collected data, because too high a bit error rate may cause the collected data to be inaccurate or unreliable. By adjusting the collection position to reduce the bit error rate, the validity of the inspection data can be guaranteed. This method constructs a feature matching library, analyzes the mutual information between the real-time collection working environment and the preset environment, estimates the bit error rate of the collection signal, and adjusts the collection position node according to the bit error rate. It can effectively reduce the bit errors of the collection signal caused by the collection working environment, improve the accuracy and reliability of the collected data, and then ensure the effectiveness of the inspection work based on these data, and improve the quality and credibility of the entire inspection process.

[0046] In actual application, the inspection method may also include the following steps: Obtaining remaining inspection task information of the intelligent inspection device, obtaining a remaining inspection path of the intelligent inspection device according to the remaining inspection task information, and dividing the remaining inspection path into a plurality of nodes; Obtaining predicted climate and environmental information of each node based on meteorological software, generating search terms based on the predicted climate and environmental information of each node, searching the big data network based on the search terms, and obtaining the channel capacity of the MIMO signal transmission system in each node; Comparing the channel capacity of the MIMO signal transmission system in each node with a preset capacity value; If there are one or more nodes whose channel capacity is greater than the preset capacity value, a node corresponding to the maximum channel capacity is selected from the nodes corresponding to the channel capacity greater than the preset capacity value, and the selected node is defined as the best signal transmission node; If the channel capacity of each node is not greater than the preset capacity value, the signal transmission parameters of the intelligent inspection device are adjusted to re-acquire the channel capacity of each node until a node with a channel capacity greater than the preset capacity value appears, and then the best signal transmission node is screened out.

[0047] It should be noted that the remaining inspection task information of the intelligent inspection equipment is first obtained, which includes relevant information such as the targets and areas that have not been inspected. Based on this information, the remaining inspection path of the intelligent inspection equipment is determined, and then the path is divided into several nodes. These nodes can be specific location points on the path, such as the locations of different devices, the intersections of different areas, etc. Dividing the path into nodes helps to plan and manage the inspection process more carefully. Meteorological software is used to obtain the predicted climate and environmental information of each node, such as temperature, humidity, rainfall probability, wind speed, etc. These climate and environmental factors will have an impact on the MIMO signal transmission system. Based on these climate and environmental information, search terms are generated, and the channel capacity of the MIMO signal transmission system in each node is obtained by searching the big data network. This step establishes the connection between the climate environment and the channel capacity, because different climate environments may cause changes in signal propagation characteristics, thereby affecting the channel capacity. The channel capacity of each node is compared with the preset capacity value. If the channel capacity of one or more nodes is greater than the preset capacity value, the node with the largest channel capacity is selected from these nodes and defined as the best signal transmission node. This best signal transmission node is the optimal location point from the perspective of signal transmission quality. Selecting it helps to ensure that the intelligent inspection device can obtain better transmission effect when transmitting data at this node. If the channel capacity of each node is not greater than the preset capacity value, it means that the current channel capacity status does not meet the requirements. At this time, adjust the signal transmission parameters of the intelligent inspection device, such as transmission power, modulation mode, antenna direction, etc. After adjustment, re-acquire the channel capacity of each node, and repeat this process until a node with a channel capacity greater than the preset capacity value appears, and then filter out the best signal transmission node. This step ensures that even if the channel capacity of all nodes is not ideal in the initial situation, the channel capacity status can be improved by adjusting the device parameters, so as to find a suitable signal transmission node. This method obtains the channel capacity by analyzing the nodes of the remaining inspection path of the intelligent inspection device, combining meteorological environmental factors, and determines the best signal transmission node by comparison and screening. When the channel capacity does not meet the requirements, the device signal transmission parameters can be adjusted to re-acquire the channel capacity. This series of operations can optimize the selection of signal transmission nodes for intelligent inspection equipment on the inspection path based on consideration of the impact of climate and environment, improve the quality of signal transmission, and ensure the effective transmission of inspection data, thereby improving the reliability and efficiency of the entire intelligent inspection process.

[0048] The present invention also discloses an intelligent inspection system based on data acquisition and multimodal AI, such as Figure 3 As shown, the intelligent inspection system based on data acquisition and multimodal AI includes a memory 100 and a processor 200. The memory 100 stores an intelligent inspection method program based on data acquisition and multimodal AI. When the intelligent inspection method program based on data acquisition and multimodal AI is executed by the processor 200, any one of the steps of the intelligent inspection method based on data acquisition and multimodal AI is implemented.

[0049] The present invention also discloses a terminal device, which is applied to any of the intelligent inspection methods based on data collection and multimodal AI, comprising: Data acquisition module, used to obtain multimodal data of inspection targets; Voxel space construction module, used to construct voxel space; A data consistency judgment and analysis module is used to perform data consistency judgment and analysis on each voxel block in the voxel space; The model comparison and analysis module is used to compare and analyze the multimodal fusion feature vector fusion model with the multimodal fusion feature vector benchmark model.

[0050] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An intelligent inspection method based on data collection and multimodal AI, characterized in that: The following steps are involved: Acquire multimodal data of the inspection target, construct a voxel space, and process each multimodal data in the voxel space to obtain a plurality of voxel blocks; Perform data consistency judgment and analysis on each voxel block in the voxel space to obtain various multimodal data groups of the inspection target within a preset time; Obtaining the modal features of each multimodal data group, processing the modal features of each multimodal data group according to a preset mapping and weight mechanism, and obtaining a multimodal fusion feature vector fusion model of the inspection target; A multimodal fusion feature vector benchmark model of the inspection target in a standard state is prefabricated in advance, and the inspection target is analyzed according to the multimodal fusion feature vector fusion model and the multimodal fusion feature vector benchmark model to obtain the working state of the inspection target.

2. According to claim 1, an intelligent inspection method based on data collection and multimodal AI is characterized in that: The multimodal data of the inspection target is obtained, and a voxel space is constructed. Each multimodal data is processed in the voxel space to obtain a number of voxel blocks, specifically: Using the sensor array mounted on the intelligent inspection equipment, multimodal data of the inspection target is collected at several preset time points; wherein the multimodal data includes visual image data, infrared image data, sound wave data, vibration data, pressure data and electrical parameter data; Constructing a voxel space, dividing the voxel space into a number of unit voxel blocks of equal size, discretizing the acquired multimodal data, and mapping the discretized multimodal data to each unit voxel block respectively; Acquire data features of multimodal data on each unit voxel block, wherein the data features include text vectors and image feature vectors; calculate cosine similarity between multimodal data on each unit voxel block according to the corresponding data features; Multimodal data with cosine similarity greater than a preset similarity are defined as the same group of multimodal data; unit voxel blocks corresponding to the multimodal data defined as the same group are merged to obtain a plurality of voxel blocks; wherein each voxel block contains multimodal data with the same data features.

3. According to claim 1, the intelligent inspection method based on data collection and multimodal AI is characterized in that: Perform data consistency judgment and analysis on each voxel block in the voxel space to obtain various multimodal data groups of the inspection target within the preset time, specifically: Acquire the multimodal data in each voxel block, perform weighted average processing on the multimodal data in each voxel block to obtain the weighted average value of the multimodal data in each voxel block, and use the weighted average value of the multimodal data in each voxel block as the voxel center of each voxel block; Calculate the Mahalanobis distance between each multimodal data and the corresponding voxel center in each voxel block, perform weighted average processing on the Mahalanobis distance between each multimodal data and the corresponding voxel center in each voxel block, and obtain the compactness of each voxel block; and compare the compactness of each voxel block with a preset threshold; If the compactness of a certain voxel block is greater than a preset threshold, indicating that the data consistency of the voxel block meets the standard, the voxel block is segmented in the voxel space, and the multimodal data in the segmented voxel block is sorted based on the acquisition timestamp to obtain one of the multimodal data groups of the inspection target within the preset time; If the compactness of a certain voxel block is not greater than the preset threshold, it means that the data consistency of the voxel block does not meet the standard, and then the voxel block is subjected to redundancy reduction and denoising processing until the compactness of the voxel block is greater than the preset threshold. Then, the voxel block is segmented in the voxel space, and the multimodal data in the segmented voxel block is sorted based on the acquisition timestamp to obtain one of the multimodal data groups of the inspection target within the preset time; This process is repeated in this way until all voxel blocks in the voxel space are analyzed, and various multimodal data sets of the inspection target within a preset time are obtained.

4. According to claim 3, the intelligent inspection method based on data collection and multimodal AI is characterized in that: If the density of a certain voxel block is not greater than the preset threshold, it means that the data consistency of the voxel block does not meet the standard, and the voxel block is subjected to redundancy reduction and denoising processing, specifically: Marking voxel blocks whose compactness is not greater than a preset threshold value respectively, and obtaining the Mahalanobis distance between each multimodal data and the corresponding voxel center in the marked voxel block; Sorting the obtained Mahalanobis distances in descending order to obtain a sorting result of the Mahalanobis distances from large to small, and generating a sorting table of each multimodal data in the labeled voxel block according to the sorting result; Deleting the first multimodal data in the sorting table from the marked voxel block, and then calculating the compactness of the marked voxel block; comparing the compactness of the marked voxel block with a preset threshold; If the compactness of the marked voxel group block is greater than a preset threshold, the iteration is stopped, and the redundancy reduction and denoising processing of the marked voxel group block is completed; If the density of the marked voxel block is still not greater than the preset threshold, the second multimodal data in the sorted table is deleted from the marked voxel block, and this process is repeated iteratively to delete the Nth multimodal data in the sorted table from the marked voxel block, until the density of the marked voxel block is greater than the preset threshold, then the iteration is stopped, and the redundancy reduction and denoising processing of the marked voxel block is completed.

5. The intelligent inspection method based on data collection and multimodal AI according to claim 1 is characterized in that: The modal features of each multimodal data group are obtained, and the modal features of each multimodal data group are processed according to the preset mapping and weight mechanism to obtain the multimodal fusion feature vector fusion model of the inspection target, which is specifically: Obtain various multimodal data groups of the inspection target within a preset time, and perform feature classification on each multimodal data group to obtain image-type multimodal data groups and data-type multimodal data groups; For a multimodal data group of an image, extract the texture features and shape features of each multimodal data in the group to obtain the image modality features of the corresponding multimodal data group; For a multimodal data group of a data class, the mean, median, mode, variance, and standard deviation of each multimodal data in the group are calculated to obtain the data modality characteristics of the corresponding multimodal data group; Construct a fusion framework, and map the acquired image modality features and data modality features into feature spaces respectively mapped to the fusion framework according to a preset mapping mechanism; In the fusion framework, the image modality features and the data modality features in each feature space are fused according to a preset weight allocation mechanism; After being processed by the fusion framework, a unified feature vector representation is output, and a multi-modal fusion feature vector fusion model of the inspection target is obtained; The multimodal fusion feature vector fusion model represents information that integrates multimodal data and is used to reflect status information of the inspected target.

6. The intelligent inspection method based on data collection and multimodal AI according to claim 1 is characterized in that: A multimodal fusion feature vector benchmark model of the inspection target in a standard state is prefabricated in advance, and the inspection target is analyzed according to the multimodal fusion feature vector fusion model and the multimodal fusion feature vector benchmark model to obtain the working state of the inspection target, specifically: Prefabricate a multi-modal fusion feature vector benchmark model of an inspection target in a standard state, and obtain a multi-modal fusion feature vector fusion model of an inspection target; Comparing the multimodal fusion feature vector fusion model of the inspection target with the multimodal fusion feature vector benchmark model dimension by dimension; For each dimension of the modal feature in the multimodal fusion feature vector fusion model of the inspection target, determine whether it falls within the value range corresponding to the multimodal fusion feature vector benchmark model; If the modal feature of a certain dimension in the multimodal fusion feature vector fusion model of the inspection target does not fall within the value range corresponding to the multimodal fusion feature vector benchmark model, it is marked as an abnormal dimension; If the modal feature of a certain dimension in the multimodal fusion feature vector fusion model of the inspection target falls within the value range corresponding to the multimodal fusion feature vector benchmark model, it is marked as a normal dimension.

7. The intelligent inspection method based on data collection and multimodal AI according to claim 6 is characterized in that: The following steps are also included: Determine whether there is an abnormal dimension in the multimodal fusion feature vector fusion model. If there is no abnormal dimension in the multimodal fusion feature vector fusion model, calibrate the inspection target as a normal state; If there are abnormal dimensions in the multimodal fusion feature vector fusion model, obtain the importance weight of each abnormal dimension in the entire multimodal fusion feature vector fusion model; If there is a situation where the importance weight of one or more abnormal dimensions in the entire multimodal fusion feature vector fusion model is greater than the preset weight value, the inspection target is marked as an abnormal state; If the importance weight of each abnormal dimension in the entire multimodal fusion feature vector fusion model is not greater than the preset weight value, the number of abnormal dimensions in the multimodal fusion feature vector fusion model is ratioed to the preset number to obtain the abnormal dimension ratio; If the proportion of the abnormal dimension is greater than the preset proportion, the inspection target is marked as an abnormal state; if the proportion of the abnormal dimension is not greater than the preset proportion, the inspection target is marked as a normal state.

8. An intelligent inspection system based on data collection and multimodal AI, characterized in that: The intelligent inspection system based on data acquisition and multimodal AI includes a memory and a processor, wherein the memory stores an intelligent inspection method program based on data acquisition and multimodal AI. When the intelligent inspection method program based on data acquisition and multimodal AI is executed by the processor, the steps of the intelligent inspection method based on data acquisition and multimodal AI as described in any one of claims 1 to 7 are implemented.

9. A terminal device, applied to the intelligent inspection method based on data collection and multimodal AI according to any one of claims 1 to 7, characterized in that: include Data acquisition module, used to obtain multimodal data of inspection targets; Voxel space construction module, used to construct voxel space; A data consistency judgment and analysis module is used to perform data consistency judgment and analysis on each voxel block in the voxel space; The model comparison and analysis module is used to compare and analyze the multimodal fusion feature vector fusion model with the multimodal fusion feature vector benchmark model.

Images