A multimodal analysis method, system, device and medium for industrial inspection scenes

Abstract

The present application discloses a multimodal analysis method, system, device and medium for industrial inspection scenes. The method includes: real-time acquisition of multimodal data of industrial inspection areas, including video images and voice data; inputting the currently acquired multimodal data into a scene type recognition model to identify the scene type; matching the corresponding multimodal fusion method based on the determined scene type; according to the matched multimodal fusion method, respectively using a deep learning algorithm to extract features from video images, using natural language processing technology to extract features from text converted from voice data, and performing multimodal fusion on the extracted features; using a deep learning algorithm to perform joint reasoning on multimodal fusion features to identify potential risks or equipment failures in industrial inspection areas; generating inspection reports and decisions based on the identified potential risks or equipment failures. The present application can realize intelligent analysis of industrial inspection scenes and improve inspection efficiency and accuracy.

Description

Technical Field

[0001] The present application relates to the technical fields of artificial intelligence, computer vision and natural language processing, and specifically to a multimodal analysis method, system, equipment and medium for industrial inspection scenarios. Background Art

[0002] The level of industrial automation and intelligence is constantly improving. As an important part of ensuring production safety and normal operation of equipment, the level of intelligence of industrial inspection is directly related to the production efficiency and economic benefits of enterprises. Traditional inspection methods rely on manual labor, which has problems such as low efficiency and prone to errors. Although some inspection systems have adopted video surveillance technology, most of them only stay at the stage of video recording and playback, lacking in-depth scene understanding and intelligent analysis.

[0003] In recent years, multimodal learning has made significant progress in the field of artificial intelligence. By integrating information from different modalities, it is possible to understand complex scenes more comprehensively. However, in the field of industrial inspection, video understanding and natural language processing algorithms based on multimodal models are still immature and difficult to meet the inspection needs of actual complex inspection scenarios. Therefore, it is necessary to further improve the intelligence level of industrial inspection, improve inspection efficiency and accuracy, and reduce the cost of manual intervention. Summary of the invention

[0004] In order to realize intelligent analysis of industrial inspection scenarios and improve inspection efficiency and accuracy, the present application provides a multimodal analysis method, system, equipment and medium for industrial inspection scenarios.

[0005] In a first aspect, the present application provides a multimodal analysis method for an industrial inspection scenario, comprising:

[0006] Real-time collection of multi-modal data in industrial inspection areas, including video images and voice data;

[0007] Input the currently collected multimodal data into a scene type recognition model to identify the scene type, including: equipment-intensive, personnel-intensive, and equipment-personnel-intensive; the scene type recognition model uses a neural network model and is generated through multimodal data training of industrial inspection areas with historically labeled scene types;

[0008] Based on the determined scene type matching corresponding multimodal fusion mode, the multimodal fusion mode includes: multi-scale feature fusion matching with equipment intensive, cross-modal interactive fusion matching with personnel intensive, and multi-scale feature and cross-modal interactive joint fusion matching with equipment and personnel intensive;

[0009] According to the matching multimodal fusion method, the deep learning algorithm is used to extract features from the video image, and the natural language processing technology is used to extract features from the text converted from the voice data, and the extracted features are fused in a multimodal manner;

[0010] Use deep learning algorithms to jointly infer multimodal fusion features to identify potential risks or equipment failures in industrial inspection areas; generate inspection reports and decisions based on the identified potential risks or equipment failures.

[0011] By adopting the above solution, different types of industrial inspection scenarios are determined and suitable multimodal fusion methods are matched based on the characteristics of different scenario types, thereby integrating multimodal information such as video images, voice text, etc., to achieve intelligent analysis of different industrial inspection scenarios, timely and accurately identify potential risks and equipment failures in industrial inspection areas, and improve inspection efficiency and accuracy.

[0012] Preferably, it also includes:

[0013] Collect environmental data of industrial inspection areas in real time;

[0014] The current environmental data, scene type and scene type matching multimodal fusion method are input into the constructed optimal fusion weight acquisition model, the optimal fusion weight corresponding to the current environmental data is output, and the extracted features are multimodally fused according to the optimal fusion weight; the constructed optimal fusion weight acquisition model is generated by training the fusion weights annotated by several experts under the multimodal fusion method of historical environmental data, scene type and scene type matching.

[0015] By adopting the above scheme, considering the impact of environmental factors on the feature extraction of video images and voice data, the design dynamically adjusts the fusion weights of different modal features according to environmental characteristics, thereby improving the efficiency and accuracy of inspections.

[0016] Preferably, it also includes:

[0017] When the multimodal fusion method based on the determined scene type matching is the joint fusion of multi-scale features and cross-modal interactions, the personnel violation rate or equipment failure rate in the potential risks of different industrial inspection areas identified in the previous preset time period at the current moment is statistically analyzed; it is determined whether there is a statistical potential risk in the same area as the industrial inspection area collected at the current moment, and the personnel violation rate has increased to a preset violation rate or the equipment failure rate has increased to a preset failure rate; if there is a statistical equipment failure rate in the same area as the industrial inspection area collected at the current moment that has increased to a preset failure rate, then when the multimodal weighted fusion is performed on the features extracted at the current moment, the feature weights of the cross-modal interactive fusion are increased; if there is a statistical potential risk in the same area as the industrial inspection area collected at the current moment that has increased to a preset violation rate, then when the multimodal weighted fusion is performed on the features extracted at the current moment, the feature weights of the multi-scale feature fusion are increased; otherwise, no adjustment is made.

[0018] By adopting the above scheme, the historically detected equipment failure rate and personnel violation rate are used as feedback data. The weights of each fusion method in the multimodal joint fusion corresponding to the feedback data are used to better detect the current equipment failure or personnel violation and improve the accuracy of the inspection.

[0019] Preferably, it also includes:

[0020] Obtain the inspection task of the industrial inspection area, obtain the size of the industrial inspection area in the inspection task, determine the preset size range of the size of the currently obtained industrial inspection area, and match the multi-scale feature fusion of different scales according to the determined preset size range; wherein, the industrial inspection areas in different preset size ranges are correspondingly set with multi-scale feature fusion of different scales.

[0021] By adopting the above scheme, features of different sizes are fused based on the size range of the industrial inspection area, so that smaller inspection areas can be fused at a finer scale to capture more detail information, and larger inspection areas can be fused at a coarser scale to increase the processing speed.

[0022] Preferably, it also includes:

[0023] Obtain inspection tasks in industrial inspection areas, obtain the inspection difficulty and number of inspection tasks in the industrial inspection areas; determine the inspection complexity level based on the obtained inspection difficulty and number of tasks, and set corresponding conditions for different inspection complexity levels, and the conditions for meeting the conditions include: the number of inspection tasks with a preset inspection difficulty reaches a preset number; for the complexity level of the currently determined inspection tasks, match cross-modal interactive fusion with different thresholds; wherein, inspection tasks at different complexity levels are correspondingly set with cross-modal interactive fusion with different thresholds.

[0024] By adopting the above scheme, the inspection complexity of the industrial inspection area is determined. For inspection scenarios with a higher level of inspection complexity, the threshold is lowered to increase the sensitivity of cross-modal interaction, and in inspection scenarios with a lower level of inspection complexity, the threshold is increased to reduce the false alarm rate.

[0025] Preferably, it also includes:

[0026] Obtain inspection tasks for industrial inspection areas, determine the inspection difficulty and number of inspection tasks in the industrial inspection areas, calculate and obtain the average recognition time for identifying potential risks or equipment failures in the industrial inspection areas under the same inspection difficulty and number of tasks in a period before the current moment; compare and determine whether the average recognition time obtained is greater than a first preset time. If the calculated average time is greater than the first preset time, transmit the currently extracted multimodal features to an edge device with rich computing resources, and use the edge device to perform joint reasoning on the multimodal fusion features to identify potential risks or equipment failures in the industrial inspection areas.

[0027] By adopting the above solution, edge devices with rich computing resources can be used to identify complex inspection tasks with long identification time, thereby improving the efficiency of industrial inspection.

[0028] Preferably, it also includes:

[0029] Compare and calculate whether the average recognition time obtained is greater than the second preset time. If it is greater than the second preset time, transmit the currently extracted multimodal features to the backup deep learning model, and use the backup deep learning model to perform joint reasoning on the multimodal fusion features to identify potential risks or equipment failures in the industrial inspection area.

[0030] By adopting the above solution, considering that the recognition time is too long and there may be a fault in the model corresponding to the recognition algorithm, the backup model can be replaced in time to timely identify potential risks or equipment failures in the industrial inspection area, thereby ensuring the accuracy of industrial inspections.

[0031] In a second aspect, the present application provides a multimodal analysis system for industrial inspection scenarios, including:

[0032] Multimodal data acquisition module, used to collect multimodal data of industrial inspection areas in real time, including video images and voice data;

[0033] The multimodal data scene determination module is used to input the currently collected multimodal data into the scene type recognition model to identify the scene type, including: equipment-intensive, personnel-intensive, and equipment-personnel-intensive; the scene type recognition model adopts a neural network model and is generated by training the multimodal data of the industrial inspection area with historically marked scene types;

[0034] A multimodal fusion mode determination module, used for matching a corresponding multimodal fusion mode based on the determined scene type, wherein the multimodal fusion mode includes: multiscale feature fusion matching with equipment intensiveness, cross-modal interactive fusion matching with personnel intensiveness, and multiscale feature and cross-modal interactive joint fusion matching with equipment and personnel intensiveness;

[0035] The multimodal data fusion module is used to extract features from video images using a deep learning algorithm and extract features from text converted from speech data using natural language processing technology according to the matched multimodal fusion method, and then perform multimodal fusion on the extracted features;

[0036] The multimodal data analysis result acquisition module is used to use deep learning algorithms to jointly infer multimodal fusion features to identify potential risks or equipment failures in industrial inspection areas; and generate inspection reports and decisions based on the identified potential risks or equipment failures.

[0037] By adopting the above solution, intelligent analysis of industrial inspection scenarios can be achieved, improving inspection efficiency and accuracy.

[0038] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method as described above.

[0039] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a program stored and executable on the memory, wherein the program implements the steps of the above method when executed by the processor.

[0040] In summary, this application has the following beneficial effects:

[0041] 1. By identifying different types of industrial inspection scenarios, matching appropriate multi-scale fusion, cross-modal interactive fusion, or multi-scale features and cross-modal interactive joint fusion according to the characteristics of different scenario types, more detailed and comprehensive video image features, voice and text features are obtained and integrated into rich multi-modal information, realizing intelligent analysis of different industrial inspection scenarios, timely and accurately identifying potential risks and equipment failures in industrial inspection areas, and improving inspection efficiency and accuracy;

[0042] 2. By collecting environmental data or identifying feedback information from inspection results, dynamically adjust the weight of feature fusion or the fusion weight of different fusion methods to achieve more accurate inspections;

[0043] 3. Considering the size range of industrial inspections and the complexity of industrial inspection tasks, the specific scale of multi-size fusion and the threshold of determining feature association of cross-modal interactive fusion are adjusted accordingly to improve inspection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flowchart of the multimodal analysis method of the industrial inspection scene described in the specific embodiment;

[0045] Figure 2 It is a structural schematic diagram of the multimodal analysis system for the industrial inspection scenario described in the specific embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] like Figure 1 As shown, the embodiment of the present application discloses a multimodal analysis method for industrial inspection scenarios, and the specific steps are as follows:

[0048] S1. Real-time collection of multimodal data in industrial inspection areas.

[0049] Specifically, in the industrial inspection scenario, according to the inspection tasks issued by the inspectors, inspection equipment such as cameras and microphones deployed in industrial inspection areas (such as production lines, equipment rooms, transmission lines, etc.) are used to collect multimodal data of the industrial inspection areas in the inspection tasks, that is, real-time collection of video images, voice data, and text record data such as equipment status change data in the industrial inspection area during the inspection process.

[0050] In this embodiment, the video data and voice data collected during the inspection process are V and A, respectively, which are composed of video frame sequences and audio signal Composition, assuming that at time t, the multimodal data of video and voice data is , the time series of multimodal data can be expressed as .

[0051] S2. Input the currently collected multimodal data into a scene type recognition model to identify the scene type.

[0052] Specifically, an industrial inspection scene type recognition model is constructed, and a neural network model is selected. The input of the model is the multimodal data of the collected industrial inspection area, and the output is the scene type, including: equipment intensive, personnel intensive, and equipment and personnel intensive. It is generated through multimodal data training of industrial inspection areas with historically labeled scene types.

[0053] S3. Match the corresponding multimodal fusion method based on the determined scene type.

[0054] Specifically, considering the adaptability of the multimodal fusion method in different scenarios, it is convenient for subsequent more accurate inspection and identification. For example, in equipment-intensive areas, more emphasis is placed on multi-scale fusion to capture the details of the equipment, and in personnel-intensive areas, more emphasis is placed on cross-modal interaction to analyze personnel behavior. Therefore, the multimodal fusion method includes: multi-scale feature fusion matching equipment-intensive, cross-modal interaction fusion matching personnel-intensive, and multi-scale feature and cross-modal interaction joint fusion matching equipment-personnel-intensive.

[0055] S4. According to the matching multimodal fusion method, complete the multimodal data feature extraction and fusion.

[0056] First, multimodal feature extraction is performed on multimodal data, including:

[0057] Generally, deep learning algorithms are used to extract features from video images, such as using convolutional neural networks to process videos and extract spatiotemporal features. in, Represents a video frame The spatiotemporal feature vector is further obtained by using a temporal convolutional network (TCN) or a long short-term memory network (LSTM) to capture the temporal relationship between video frames. For speech data, the speech signal is transformed into Converted to text T, represented as ,in, For the i-th word, natural language processing technology or pre-trained language models (such as BERT or GPT) are used to extract features from the text converted from voice data to obtain the semantic representation of the text: H = BERT (T), where H represents the semantic feature vector of the text. In addition, in addition to the voice data converted into text, the text T can also include equipment operating status data, personnel operation guides, safety regulations, etc. originally recorded in text form.

[0058] When feature extraction is performed according to the matching multimodal fusion method, it specifically includes:

[0059] For video image and voice data, cross-modal interactive fusion is selected, and the features F and H extracted by the above general feature extraction method are used for feature matching. If the matching degree between the video features and the voice features exceeds the pre-matching threshold, that is, the matching degree between these features calculated by a certain matching algorithm (such as distance measurement, correlation analysis, etc.) exceeds the pre-matching threshold, it is considered that there is a correlation between the features, and the video image features and voice data features with correlation are obtained. Otherwise, it is not considered to be associated. For example, "conveyor belt jam" in the video feature and "device A temperature abnormality" in the voice data conversion text feature are considered as associated feature information.

[0060] For video images and speech data, if multi-scale feature fusion is used, multi-scale feature extraction is required. The corresponding video images can use convolution layers with different receptive fields or construct feature pyramids to extract features from feature layers of different scales. Perform multi-scale feature extraction to extract features Indicates that the nth convolutional layer obtains the extracted features; for text converted from speech data, word-level features, sentence-level features, or instruction-level features can be extracted respectively, that is, are the features extracted at n scales.

[0061] For video images and speech data, multi-scale features and cross-modal interaction are used for joint fusion. This requires multi-scale video image features F and text features H converted from speech data, and also requires obtaining video image features and speech data features that have a correlation relationship. .

[0062] Secondly, according to the matching multimodal fusion method, the acquired multimodal features are fused.

[0063] The formula for fusion using multi-scale feature fusion is as follows:

[0064] in, is the fusion weight, It represents the fusion operation, F and H are the extracted multi-scale features, each scale feature has a corresponding weight, and Z is the comprehensive feature after fusion.

[0065] The formula for fusion using cross-scale feature fusion is as follows:

[0066] in, is the fusion weight, and .

[0067] The formula for the joint fusion of multi-scale features and cross-modal interactions is as follows:

[0068] .

[0069] S5. Use deep learning algorithms to jointly reason about multimodal fusion features to identify potential risks or equipment failures in industrial inspection areas.

[0070] Specifically, a deep learning model is constructed using a deep learning algorithm to perform joint reasoning on multimodal fusion features. It can be assumed that the reasoning process is expressed as a classification problem, that is, the deep learning model is a classification model. The classification model MLP is used to classify Z to obtain the probability distribution of equipment failure or personnel violation risks. The formula is as follows:

[0071] Among them, y is the category label of personnel violation or equipment failure.

[0072] S6. Generate inspection reports and decisions based on identified potential risks or equipment failures.

[0073] Specifically, a visual interface can be designed to directly display the generated inspection report, and make corresponding preset decisions based on the typical personnel violations and equipment failures in the inspection report, such as timely generating voice prompts for personnel violations and timely generating equipment failure prompt information and sending them to the maintenance personnel terminal.

[0074] In a specific embodiment, considering that different environmental factors have an impact on the extraction of video features and text features of speech conversion, in order to extract better features and perform subsequent risk and fault identification, the method further includes:

[0075] Collect environmental data of industrial inspection areas in real time; the environmental data includes: light, noise, etc.

[0076] Construct an optimal fusion weight acquisition model, select a neural network model, the input of the model is the current environmental data, scene type and multimodal fusion method of scene type matching, and the output is the optimal fusion weight corresponding to the current environmental data, which is specifically generated by training the fusion weights annotated by several experts under the multimodal fusion method of different historical environmental data, scene types and scene type matching;

[0077] The current environmental data, scene type, and multimodal fusion method matching the scene type are input into the optimal fusion weight acquisition model, and the optimal fusion weight corresponding to the current environmental data is output. The extracted features are multimodally fused according to the optimal fusion weight, that is, the weights of video features, voice data converted into text features, or features with a correlation between the two are adjusted, thereby dynamically completing the weight adjustment and improving the accuracy of subsequent inspections.

[0078] For example: in an environment with sufficient light, the weight of the video modal features will be appropriately increased compared to the optimal weight before adjustment to improve the utilization of visual information; in an environment with high noise, the weight of the audio modal features will be increased compared to the optimal weight before adjustment to better capture abnormal information in the sound.

[0079] In a specific embodiment, taking into account real-time feedback data such as an increase in equipment failure rate or personnel violation rate, the corresponding feature weights can be dynamically adjusted to achieve better inspection results. The method also includes:

[0080] When the multimodal fusion method based on the determined scene type matching is the joint fusion of multi-scale features and cross-modal interactions, the personnel violation rate or equipment failure rate in the potential risks of different industrial inspection areas identified in the previous preset time period at the current moment is statistically determined; the previous preset time period is artificially set.

[0081] Determine whether there is a statistical potential risk in the same area as the industrial inspection area collected at the current moment, in which the personnel violation rate has increased to a preset violation rate or the equipment failure rate has increased to a preset failure rate;

[0082] If the equipment failure rate in the statistics of the same industrial inspection area as the one collected at the current moment rises to the preset failure rate, it means that there are many faulty equipment in the current industrial inspection area, and more attention needs to be paid to the inspection of the faulty equipment. The weight of multi-scale fusion can be further increased, so that when performing multi-modal weighted fusion on the features extracted at the current moment, the feature weight of cross-modal interactive fusion is increased;

[0083] If the personnel violation rate in the statistical potential risk belonging to the same area as the industrial inspection area collected at the current moment rises to the preset violation rate, then the current personnel have many violations and the weight of cross-modal interaction needs to be increased. When performing multi-modal weighted fusion on the features extracted at the current moment, the feature weight of multi-scale feature fusion is increased; otherwise, the feature weight is not adjusted, and the weight of multi-scale feature fusion preset in the initial state and the feature weight of cross-modal interaction fusion are maintained; among which, the specific feature weight increased each time can be pre-set and the weight increase threshold can be set.

[0084] In a specific embodiment, taking into account the size ranges of different industrial inspection areas and different inspection difficulties, the requirements for feature extraction are different. Correspondingly, different feature fusion methods can be used to adaptively set the parameter content of the feature fusion method according to the specific size and difficulty conditions to improve the inspection accuracy; the method also includes: obtaining the inspection task of the industrial inspection area, and obtaining the size of the industrial inspection area in the inspection task.

[0085] According to the size of the currently acquired industrial inspection area, the preset size range in which it is located is determined, such as: a first preset size range, a second preset size range and a third preset size range, and the size range gradually increases; according to the determined preset size range, multi-scale feature fusion of different scales is matched, and the multi-scale feature fusion of different scales specifically refers to the number of convolution layers of different sizes, including: multi-scale feature fusion with a first number of convolution layers of different sizes, multi-scale feature fusion with a second number of convolution layers of different sizes, and multi-scale feature fusion with a third number of convolution layers of different sizes, etc., and the number gradually decreases.

[0086] Among them, industrial inspection areas in different preset size ranges are correspondingly set with multi-scale feature fusions of different scales, such as: the first preset size range corresponds to a multi-scale feature fusion matching a first number of convolutional layers of different sizes, so that small areas can be inspected in detail and large areas can be inspected quickly.

[0087] In addition, it also includes: obtaining the inspection tasks of the industrial inspection area, obtaining the inspection difficulty and task quantity of the inspection tasks in the industrial inspection area; among which, the inspection difficulty includes simple, medium and difficult;

[0088] The inspection complexity level (simple, general and complex, with increasing complexity levels) is determined based on the inspection difficulty and the number of tasks obtained. Different inspection complexity levels are set to meet the conditions, which include: the number of inspection tasks with a preset inspection difficulty reaches a preset number; for example: the first condition: when the inspection tasks with a difficult inspection difficulty reaches a first preset number, the inspection complexity level is determined to be complex; the second condition: when the inspection tasks with a medium inspection difficulty reaches a second preset number, the inspection complexity level is determined to be general; the third condition: when the inspection tasks with a simple inspection difficulty reaches a third preset number, the inspection complexity level is determined to be simple.

[0089] According to the complexity level of the currently determined inspection task, cross-modal interaction fusion with different thresholds is matched; among them, inspection tasks at different complexity levels are correspondingly set with cross-modal interaction fusion with different thresholds; among them, the higher the complexity level, the higher the setting, the lower the threshold, and the higher the sensitivity of cross-modal interaction;

[0090] In inspection scenarios with a lower level of inspection complexity, increase the threshold to reduce the false alarm rate.

[0091] In a specific embodiment, considering that accurate inspection needs to be achieved while ensuring inspection efficiency, the method further includes:

[0092] Obtain the inspection tasks of the industrial inspection area, determine the inspection difficulty and number of tasks in the industrial inspection area, and calculate the average recognition time for identifying potential risks or equipment failures in the industrial inspection area under the same inspection difficulty and number of tasks in a period before the current moment;

[0093] Compare and calculate whether the average recognition time obtained is greater than the first preset time. If the calculated average time is greater than the first preset time, it indicates that the average recognition time for identifying potential risks or equipment failures in the current industrial inspection area under the current inspection difficulty and task quantity conditions is long, and the computing power of existing computing resources is limited. Therefore, the currently extracted multimodal features are transmitted to the edge device with rich computing resources, where the rich computing resources may include a high-performance processor that reaches a preset processing speed and a large-capacity memory that reaches a preset capacity. The first preset time is set according to the needs of the inspection personnel.

[0094] The deep learning model used to identify potential risks or equipment failures on edge devices is used to jointly infer multimodal fusion features to identify potential risks or equipment failures in industrial inspection areas.

[0095] In addition, the method further comprises:

[0096] Compare and calculate whether the average recognition time obtained is greater than a second preset time. If it is greater than the second preset time, it indicates that recognition failure may occur. The second preset time is determined based on the average recognition time of the potential risks or equipment failures in the industrial inspection area that were not successfully identified under the same inspection difficulty and task quantity conditions in a period of time before the current moment.

[0097] The currently extracted multimodal features are transmitted to the backup deep learning model, and the backup deep learning model is used to perform joint reasoning on the multimodal fusion features to identify potential risks or equipment failures in the industrial inspection area.

[0098] like Figure 2 As shown, the embodiment of the present application discloses a multimodal analysis system for industrial inspection scenarios, including:

[0099] The multimodal data acquisition module 101 is used to collect multimodal data of the industrial inspection area in real time, including video images and voice data;

[0100] The multimodal data scene determination module 102 is used to input the currently collected multimodal data into a scene type recognition model to identify the scene type, including: equipment-intensive, personnel-intensive, and equipment-personnel-intensive; the scene type recognition model adopts a neural network model and is generated by training multimodal data of industrial inspection areas with historically labeled scene types;

[0101] A multimodal fusion mode determination module 103 is used to match the corresponding multimodal fusion mode based on the determined scene type, wherein the multimodal fusion mode includes: multiscale feature fusion matching with equipment intensiveness, cross-modal interactive fusion matching with personnel intensiveness, and multiscale feature and cross-modal interactive joint fusion matching with equipment and personnel intensiveness;

[0102] The multimodal data fusion module 104 is used to extract features from the video image using a deep learning algorithm and extract features from the text converted from the voice data using a natural language processing technology according to the matched multimodal fusion mode, and perform multimodal fusion on the extracted features;

[0103] The multimodal data analysis result acquisition module 105 is used to use a deep learning algorithm to perform joint reasoning on multimodal fusion features to identify potential risks or equipment failures in industrial inspection areas; and generate inspection reports and decisions based on the identified potential risks or equipment failures.

[0104] In a specific embodiment, the multimodal data acquisition module 101 in the system is also used to collect environmental data of the industrial inspection area in real time; the multimodal data fusion module 104 is also used to input the current environmental data, scene type and the multimodal fusion method of scene type matching into the constructed optimal fusion weight acquisition model, output the optimal fusion weight corresponding to the current environmental data, and perform multimodal fusion on the extracted features according to the optimal fusion weight; the constructed optimal fusion weight acquisition model is generated by training the fusion weights annotated by several experts under the multimodal fusion method of historical environmental data, scene type and scene type matching.

[0105] In a specific embodiment, the multimodal fusion mode determination module 103 in the system is also used to, when the multimodal fusion mode based on the determined scene type matching is the joint fusion of multi-scale features and cross-modal interactions, statistically identify the personnel violation rate or equipment failure rate in the potential risks of different industrial inspection areas in the previous preset time period at the current moment; determine whether there is a statistical potential risk that belongs to the same area as the industrial inspection area collected at the current moment, and the personnel violation rate has increased to a preset violation rate or the equipment failure rate has increased to a preset failure rate; if there is a statistical equipment failure rate that belongs to the same area as the industrial inspection area collected at the current moment and has increased to a preset failure rate, then when performing multimodal weighted fusion on the features extracted at the current moment, the feature weights of the cross-modal interactive fusion are increased; if there is a statistical potential risk that belongs to the same area as the industrial inspection area collected at the current moment and the personnel violation rate has increased to a preset violation rate, then when performing multimodal weighted fusion on the features extracted at the current moment, the feature weights of the multi-scale feature fusion are increased; otherwise, no adjustment is made.

[0106] In a specific embodiment, the multimodal fusion method determination module 103 in the system is also used to obtain inspection tasks in industrial inspection areas, obtain the size of the industrial inspection areas in the inspection tasks, determine the preset size range of the size of the currently obtained industrial inspection area, and match multi-scale feature fusion of different scales according to the determined preset size range; wherein, industrial inspection areas in different preset size ranges are correspondingly provided with multi-scale feature fusion of different scales; it is also used to obtain inspection tasks in industrial inspection areas, obtain the inspection difficulty and the number of tasks of the inspection tasks in the industrial inspection areas; determine the inspection complexity level according to the obtained inspection difficulty and the number of tasks, and different inspection complexity levels are correspondingly set with satisfying conditions, and the satisfying conditions include: the number of inspection tasks with a preset inspection difficulty reaches a preset number; for the complexity level of the currently determined inspection task, match cross-modal interactive fusion with different thresholds; wherein, inspection tasks at different complexity levels are correspondingly provided with cross-modal interactive fusion with different thresholds.

[0107] In a specific embodiment, the multimodal data analysis result acquisition module 105 in the system is also used to obtain inspection tasks in the industrial inspection area, determine the inspection difficulty and number of inspection tasks in the industrial inspection area, calculate and obtain the average recognition time for identifying potential risks or equipment failures in the industrial inspection area under the same inspection difficulty and number of tasks in a period before the current moment; compare and calculate whether the average recognition time obtained is greater than a first preset time. If the calculated average time is greater than the first preset time, the currently extracted multimodal features are transmitted to an edge device with rich computing resources, and the edge device is used to perform joint reasoning on the multimodal fusion features to identify potential risks or equipment failures in the industrial inspection area.

[0108] In a specific embodiment, the multimodal data analysis result acquisition module 105 in the system is also used to compare whether the calculated average recognition time is greater than a second preset time. If it is greater than the second preset time, the currently extracted multimodal features are transmitted to the backup deep learning model, and the backup deep learning model is used to perform joint reasoning on the multimodal fusion features to identify potential risks or equipment failures in the industrial inspection area.

[0109] The embodiment of the present application also discloses a computer-readable storage medium.

[0110] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and execute a multimodal analysis method for the above-mentioned industrial inspection scenario. The computer-readable storage medium includes, for example: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0111] The embodiment of the present application also discloses a computer device.

[0112] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the multimodal analysis method for the above-mentioned industrial inspection scenario.

[0113] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A multimodal analysis method for industrial inspection scenarios, characterized in that: include: Real-time collection of multi-modal data in industrial inspection areas, including video images and voice data; Input the currently collected multimodal data into a scene type recognition model to identify the scene type, including: equipment-intensive, personnel-intensive, and equipment-personnel-intensive; the scene type recognition model uses a neural network model and is generated through multimodal data training of industrial inspection areas with historically labeled scene types; Based on the determined scene type matching corresponding multimodal fusion mode, the multimodal fusion mode includes: multi-scale feature fusion matching with equipment intensive, cross-modal interactive fusion matching with personnel intensive, and multi-scale feature and cross-modal interactive joint fusion matching with equipment and personnel intensive; According to the matching multimodal fusion method, the deep learning algorithm is used to extract features from the video image, and the natural language processing technology is used to extract features from the text converted from the voice data, and the extracted features are fused in a multimodal manner; specifically, the following are included: The formula for fusion using multi-scale feature fusion is as follows: ;in, is the fusion weight, represents the fusion operation, F and H are the extracted multi-scale features, each scale feature has a corresponding weight, and Z is the comprehensive feature after fusion; The formula for fusion using cross-scale feature fusion is as follows: ;in, is the fusion weight, and ; The formula for the joint fusion of multi-scale features and cross-modal interactions is as follows: ； Use deep learning algorithms to jointly infer multimodal fusion features to identify potential risks or equipment failures in industrial inspection areas; generate inspection reports and decisions based on the identified potential risks or equipment failures.

2. The multimodal analysis method for industrial inspection scenarios according to claim 1 is characterized in that: Also includes: Collect environmental data of industrial inspection areas in real time; The current environmental data, scene type and scene type matching multimodal fusion method are input into the constructed optimal fusion weight acquisition model, the optimal fusion weight corresponding to the current environmental data is output, and the extracted features are multimodally fused according to the optimal fusion weight; the constructed optimal fusion weight acquisition model is generated by training the fusion weights annotated by several experts under the multimodal fusion method of historical environmental data, scene type and scene type matching.

3. The multimodal analysis method for industrial inspection scenarios according to claim 1 is characterized in that: Also includes: When the multimodal fusion method based on the determined scene type matching is the joint fusion of multi-scale features and cross-modal interactions, the personnel violation rate or equipment failure rate in the potential risks of different industrial inspection areas identified in the previous preset time period at the current moment is statistically analyzed; Determine whether there is a statistical potential risk belonging to the same area as the industrial inspection area collected at the current moment, in which the personnel violation rate has increased to a preset violation rate or the equipment failure rate has increased to a preset failure rate; if there is a statistical equipment failure rate belonging to the same area as the industrial inspection area collected at the current moment, in which the failure rate has increased to a preset failure rate, when performing multimodal weighted fusion on the features extracted at the current moment, increase the feature weights of cross-modal interactive fusion; If the personnel violation rate in the statistical potential risk belonging to the same area as the industrial inspection area collected at the current moment rises to the preset violation rate, the feature weight of the multi-scale feature fusion is increased when the multimodal weighted fusion is performed on the features extracted at the current moment; otherwise, no adjustment is made.

4. The multimodal analysis method for industrial inspection scenarios according to claim 1 is characterized in that: Also includes: Obtain the inspection task of the industrial inspection area, obtain the size of the industrial inspection area in the inspection task, determine the preset size range of the size of the currently obtained industrial inspection area, and match the multi-scale feature fusion of different scales according to the determined preset size range; wherein, the industrial inspection areas in different preset size ranges are correspondingly set with multi-scale feature fusion of different scales.

5. The multimodal analysis method for industrial inspection scenarios according to claim 1 is characterized in that: Also includes: Obtain the inspection tasks in the industrial inspection area, and obtain the inspection difficulty and number of tasks in the industrial inspection area; The inspection complexity level is determined based on the obtained inspection difficulty and number of tasks, and different inspection complexity levels are correspondingly set to meet the conditions, and the conditions are included in that: the number of inspection tasks with a preset inspection difficulty reaches a preset number; for the complexity level of the currently determined inspection tasks, cross-modal interaction fusion with different thresholds is matched; wherein, inspection tasks at different complexity levels are correspondingly set with cross-modal interaction fusion with different thresholds.

6. The multimodal analysis method for industrial inspection scenarios according to claim 1 is characterized in that: Also includes: Obtain inspection tasks for industrial inspection areas, determine the inspection difficulty and number of inspection tasks in the industrial inspection areas, calculate and obtain the average recognition time for identifying potential risks or equipment failures in the industrial inspection areas under the same inspection difficulty and number of tasks in a period before the current moment; compare and determine whether the average recognition time obtained is greater than a first preset time. If the calculated average time is greater than the first preset time, transmit the currently extracted multimodal features to an edge device with rich computing resources, and use the edge device to perform joint reasoning on the multimodal fusion features to identify potential risks or equipment failures in the industrial inspection areas.

7. The multimodal analysis method for industrial inspection scenarios according to claim 6 is characterized in that: Also includes: Compare and calculate whether the average recognition time obtained is greater than the second preset time. If it is greater than the second preset time, transmit the currently extracted multimodal features to the backup deep learning model, and use the backup deep learning model to perform joint reasoning on the multimodal fusion features to identify potential risks or equipment failures in the industrial inspection area.

8. A multimodal analysis system for industrial inspection scenarios, characterized in that: include: Multimodal data acquisition module, used to collect multimodal data of industrial inspection areas in real time, including video images and voice data; The multimodal data scene determination module is used to input the currently collected multimodal data into the scene type recognition model to identify the scene type, including: equipment-intensive, personnel-intensive, and equipment-personnel-intensive; the scene type recognition model adopts a neural network model and is generated by training the multimodal data of the industrial inspection area with historically marked scene types; A multimodal fusion mode determination module, used for matching a corresponding multimodal fusion mode based on the determined scene type, wherein the multimodal fusion mode includes: multiscale feature fusion matching with equipment intensiveness, cross-modal interactive fusion matching with personnel intensiveness, and multiscale feature and cross-modal interactive joint fusion matching with equipment and personnel intensiveness; The multimodal data fusion module is used to extract features from video images using deep learning algorithms and from text converted from speech data using natural language processing technology according to the matched multimodal fusion method, and to perform multimodal fusion on the extracted features; specifically, it includes: The formula for fusion using multi-scale feature fusion is as follows: ;in, is the fusion weight, represents the fusion operation, F and H are the extracted multi-scale features, each scale feature has a corresponding weight, and Z is the comprehensive feature after fusion; The formula for fusion using cross-scale feature fusion is as follows: ;in, is the fusion weight, and ; The formula for the joint fusion of multi-scale features and cross-modal interactions is as follows: ； The multimodal data analysis result acquisition module is used to use deep learning algorithms to jointly infer multimodal fusion features to identify potential risks or equipment failures in industrial inspection areas; and generate inspection reports and decisions based on the identified potential risks or equipment failures.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises a memory, a processor and a program stored and executable on the memory, and the program implements the steps of the method according to any one of claims 1 to 7 when executed by the processor.

Images