Multi-modal information fusion target identification method and system based on machine learning

Abstract

The invention provides a multi-modal information fusion target identification method and system based on machine learning. The method comprises the following steps: extracting audio sudden change features and picture abnormal features of a current target plant from multi-modal data through a training-based machine learning model; determining a plurality of abnormal segmentation images according to the image abnormal features, determining abnormal boundaries according to the audio mutation features and the spatial layout of the current target plant, and performing feature fusion on all the abnormal segmentation images and the abnormal boundaries to obtain an abnormal positioning area of the current target plant; performing layer-by-layer analysis on the area of the current target plant according to the abnormal positioning area in combination with a machine learning algorithm so as to obtain an abnormal influence index of the current target plant during operation; and identifying the abnormal condition of the current target plant in operation through the abnormal influence index. According to the scheme, the abnormal area in the plant can be quickly positioned through the multi-modal information, so that the safety of the plant in the operation process is improved.

Description

Technical Field

[0001] The present application relates to the field of information fusion technology, and more specifically, to a multimodal information fusion target recognition method and system based on machine learning. Background Art

[0002] Information fusion refers to the process of collecting data from multiple sources, formats or sensors and processing them comprehensively to obtain more accurate, comprehensive and reliable information or conclusions. The goal of information fusion is to improve the quality and efficiency of decision-making by integrating information from different sources to reduce uncertainty and increase the credibility of results.

[0003] Multimodal information fusion target recognition based on machine learning refers to the use of machine learning technology to integrate data from multiple modes (such as images, texts, sounds, etc.) to improve the accuracy and robustness of target recognition. By making full use of the complementary information between different modes and combining machine learning models to extract and fuse features, more effective target recognition can be achieved. In the existing multimodal information fusion target recognition process, features are extracted from each piece of information in the multimodal information of the plant, and then all the extracted features are fused to identify abnormal situations in the plant. Since the collected multimodal information includes other information in addition to abnormal information, directly extracting features from the multimodal information will cause the extracted features to contain features of other interference information in addition to the target features, which will lead to deviations in the recognition of the plant and redundant time in the recognition process. Therefore, how to quickly locate abnormal areas in the plant through multimodal information to improve the safety of the plant during operation has become a problem faced by the industry. Summary of the invention

[0004] The present application provides a multimodal information fusion target recognition method and system based on machine learning, which can realize the rapid positioning of abnormal areas in a factory through multimodal information, thereby improving the safety of the factory during operation.

[0005] In a first aspect, the present application provides a multimodal information fusion target recognition method based on machine learning, comprising the following steps: Monitoring multimodal data of the current target plant in operation, wherein the multimodal data includes image data and audio data of the target plant; Extracting audio mutation features and image abnormal features of the current target factory from the multimodal data based on the trained machine learning model; Decouple the abnormal features of the image to obtain multiple abnormal segmentation maps for abnormal identification of the current target plant, determine the abnormal boundary when the sound in the current target plant is abnormal according to the audio mutation feature and the spatial layout of the current target plant, and perform feature fusion of all the abnormal segmentation maps and the abnormal boundary based on the convolutional neural network of machine learning to obtain the abnormal positioning area of ​​the current target plant; The abnormal location area is used as a constraint condition for abnormal identification of the current target plant, and the area of ​​the current target plant is analyzed layer by layer in combination with a machine learning algorithm, so as to obtain the abnormal impact index of the current target plant during operation; The abnormal condition of the current target plant in operation is identified through the abnormal impact index.

[0006] In some embodiments, extracting the audio mutation features and image abnormal features of the current target factory from the multimodal data based on the trained machine learning model specifically includes: Get the trained machine learning model; Using the multimodal data as initialization parameters of the machine learning model; The audio mutation characteristics and image abnormal characteristics of the current target factory are determined based on the machine learning model.

[0007] In some embodiments, feature decoupling of the abnormal features of the image to obtain multiple abnormal segmentation graphs for abnormal identification of the current target plant specifically includes: Determine the image segmentation threshold of the current target factory building; Selecting an abnormal picture with the abnormal picture feature as a selected abnormal picture, segmenting the selected abnormal picture by using the image segmentation threshold, and obtaining a plurality of segmented pictures of the selected abnormal picture; Determine the abnormal segmentation map corresponding to the abnormal image selected when performing abnormal identification on the current target plant based on all the segmentation maps; Continue to determine the abnormal segmentation maps corresponding to the remaining abnormal images when performing abnormality identification on the current target plant.

[0008] In some embodiments, determining the abnormal boundary when the sound in the current target plant is abnormal according to the audio mutation feature and the spatial layout of the current target plant specifically includes: Get the spatial layout of the current target plant; Determine multiple audio mutation areas of the current target plant during operation according to the audio mutation characteristics and the spatial layout; The abnormal boundary when the sound in the current target factory is abnormal is determined through all audio mutation areas.

[0009] In some embodiments, the convolutional neural network based on machine learning performs feature fusion on all abnormal segmentation maps and the abnormal boundaries, and then obtains the abnormal positioning area of ​​the current target plant, which specifically includes: The convolutional neural network based on machine learning performs feature fusion on all abnormal segmentation maps and the abnormal boundaries to obtain abnormal fusion information of the current target plant; Determine multiple abnormal fusion areas of the current target plant according to the abnormal fusion information; The abnormal location area of ​​the current target plant is determined through all abnormal fusion areas.

[0010] In some embodiments, the abnormal location area is used as a constraint condition for abnormal identification of the current target plant, and the area of ​​the current target plant is analyzed layer by layer in combination with a machine learning algorithm, so as to obtain the abnormal impact index of the current target plant during operation, which specifically includes: Using the abnormal location area as a constraint condition when performing abnormality identification on the current target plant; Determine a plurality of floor-by-floor outliers in the current target plant building according to the constraint conditions, the spatial layout of the current target plant building, and a machine learning algorithm; The abnormal impact index of the current target plant during operation is determined through all the abnormal values ​​layer by layer.

[0011] In some embodiments, identifying the abnormal condition of the current target plant in operation by using the abnormal impact index specifically includes: Determine the abnormal impact threshold range of the abnormal conditions in the operation of the current target plant; If the abnormal impact index is less than the lower limit of the abnormal impact threshold range, the abnormal condition of the current target plant in operation is a safe state; If the abnormal impact index is within the abnormal impact threshold range, the abnormal condition of the current target plant in operation is in a pending state; If the abnormal impact index is greater than the upper limit value of the abnormal impact threshold range, the abnormal condition of the current target plant in operation is an emergency state.

[0012] In a second aspect, the present application provides a multimodal information fusion target recognition system based on machine learning, comprising: A monitoring module, used to monitor the multimodal data of the current target plant in operation, wherein the multimodal data includes image data and audio data of the target plant; A processing module, used to extract audio mutation features and image abnormal features of the current target factory building from the multimodal data based on the trained machine learning model; The processing module is further used to perform feature decoupling on the abnormal features of the image to obtain multiple abnormal segmentation maps when abnormal identification is performed on the current target plant, determine the abnormal boundary when the sound in the current target plant is abnormal according to the audio mutation feature and the spatial layout of the current target plant, and perform feature fusion on all the abnormal segmentation maps and the abnormal boundary based on the convolutional neural network of machine learning, so as to obtain the abnormal positioning area of ​​the current target plant; The processing module is further used to use the abnormal location area as a constraint condition for abnormal identification of the current target plant, and combine the machine learning algorithm to analyze the area of ​​the current target plant layer by layer, so as to obtain the abnormal impact index of the current target plant during operation; The execution module is used to identify the abnormal condition of the current target plant in operation through the abnormal impact index.

[0013] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores code, and the processor is configured to obtain the code and execute the above-mentioned multimodal information fusion target recognition method based on machine learning.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multimodal information fusion target recognition method based on machine learning.

[0015] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects: In the multimodal information fusion target recognition method and system based on machine learning provided by the present application, firstly, the multimodal data of the current target plant in operation is monitored, and the multimodal data includes the image data and audio data of the target plant; based on the trained machine learning model, the audio mutation feature and the image abnormal feature of the current target plant are extracted from the multimodal data; the image abnormal feature is feature decoupled to obtain multiple abnormal segmentation maps when the current target plant is abnormally identified, and the abnormal boundary when the sound in the current target plant is abnormal is determined according to the audio mutation feature and the spatial layout of the current target plant. The convolutional neural network based on machine learning performs feature fusion on all the abnormal segmentation maps and the abnormal boundary, and then obtains the abnormal positioning area of ​​the current target plant; the abnormal positioning area is used as a constraint condition when the current target plant is abnormally identified, and the area of ​​the current target plant is analyzed layer by layer in combination with the machine learning algorithm, so as to obtain the abnormal impact index of the current target plant in operation; the abnormal condition of the current target plant in operation is identified by the abnormal impact index.

[0016] It can be seen that in the process of multimodal information fusion target recognition, the present application firstly extracts audio mutation features and image abnormality features from the multimodal data in the current target plant based on machine learning, and the audio mutation features are used to identify abnormal sound conditions in the current target plant; the image abnormality features are used to identify abnormal objects in the current target plant, so as to timely locate abnormal objects in the current target plant; a segmentation map of the current target plant for abnormal recognition is extracted from the image abnormality features, and then a plurality of abnormal segmentation maps are obtained, and the abnormal segmentation map is convenient for quickly locating abnormal areas in the current target plant; the boundary area where there is a sound mutation in the current target plant is extracted from the audio mutation features, and then the abnormal boundary is obtained, and the abnormal boundary can locate the abnormal situation in the current target plant, and then judge the abnormal situation; secondly, the volume based on machine learning The multiplication neural network fuses the features of all the abnormal segmentation maps and the abnormal boundaries, and then analyzes the area for locating the abnormal area when the current target plant is abnormal from the fused features, and determines the abnormal positioning area of ​​the current target plant. The abnormal positioning area can quickly locate the abnormal area in the current target plant, reducing the time for identifying the abnormality in the current target plant; thus, the abnormal positioning area is used as a constraint condition for identifying the abnormality of the current target plant, and the area of ​​the current target plant is analyzed layer by layer in combination with the machine learning algorithm, and then the abnormal impact index of the current target plant during operation is obtained. The abnormal impact index represents the parameter value of the degree of influence of the abnormal situation of the current target plant during operation, which is used to divide the abnormal situation in the current target plant; finally, the abnormal situation of the current target plant in operation is identified by the abnormal impact index. The above scheme can realize the rapid positioning of the abnormal area in the plant through multimodal information, thereby improving the safety of the plant during operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is an exemplary flow chart of a multimodal information fusion target recognition method based on machine learning according to some embodiments of the present application; Figure 2 is a schematic diagram of implementing average segmentation of abnormal images according to some embodiments of the present application; Figure 3 is an exemplary flow chart of determining an abnormal impact index according to some embodiments of the present application; Figure 4 is a schematic diagram of the structure of a multimodal information fusion target recognition system based on machine learning according to some embodiments of the present application; Figure 5 It is a structural diagram of a computer device for implementing a multimodal information fusion target recognition method based on machine learning as shown in some embodiments of the present application. DETAILED DESCRIPTION

[0018] In order to better understand the technical solution of the present application, the technical solution of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0019] refer to Figure 1 , which is an exemplary flow chart of a multimodal information fusion target recognition method based on machine learning according to some embodiments of the present application. The multimodal information fusion target recognition method 100 based on machine learning mainly includes the following steps: In step 101, multimodal data of a current target plant in operation is monitored, where the multimodal data includes image data and audio data of the target plant.

[0020] In specific implementation, the multimodal data of the current target factory in operation is monitored by multimodal acquisition equipment in the prior art (such as edge computing terminals, multimodal bracelets). In this embodiment, the multimodal data includes image data and audio data of the target factory. The image data represents a collection of images of various areas in the current target factory, and the audio data represents the audio generated in the current target factory. In other embodiments, other methods can also be used for monitoring, which is not limited here.

[0021] In step 102, audio mutation features and image abnormal features of the current target factory are extracted from the multimodal data based on the trained machine learning model.

[0022] In some embodiments, extracting the audio mutation features and image abnormal features of the current target plant from the multimodal data based on the trained machine learning model is implemented by the following steps: Get the trained machine learning model; Using the multimodal data as initialization parameters of the machine learning model; The audio mutation characteristics and image abnormal characteristics of the current target factory are determined based on the machine learning model.

[0023] In specific implementation, obtaining a trained machine learning model can be achieved in the following manner, namely: obtaining a trained machine learning model from a database of the current target project, wherein the machine learning model includes an audio feature extraction algorithm and an image feature extraction algorithm, and the features of the image data and the features of the audio data in the multimodal data can be extracted through this machine learning model. The machine learning model is trained based on the historical multimodal data in the current target plant. As a preferred embodiment, the machine learning model can be trained by combining the deep learning method in the prior art with the historical multimodal data. In other embodiments, other methods can be used to determine, which are not limited here.

[0024] It should be noted that the audio mutation feature in the present application represents the feature of the mutation of the audio data in the current target factory, and the audio mutation feature includes the set of all mutated audios, which is used to identify the sound conditions of the current target factory, so as to facilitate finding the abnormal conditions of the current target factory; the picture abnormality feature represents the feature of the abnormal conditions of the image data in the current target factory, and the picture abnormality feature includes the set of all abnormal images, and the image content in the picture abnormality feature is the image of the area where the abnormality occurs in the current target factory, which is used to identify the abnormalities of objects in the current target factory, so as to facilitate timely positioning of abnormal objects in the current target factory.

[0025] In step 103, feature decoupling is performed on the abnormal features of the image to obtain multiple abnormal segmentation maps for abnormal identification of the current target plant, and the abnormal boundary when the sound in the current target plant is abnormal is determined according to the audio mutation feature and the spatial layout of the current target plant. The convolutional neural network based on machine learning performs feature fusion on all the abnormal segmentation maps and the abnormal boundary to obtain the abnormal positioning area of ​​the current target plant.

[0026] In some embodiments, feature decoupling of the abnormal features of the image to obtain multiple abnormal segmentation maps for abnormal identification of the current target plant can be achieved by using the following steps: Determine the image segmentation threshold of the current target factory building; Selecting an abnormal picture with the abnormal picture feature as a selected abnormal picture, segmenting the selected abnormal picture according to the image segmentation threshold, and obtaining a plurality of segmented pictures of the selected abnormal picture; Determine the abnormal segmentation map corresponding to the abnormal image selected when performing abnormal identification on the current target plant based on all the segmentation maps; Continue to determine the abnormal segmentation maps corresponding to the remaining abnormal images when performing abnormality identification on the current target plant.

[0027] In specific implementation, determining the image segmentation threshold of the current target plant can be achieved in the following manner, namely: determining the image segmentation threshold of the current target plant by combining the image data in the multimodal data with the iterative threshold segmentation method, wherein the image segmentation threshold represents the parameter value of the segmentation degree of the image in the current target plant, and is used to segment the image in the current target plant; segmenting the selected abnormal picture by the image segmentation threshold, and obtaining multiple segmented pictures of the selected abnormal picture can be achieved in the following manner, namely: performing average segmentation on the selected abnormal picture by the image segmentation threshold, and using each picture obtained by the average segmentation as a segmented picture of the selected abnormal picture. For example, if the image segmentation threshold is 6, the selected abnormal picture is evenly segmented into 6 pictures, and the 6 pictures are all used as segmented pictures, such as Figure 2As described above, the diagram of the average segmentation of abnormal images shown in some embodiments may be determined in other ways in other embodiments, which is not limited here.

[0028] In specific implementation, the abnormal segmentation map corresponding to the selected abnormal image when the current target plant is subjected to abnormal identification according to all the segmented images can be implemented in the following manner, namely: selecting a segmented image as the selected segmented image, extracting the abnormal contour from the selected segmented image through the edge detection method in the prior art (such as Canny edge detector, Sobel operator, etc.), continuing to determine the remaining abnormal contours, merging all the abnormal contours through the morphological operation technology in the prior art (such as expansion operation, connected region analysis), and using the merged map as the abnormal segmentation map corresponding to the selected abnormal image when the current target plant is subjected to abnormal identification; in other embodiments, other methods can also be used for determination, which are not limited here.

[0029] It should be noted that the abnormal segmentation map in the present application is a segmentation map that reflects the abnormal identification of the current target plant, which is used to judge the abnormal situation in the current target plant, so as to facilitate timely adjustment of the current target plant.

[0030] In some embodiments, determining the abnormal boundary when the sound in the current target plant is abnormal according to the audio mutation characteristics and the spatial layout of the current target plant can be achieved by using the following steps: Get the spatial layout of the current target plant; Determine multiple audio mutation areas of the current target plant during operation according to the audio mutation characteristics and the spatial layout; The abnormal boundary when the sound in the current target factory is abnormal is determined through all audio mutation areas.

[0031] In specific implementation, obtaining the spatial layout of the current target factory building can be achieved in the following manner, namely: obtaining the spatial layout of the current target factory building from the drawings of the current target factory building, wherein the spatial layout represents the arrangement of objects, facilities, and functional areas in the current target factory building, and the spatial layout includes the specific positions of objects, facilities, and functional areas, which are used to determine the various positions of the current target factory building; determining multiple audio mutation areas of the current target factory building during operation according to the audio mutation characteristics and the spatial layout can be achieved in the following manner, namely: obtaining the distance and direction of the sound source of each audio in the audio mutation characteristics emitted in the current target factory building to the multimodal acquisition device from the multimodal database of the current target factory building , the position of each audio in the current target plant is located by combining the distance, direction and spatial layout of each audio through the audio positioning system in the prior art, and the adjacent positions of all positions are connected, and the connected areas are used as multiple audio mutation areas of the current target plant during operation, that is: since there are multiple abnormal areas in the current target plant, all the above positions are not all adjacent, and there are multiple areas after all connections, wherein the audio mutation area represents the area where there is a mutation sound in the current target plant, that is: there may be an abnormality in the occurrence of a mutation sound, which is used to judge the abnormal situation of the current target plant. In other embodiments, other methods can also be used for determination, which is not limited here.

[0032] In specific implementation, determining the abnormal boundary when the sound in the current target factory is abnormal through all audio mutation areas can be achieved in the following manner, namely: calculating the farthest distance and the closest distance from each audio mutation area to the multimodal acquisition device through the Euclidean distance in the prior art, connecting the points corresponding to the farthest distances to obtain the farthest envelope, connecting the points corresponding to the closest distances to obtain the closest envelope, and taking the area between the farthest distance and the closest distance as the abnormal boundary when the sound in the current target factory is abnormal. In other embodiments, other methods can also be used for determination, which are not limited here.

[0033] It should be noted that the abnormal boundary in the present application is a boundary area reflecting the existence of a sudden sound change in the current target plant, which is used to locate the abnormal situation in the current target plant, and then judge the abnormal situation so that the current target plant can take corresponding measures.

[0034] In some embodiments, since the convolutional neural network is based on machine learning and belongs to a branch of machine learning, the convolutional neural network based on machine learning in this application performs feature fusion of all abnormal segmentation maps and the abnormal boundaries, and then obtains the abnormal positioning area of ​​the current target plant, which can be achieved by the following steps: The convolutional neural network based on machine learning performs feature fusion on all abnormal segmentation maps and the abnormal boundaries to obtain abnormal fusion information of the current target plant; Determine multiple abnormal fusion areas of the current target plant according to the abnormal fusion information; The abnormal location area of ​​the current target plant is determined through all abnormal fusion areas.

[0035] In specific implementation, a convolutional neural network based on machine learning performs feature fusion of all abnormal segmentation maps and the abnormal boundaries to obtain abnormal fusion information of the current target plant, which can be implemented in the following manner, namely: a convolutional neural network based on machine learning performs feature fusion of all abnormal segmentation maps and abnormal boundaries, and uses the fused information as the abnormal fusion information of the current target plant, wherein the abnormal fusion information represents the fusion information of the abnormal area features in the current target plant, and the abnormal fusion information includes abnormal area information and boundary information; determining multiple abnormal fusion areas of the current target plant according to the abnormal fusion information can be implemented in the following manner, namely: extracting multiple abnormal fusion areas of the current target plant from the abnormal fusion information through the fusion feature map segmentation method in the prior art, wherein the abnormal fusion area represents the abnormal area of ​​the current target plant identified by combining audio and pictures, and is used to locate the abnormal area in the current target plant; in other embodiments, other methods can also be used for determination, which are not limited here.

[0036] In specific implementation, determining the abnormal positioning area of ​​the current target plant through all abnormal fusion areas can be achieved in the following manner, namely: obtaining historical abnormal area data in the current target plant from the database of the current target plant, wherein the historical abnormal area data is a collection of all historical abnormal areas, and the historical abnormal areas in the historical abnormal area data represent areas where abnormal conditions occur in the current target plant, dividing the maximum historical abnormal area in the historical abnormal area data by the minimum historical abnormal area, performing a logarithmic operation on the value obtained by the division with a base of 10, multiplying the value obtained by the logarithmic operation by the average area of ​​all historical abnormal areas in the historical abnormal area data, and using the multiplied value as the abnormal area threshold in the current target plant, wherein the abnormal The regional threshold represents the judgment value of the abnormal area that the abnormal situation has no impact on the current target plant, and is used to judge the abnormal area in the current target plant, select an abnormal fusion area as the selected abnormal fusion area, and compare the area of ​​the selected abnormal fusion area with the abnormal area threshold. If the area of ​​the selected abnormal fusion area is greater than or equal to the abnormal area threshold, the selected abnormal fusion area is used as the abnormal positioning sub-area of ​​the current target plant. If the area of ​​the selected abnormal fusion area is less than the abnormal area threshold, the selected abnormal fusion area is removed, and the remaining abnormal fusion areas are continued to be judged to obtain multiple abnormal positioning sub-areas, and the collection of all abnormal positioning sub-areas is used as the abnormal positioning area. In other embodiments, other methods can also be used for determination, which is not limited here.

[0037] It should be noted that the abnormal location area in the present application is an area that reflects the location of the abnormal area when an abnormality occurs in the current target factory building. It is used to analyze the abnormal situation in the current target factory building, facilitate the rapid location of the abnormal area in the current target factory building, and reduce the time for identifying abnormalities in the current target factory building.

[0038] In step 104, the abnormal location area is used as a constraint condition for abnormal identification of the current target plant, and the area of ​​the current target plant is analyzed layer by layer in combination with a machine learning algorithm to obtain the abnormal impact index of the current target plant during operation.

[0039] In some embodiments, reference Figure 3 As shown, this figure is a schematic diagram of the process of determining the abnormal impact index in some embodiments of the present application. In this embodiment, the abnormal positioning area is used as a constraint condition for abnormal identification of the current target plant, and the area of ​​the current target plant is analyzed layer by layer in combination with a machine learning algorithm, and then the abnormal impact index of the current target plant during operation is obtained. The following steps can be used to achieve: First, in step 1041, the abnormal location area is used as a constraint condition for abnormality identification of the current target plant; Secondly, in step 1042, a plurality of layer-by-layer outliers in the current target plant are determined according to the constraint conditions, the spatial layout of the current target plant, and the machine learning algorithm; Finally, in step 1043, the abnormal impact index of the current target plant during operation is determined through all the abnormal values ​​layer by layer.

[0040] In specific implementation, the following method can be used to determine multiple layer-by-layer outliers in the current target plant according to the constraint conditions, the spatial layout of the current target plant and the machine learning algorithm, namely: by using a machine learning algorithm (such as a regression algorithm, a neural network, etc.) to establish a layer-by-layer outlier model, using the constraint conditions as constraint parameters of the layer-by-layer outlier model, using the spatial layout of the current target plant as the initialization parameters of the layer-by-layer outlier model, and outputting multiple layer-by-layer outliers in the current target plant through the layer-by-layer outlier model, wherein the layer-by-layer outliers represent parameter values ​​of the abnormal conditions at the corresponding level of the current target plant, and are used to analyze the abnormal conditions of each level in the current target plant; the layer-by-layer outlier model, for example: multiple layer-by-layer outliers = constraint conditions * A + spatial layout of the current target plant * B, wherein A and B are weight coefficients, A and B can be determined based on a large number of layer-by-layer outliers, and can also be determined in other ways in other embodiments, which are not limited here.

[0041] In specific implementation, determining the abnormal impact index of the current target plant during operation through all the layer-by-layer abnormal values ​​can be achieved in the following manner, namely: dividing the largest layer-by-layer abnormal value by the smallest layer-by-layer abnormal value, performing a logarithmic operation with a base of 2 on the value obtained by the division, multiplying the value obtained by the logarithmic operation by the average value of all the layer-by-layer abnormal values, and using the multiplied value as the abnormal impact index of the current target plant during operation. In other embodiments, other methods can also be used for implementation, which are not limited here.

[0042] It should be noted that the abnormal impact index in this application represents the parameter value of the impact degree of abnormal conditions during the operation of the current target plant, and is used to divide the abnormal conditions in the current target plant to facilitate the target plant to make existing response strategies.

[0043] In step 105, the abnormal condition of the current target plant in operation is identified by using the abnormal impact index.

[0044] In some embodiments, identifying the abnormal condition of the current target plant in operation by using the abnormal impact index can be achieved by using the following steps: Determine the abnormal impact threshold range of the abnormal conditions in the operation of the current target plant; If the abnormal impact index is less than the lower limit of the abnormal impact threshold range, the abnormal condition of the current target plant in operation is a safe state; If the abnormal impact index is within the abnormal impact threshold range, the abnormal condition of the current target plant in operation is in a pending state; If the abnormal impact index is greater than the upper limit value of the abnormal impact threshold range, the abnormal condition of the current target plant in operation is an emergency state.

[0045] In specific implementation, when the abnormal condition of the current target plant during operation is in a safe state, the current target plant will not be processed; when the abnormal condition of the current target plant during operation is in a pending state, a response strategy needs to be made for the current target plant in time; when the abnormal condition of the current target plant during operation is in an emergency state, the operation of the current target plant is stopped and a response strategy is made. Other methods can also be used to implement this in other embodiments, which are not limited here.

[0046] It should be noted that in the present application, the abnormal impact threshold range can be set accordingly according to the specific needs of the current target plant. For example, if the current target plant processes products with a higher safety factor, the abnormal impact threshold range can be set within a high range. If the current target plant processes dangerous products, the abnormal impact threshold range can be set within a low range. In other embodiments, for example, when the current target plant processes chemical products, the abnormal impact threshold range can be set within a high range, thereby improving the safety of the current target plant during operation.

[0047] In addition, in another aspect of the present application, in some embodiments, the present application provides a multimodal information fusion target recognition system based on machine learning, referring to Figure 4 , which is a schematic diagram of the structure of a multimodal information fusion target recognition system based on machine learning according to some embodiments of the present application. The multimodal information fusion target recognition system based on machine learning 400 includes: a monitoring module 401, a processing module 402 and an execution module 403, which are respectively described as follows: Monitoring module 401, in this application, the monitoring module 401 is mainly used to monitor the multimodal data of the current target plant in operation, and the multimodal data includes image data and audio data of the target plant; Processing module 402, in the present application, is used to extract audio mutation features and image abnormal features of the current target plant from the multimodal data based on the trained machine learning model; It should be noted that the processing module 402 in the present application is also used to perform feature decoupling on the abnormal features of the image, obtain multiple abnormal segmentation maps when the current target plant is abnormally identified, determine the abnormal boundary when the sound in the current target plant is abnormal according to the audio mutation feature and the spatial layout of the current target plant, and perform feature fusion of all the abnormal segmentation maps and the abnormal boundary based on the convolutional neural network of machine learning, so as to obtain the abnormal positioning area of ​​the current target plant; In addition, it should be noted that the processing module 402 in the present application is also used to use the abnormal location area as a constraint condition for abnormal identification of the current target plant, and combine the machine learning algorithm to analyze the area of ​​the current target plant layer by layer, so as to obtain the abnormal impact index of the current target plant during operation; Execution module 403, in the present application, the execution module 403 is mainly used to identify the abnormal condition of the current target plant in operation through the abnormal impact index.

[0048] In addition, the present application also provides a computer device, which includes a memory and a processor, the memory stores code, and the processor is configured to obtain the code and execute the above-mentioned multimodal information fusion target recognition method based on machine learning.

[0049] In some embodiments, reference Figure 5 , which is a schematic diagram of the structure of a computer device for implementing a multimodal information fusion target recognition method based on machine learning according to some embodiments of the present application. The multimodal information fusion target recognition method based on machine learning in the above embodiment can be Figure 5 The computer device 500 shown in the figure is implemented, and the computer device 500 includes at least one processor 501, a communication bus 502, a memory 503 and at least one communication interface 504.

[0050] The processor 501 may be a general-purpose central processing unit (CPU) or an application specific integrated circuit (ASIC).

[0051] The communication bus 502 may be used to transmit information between the above-mentioned components.

[0052] The memory 503 may be a read only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read only memory (EEPROM), a compact disc read only memory (CD ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 503 may exist independently and be connected to the processor 501 via the communication bus 502. The memory 503 may also be integrated with the processor 501.

[0053] The memory 503 is used to store the program code for executing the solution of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the program code stored in the memory 503. The program code may include one or more software modules. The method used in the above embodiment can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.

[0054] The communication interface 504 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0055] In a specific implementation, as an embodiment, a computer device may include multiple processors, each of which may be a single-core (single CPU) processor or a multi-core (multi CPU) processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0056] The above-mentioned computer device may be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device may be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of computer device.

[0057] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multimodal information fusion target recognition method based on machine learning.

[0058] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0059] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A multimodal information fusion target recognition method based on machine learning, characterized in that: The steps include: Monitoring multimodal data of the current target plant in operation, wherein the multimodal data includes image data and audio data of the target plant; Extracting audio mutation features and image abnormal features of the current target factory from the multimodal data based on the trained machine learning model; Decouple the abnormal features of the image to obtain multiple abnormal segmentation maps for abnormal identification of the current target plant, determine the abnormal boundary when the sound in the current target plant is abnormal according to the audio mutation feature and the spatial layout of the current target plant, and perform feature fusion of all the abnormal segmentation maps and the abnormal boundary based on the convolutional neural network of machine learning to obtain the abnormal positioning area of ​​the current target plant; The abnormal location area is used as a constraint condition for abnormal identification of the current target plant, and the area of ​​the current target plant is analyzed layer by layer in combination with a machine learning algorithm, so as to obtain the abnormal impact index of the current target plant during operation; The abnormal condition of the current target plant in operation is identified through the abnormal impact index.

2. The method according to claim 1, characterized in that The audio mutation features and image abnormal features of the current target factory building are extracted from the multimodal data based on the trained machine learning model, specifically including: Get the trained machine learning model; Using the multimodal data as initialization parameters of the machine learning model; The audio mutation characteristics and image abnormal characteristics of the current target factory are determined based on the machine learning model.

3. The method according to claim 1, characterized in that Decoupling the abnormal features of the image to obtain multiple abnormal segmentation maps for abnormal identification of the current target plant specifically includes: Determine the image segmentation threshold of the current target factory building; Selecting an abnormal picture with the abnormal picture feature as a selected abnormal picture, segmenting the selected abnormal picture according to the image segmentation threshold, and obtaining a plurality of segmented pictures of the selected abnormal picture; Determine the abnormal segmentation map corresponding to the abnormal image selected when performing abnormal identification on the current target plant based on all the segmentation maps; Continue to determine the abnormal segmentation maps corresponding to the remaining abnormal images when performing abnormality identification on the current target plant.

4. The method according to claim 1, characterized in that Determining the abnormal boundary when the sound in the current target factory building is abnormal according to the audio mutation characteristics and the spatial layout of the current target factory building specifically includes: Get the spatial layout of the current target plant; Determine multiple audio mutation areas of the current target plant during operation according to the audio mutation characteristics and the spatial layout; The abnormal boundary when the sound in the current target factory is abnormal is determined through all audio mutation areas.

5. The method according to claim 1, characterized in that The convolutional neural network based on machine learning integrates all the abnormal segmentation maps and the abnormal boundaries to obtain the abnormal positioning area of ​​the current target plant, which specifically includes: The convolutional neural network based on machine learning performs feature fusion on all abnormal segmentation maps and the abnormal boundaries to obtain abnormal fusion information of the current target plant; Determine multiple abnormal fusion areas of the current target plant according to the abnormal fusion information; The abnormal location area of ​​the current target plant is determined through all abnormal fusion areas.

6. The method according to claim 1, characterized in that The abnormal location area is used as a constraint condition for abnormal identification of the current target plant, and the area of ​​the current target plant is analyzed layer by layer in combination with a machine learning algorithm, so as to obtain the abnormal impact index of the current target plant during operation, which specifically includes: Using the abnormal location area as a constraint condition when performing abnormality identification on the current target plant; Determine a plurality of floor-by-floor outliers in the current target plant building according to the constraint conditions, the spatial layout of the current target plant building, and a machine learning algorithm; The abnormal impact index of the current target plant during operation is determined through all the abnormal values ​​layer by layer.

7. The method according to claim 1, characterized in that Identifying the abnormal condition of the current target plant in operation through the abnormal impact index specifically includes: Determine the abnormal impact threshold range of the abnormal conditions in the operation of the current target plant; If the abnormal impact index is less than the lower limit of the abnormal impact threshold range, the abnormal condition of the current target plant in operation is a safe state; If the abnormal impact index is within the abnormal impact threshold range, the abnormal condition of the current target plant in operation is in a pending state; If the abnormal impact index is greater than the upper limit of the abnormal impact threshold range, the abnormal condition of the current target plant in operation is an emergency state.

8. A multimodal information fusion target recognition system based on machine learning, characterized in that: include: A monitoring module, used to monitor the multimodal data of the current target plant in operation, wherein the multimodal data includes image data and audio data of the target plant; A processing module, used to extract audio mutation features and image abnormal features of the current target factory building from the multimodal data based on the trained machine learning model; The processing module is further used to perform feature decoupling on the abnormal features of the image to obtain multiple abnormal segmentation maps when abnormal identification is performed on the current target plant, determine the abnormal boundary when the sound in the current target plant is abnormal according to the audio mutation feature and the spatial layout of the current target plant, and perform feature fusion on all the abnormal segmentation maps and the abnormal boundary based on the convolutional neural network of machine learning, so as to obtain the abnormal positioning area of ​​the current target plant; The processing module is further used to use the abnormal location area as a constraint condition for abnormal identification of the current target plant, and combine the machine learning algorithm to analyze the area of ​​the current target plant layer by layer, so as to obtain the abnormal impact index of the current target plant during operation; The execution module is used to identify the abnormal condition of the current target plant in operation through the abnormal impact index.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores codes, and the processor is configured to obtain the codes and execute the multimodal information fusion target recognition method based on machine learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal information fusion target recognition method based on machine learning is implemented.

Images