Machine room monitoring method and device, equipment, storage medium and program product

Abstract

The invention provides a machine room monitoring method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining to-be-processed multi-modal machine room data which comprises sensor data, image data and audio data; performing weighted fusion on the sensor data to obtain fused sensor feature data; inputting the image data into a pre-trained convolutional neural network model to obtain image data after feature extraction; inputting the audio data into a pre-trained long-short term memory network model to obtain audio data after feature extraction; analyzing semantic consistency of the sensor feature data, the image data after feature extraction and the audio data after feature extraction through a pre-trained contrast learning model to obtain fusion data; and inputting the fusion data into the pre-trained machine room state prediction model to obtain a machine room state monitoring result, thereby improving the accuracy of machine room monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a machine room monitoring method and device, equipment, storage medium and program product. BACKGROUND

[0002] As the core hub of data storage and processing, information machine room has become an indispensable infrastructure for enterprises. With the increasing scale and complexity of machine rooms, traditional manual management methods have been difficult to meet the needs of complex environments and efficient operation and maintenance.

[0003] In the prior art, machine room monitoring usually relies on sensors to collect environmental data of the machine room, and the machine room is monitored in real time according to the directly collected data.

[0004] However, in the prior art, there are environmental differences between different types of machine rooms, and the directly collected sensor data has the problem of inaccurate machine room monitoring. SUMMARY

[0005] The embodiments of the present application provide a machine room monitoring method, device, equipment, storage medium and program product, to solve the problem of inaccurate machine room monitoring in the prior art.

[0006] In a first aspect, the embodiments of the present application provide a machine room monitoring method, comprising:

[0007] Obtaining multi-modal machine room data to be processed, wherein the multi-modal machine room data includes sensor data, image data and audio data;

[0008] Weighted fusion is performed on the sensor data to obtain fused sensor feature data;

[0009] The image data is input into a pre-trained convolutional neural network model to obtain feature-extracted image data;

[0010] The audio data is input into a pre-trained long short-term memory network model to obtain feature-extracted audio data;

[0011] The semantic consistency of the sensor feature data, the feature-extracted image data and the feature-extracted audio data is analyzed by a pre-trained contrastive learning model to obtain fusion data;

[0012] The fusion data is input into a pre-trained machine room state prediction model to obtain a machine room state monitoring result.

[0013] In a possible implementation, the weighted fusion of the sensor data to obtain the fused sensor feature data comprises: performing encoding processing on the sensor data to obtain mapped sensor data; calculating a weight score of the sensor data according to an attention mechanism; and performing weighted fusion on the mapped sensor data according to the weight score of the sensor data to obtain the fused sensor feature data.

[0014] In a possible implementation, the inputting of the image data into the pre-trained convolutional neural network model to obtain the feature-extracted image data comprises: performing convolutional pooling processing on the image data to obtain low-level features; performing stack processing on the low-level features to generate image semantic information; and performing global fusion on the image semantic information to obtain the feature-extracted image data.

[0015] In a possible implementation, the inputting of the audio data into the pre-trained long short-term memory network model to obtain the feature-extracted audio data comprises: performing frame processing on the audio data to obtain an audio data feature sequence; filtering the audio data feature sequence through a forget gate of the long short-term memory network model to obtain a filtered audio data feature sequence; and outputting the filtered audio data feature sequence through an output gate of the long short-term memory network model to obtain the feature-extracted audio data.

[0016] In a possible implementation, the analyzing of semantic consistency of the sensor feature data, the feature-extracted image data, and the feature-extracted audio data by the pre-trained contrastive learning model to obtain the fused data comprises: performing mapping processing on the sensor feature data, the feature-extracted image data, and the feature-extracted audio data to obtain mapped semantic data; establishing a sensor-image-audio loss function; and analyzing semantic consistency of the semantic data according to the mapped semantic data and the sensor-image-audio loss function to obtain the fused data.

[0017] In a possible implementation, the inputting of the fused data into the pre-trained computer room state prediction model to obtain the computer room state monitoring result comprises: calculating a computer room state prediction probability of the fused data through a pre-trained classifier; calculating a computer room state prediction loss function according to the computer room state prediction probability; and predicting a computer room state according to the computer room state prediction loss function to obtain the computer room state monitoring result.

[0018] In a second aspect, an embodiment of the present application provides a computer room monitoring device, comprising:

[0019] An acquisition module is configured to acquire multi-modal machine room data to be processed, wherein the multi-modal machine room data comprises sensor data, image data and audio data;

[0020] A fusion module is configured to perform weighted fusion on the sensor data to obtain fused sensor feature data;

[0021] A first output module is configured to input the image data into a pre-trained convolutional neural network model to obtain feature-extracted image data;

[0022] A second output module is configured to input the audio data into a pre-trained long short-term memory network model to obtain feature-extracted audio data;

[0023] A contrast learning module is configured to analyze semantic consistency of the sensor feature data, the feature-extracted image data and the feature-extracted audio data by using a pre-trained contrast learning model to obtain fusion data;

[0024] A third output module is configured to input the fusion data into a pre-trained machine room state prediction model to obtain a machine room state monitoring result.

[0025] In a third aspect, an embodiment of the present application provides a machine room monitoring device, comprising a memory and a processor.

[0026] The memory stores computer execution instructions.

[0027] The processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0029] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0030] The machine room monitoring method, device, equipment, storage medium and program product provided by the embodiment of the application obtain multi-modal machine room data including sensors, images and audio, fuse the sensor data by a weighted fusion method, extract the image data by a convolutional neural network, extract the audio data by a long short-term memory network, analyze the semantic consistency of the sensor data, the image data and the audio data by contrast learning, learn the difference and similarity between different modal data, predict the fused data by a machine room state prediction model, and obtain a monitoring result. Compared with the prior art, the method is suitable for different types of machine rooms, and the accuracy of machine room monitoring is improved. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0032] Figure 1 The system structure schematic diagram of the computer equipment provided by the embodiment of the application is shown in the following figure.

[0033] Figure 2 The flowchart of the machine room monitoring method provided by the embodiment of the application is shown in the following figure.

[0034] Figure 3 The structure schematic diagram of the machine room monitoring device provided by the embodiment of the application is shown in the following figure.

[0035] Figure 4 The structure schematic diagram of the machine room monitoring device provided by the embodiment of the application is shown in the following figure.

[0036] The above-mentioned figures have shown the specific embodiments of the application, which will be described in more detail hereinafter. These figures and the description are not intended to limit the scope of the concept of the application by any means, but to illustrate the concept of the application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0037] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same or similar components. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the application. Instead, they are merely examples of devices and methods consistent with some aspects of the application, as detailed in the appended claims.

[0038] Information room as the core hub of data storage and processing has become an indispensable infrastructure for enterprises. With the increasing size and complexity of the computer room, the traditional manual management method has been difficult to meet the needs of complex environment and efficient operation and maintenance. In the prior art, the computer room monitoring usually relies on sensors to collect environmental data of the computer room, and the computer room is monitored in real time according to the directly collected data. However, in the prior art, there are environmental differences between different types of computer rooms, and the directly collected sensor data has the problem of inaccurate computer room monitoring.

[0039] To solve the above technical problems, the embodiment of the application proposes the following technical concept: the inventor considers obtaining multi-modal computer room data including sensors, images and audio, considers using a weighted fusion method to fuse sensor data, considers using a convolutional neural network to extract image data, considers using a long short-term memory network to extract audio data, uses contrastive learning to analyze the semantic consistency of sensor data, image data and audio data, learns the difference and similarity between different modal data, and uses a computer room state prediction model to predict the fused data to obtain a monitoring result. The following detailed embodiments are described in detail.

[0040] Figure 1 The system structure schematic diagram of the computer equipment provided by the embodiment of the application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the computer equipment includes a receiving device 101, a processing device 102 and a display device 103.

[0041] It can be understood that the structure shown in the embodiment of the application does not constitute a specific limitation on the computer room monitoring method. In other feasible embodiments of the application, the above architecture can include more or fewer components than the diagram, or combine certain components, or split certain components, or different component arrangement, which can be determined according to actual application scenarios, and is not limited herein. Figure 1 The components shown in the figure can be realized by hardware, software or a combination of software and hardware.

[0042] In the specific implementation process, the receiving device 101 can be an input / output interface or a communication interface, and can obtain multi-modal computer room data to be processed.

[0043] The processing device 102 can generate a computer room state monitoring result.

[0044] The display device 103 can be used to display the computer room state monitoring result and the like.

[0045] The display device can also be a touch display screen, which is used to receive user instructions while displaying the above-mentioned content, so as to realize the operation interaction with the user.

[0046] It should be understood that the above-mentioned processing device can be implemented by a processor reading instructions in a memory and executing the instructions, or it can be implemented by a chip circuit.

[0047] In addition, the network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0048] Figure 2 The flow chart of the computer room monitoring method provided in this application is as follows: Figure 2 As shown, the method includes:

[0049] S201: Acquire multimodal computer room data to be processed, where the multimodal computer room data includes sensor data, image data, and audio data.

[0050] In this embodiment, the sensor data includes but is not limited to temperature and humidity data, power distribution status data, air conditioning operation parameters, current and voltage, and energy consumption monitoring parameters.

[0051] Among them, temperature and humidity data, power distribution status data and air conditioning operation parameters are directly collected by sensors, and current, voltage and energy consumption monitoring parameters are obtained by reading the device through monitoring software.

[0052] S202: Perform weighted fusion on the sensor data to obtain fused sensor feature data.

[0053] Specifically, the sensor data is encoded and mapped to a feature space of uniform dimension. The weight score of the sensor data is calculated according to the attention mechanism, and weighted fusion is performed according to the weight score to obtain the fused sensor data.

[0054] S203: Input the image data into the pre-trained convolutional neural network model to obtain image data after feature extraction.

[0055] Specifically, a convolutional neural network is used to convolve and pool the image data to obtain low-level features, which are then stacked to obtain high-level semantic information. Global fusion is performed based on the high-level semantic information to obtain image data after feature extraction.

[0056] S204: Input the audio data into the pre-trained long short-term memory network model to obtain audio data after feature extraction.

[0057] Specifically, a long short-term memory network model is used to capture audio timing features and long-term dependencies, and the hidden state of the last time step is used as the audio data after feature extraction.

[0058] S205: Analyzing semantic consistency of the sensor feature data, the image data after feature extraction and the audio data after feature extraction by the pre-trained contrast learning model to obtain fused data.

[0059] Specifically, the sensor feature data, the image data after feature extraction and the audio data after feature extraction are mapped to the same space, three types of loss functions of sensor-image, sensor-audio and image-audio are established, and the semantic consistency is analyzed according to the three types of loss functions of sensor-image, sensor-audio and image-audio to obtain the fused data.

[0060] S206: Inputting the fused data into the pre-trained machine room state prediction model to obtain a machine room state monitoring result.

[0061] Specifically, the classifier is trained to classify and predict the fused data, binary cross entropy is used as the loss function of the classifier, the three types of loss functions of sensor-image, sensor-audio and image-audio are combined to obtain a loss function for machine room state prediction, and the machine room state is predicted according to the loss function for machine room state prediction to obtain the machine room state monitoring result.

[0062] From the above embodiment, it can be seen that by acquiring multi-modal machine room data including sensors, images and audio, sensor data is fused by a weighted fusion method, image data is extracted by a convolutional neural network, audio data is extracted by a long short-term memory network, semantic consistency of sensor data, image data and audio data is analyzed by contrast learning, differences and similarities between different modal data are learned, fused data is predicted by a machine room state prediction model to obtain a monitoring result. Compared with the prior art, different types of machine rooms are adapted, and the accuracy of machine room monitoring is improved.

[0063] In an embodiment of the present application, step S202 comprises:

[0064] S2021: Encoding the sensor data to obtain mapped sensor data.

[0065] Specifically, a linear transformation layer is used to encode the sensor data, and the sensor data is mapped to a unified dimensional feature space. Let the data matrix corresponding to the i-th sensor data be The fixed-dimensional feature representation is obtained by linear transformation, specifically as follows:

[0066]

[0067] In the formula, represents the feature representation of the i-th sensor data; represents the data matrix corresponding to the i-th sensor data; a learnable weight matrix representing the ith sensor data.

[0068] S2022: Calculate the weight score of the sensor data according to the attention mechanism.

[0069] Specifically, the attention mechanism calculates the weight score of each sensor data as:

[0070]

[0071] In the formula, represents a learnable weight matrix; represents a learnable bias vector; represents the transpose operation of the feature representation of the ith sensor data; represents the attention weight score of the ith sensor data.

[0072] Specifically, the weight score is calculated by softmax normalization to obtain , so that the sum of the weights of all sensor data is 1, which is represented as:

[0073]

[0074] In the formula, represents the normalized weight score of the ith sensor data; represents the attention weight score of the ith sensor data; j represents the type of sensor data.

[0075] S2023: Weighted fusion of the mapped sensor data according to the weight score of the sensor data, to obtain the fused sensor feature data.

[0076] Specifically, the normalized weight score is used to perform weighted summation on the feature representation of the sensor data, to obtain the global sensor feature representation:

[0077]

[0078] In the formula, represents the fused sensor feature data; represents the normalized weight score of the ith sensor data; represents the feature representation of the ith sensor data.

[0079] From the above embodiments, it can be seen that by using linear transformation to encode the sensor data, the sensor data is mapped to a unified space dimension, which is convenient for fusion modeling. The attention mechanism is introduced to calculate the importance of different sensor data information, enhance the attention of key features in the fusion representation, and improve the accuracy of anomaly detection.

[0080] In one embodiment of the present application, step S203 comprises:

[0081] S2031: performing convolutional pooling processing on the image data to obtain low-level features.

[0082] In this embodiment, a convolutional neural network is used to extract image data in the machine room environment.

[0083] In this embodiment, the low-level features include but are not limited to edge features, texture features, and color features.

[0084] S2032: stacking the low-level features to generate image semantic information.

[0085] In this embodiment, the image semantic information includes but is not limited to object contour, object surface material, and object color.

[0086] S2033: globally fusing the image semantic information to obtain the image data after feature extraction.

[0087] Specifically, the image semantic information is globally fused to obtain the image data after feature extraction, which is represented as:

[0088]

[0089] In the formula, represents the image data after feature extraction; represents an activation function; represents a weight matrix of the fully connected layer; represents an image feature vector output by the previous layer; represents a bias term of the fully connected layer.

[0090] As can be seen from the above embodiments, by learning and extracting key features in image data through a convolutional neural network, performing convolutional pooling processing on the image data to obtain low-level features, and integrating global information through a fully connected layer to extract image data, the accuracy of extracting image data is improved.

[0091] In one embodiment of the present application, step S204 comprises:

[0092] S2041: performing frame processing on the audio data to obtain an audio data feature sequence.

[0093] In this embodiment, a long short-term memory network model is used to capture audio timing features to obtain an audio data feature sequence.

[0094] S2042: filtering the audio data feature sequence through a forget gate of the long short-term memory network model to obtain a filtered audio data feature sequence.

[0095] In the embodiment, the filtered audio data feature sequence is represented as .

[0096] S2043: Output the fused filtered audio data feature sequence through the output gate of the long short-term memory network model to obtain the audio data after feature extraction.

[0097] Specifically, the long short-term memory network model updates the hidden state and the cell state , captures long-term dependencies, and selects the hidden state of the last time step as the audio data after feature extraction.

[0098] Wherein, the long-term dependency is represented as:

[0099]

[0100] In the formula, represents the long short-term memory network; represents the long-term dependency of the tth time step, represents the audio data of the tth time step; represents the hidden state of the t-1th time step; represents the cell state of the t-1th time step; and t represents the time step.

[0101] In the embodiment, the audio data after feature extraction is represented as .

[0102] From the above embodiment, it can be seen that the long short-term memory network model is used for frame processing of audio data to obtain an audio data feature sequence, the feature sequence is filtered through a forget gate, the feature sequence is output fused through an output gate to obtain audio data after feature extraction, and the long short-term memory network is used to capture audio time sequence features, thereby enhancing the expression ability of audio information.

[0103] In an embodiment of the present application, step S205 includes:

[0104] S2051: Map and process the sensor feature data, the image data after feature extraction, and the audio data after feature extraction to obtain mapped semantic data.

[0105] In the embodiment, the sensor feature data, the image data after feature extraction, and the audio data after feature extraction are mapped to the same semantic space to obtain the mapped semantic data.

[0106] S2052: Establish a sensor-image-audio loss function.

[0107] In the embodiment, the sensor-image-audio loss function comprises a sensor-image loss function, a sensor-audio loss function and an image-audio loss function.

[0108] In the embodiment, the sensor-image loss function is represented as:

[0109]

[0110] In the formula, represents the sensor-image loss function; represents the total number of samples; represents the temperature coefficient; represents the feature data of the i-th sensor sample; represents the image data of the i-th feature.

[0111] In the embodiment, the sensor-audio loss function is represented as:

[0112]

[0113] In the formula, represents the sensor-audio loss function; represents the total number of samples; represents the temperature coefficient; represents the feature data of the i-th sensor sample; represents the audio data of the i-th feature.

[0114] In the embodiment, the image-audio loss function is represented as:

[0115]

[0116] In the formula, represents the image-audio loss function; represents the total number of samples; represents the temperature coefficient; represents the image data of the i-th feature; represents the audio data of the i-th feature.

[0117] S2053: According to the mapped semantic data and the sensor-image-audio loss function, the semantic consistency of the semantic data is analyzed to obtain fusion data.

[0118] In the embodiment, the fusion data is represented as:

[0119]

[0120] In the formula, represents the fusion data; represents the sensor feature data; represent image data after feature extraction; represent audio data after feature extraction.

[0121] From the above embodiments, by mapping the sensor data, image data and audio data to a unified space through the pre-trained contrast learning model, establishing the sensor-image-audio loss function, minimizing the modal variance of the sensor, image and audio data through the loss function, aligning the multi-modal data through contrast learning, learning the difference and similarity between different modal data, and improving the unified representation ability of multi-modal data.

[0122] In an embodiment of the present application, step S206 includes:

[0123] S2061: Calculate the machine room state prediction probability of the fusion data by the pre-trained classifier.

[0124] In this embodiment, the machine room state prediction probability is represented as .

[0125] Specifically, the classifier is trained to classify and predict the fusion data to determine whether the current machine room state is abnormal.

[0126] S2062: Calculate the loss function of the machine room state prediction according to the machine room state prediction probability.

[0127] In this embodiment, the loss function of the machine room state prediction is a joint loss function.

[0128] In this embodiment, binary cross entropy is used as the loss function of the classifier:

[0129]

[0130] In the formula, represents the binary cross entropy loss function; represents the true label of the i-th sample; represents the machine room state prediction probability; represents the total number of samples.

[0131] In this embodiment, the joint loss function is determined according to the sensor-image-audio loss function and the binary cross entropy loss function, and the model is trained according to the joint loss function.

[0132] In this embodiment, the joint loss function is represented as:

[0133]

[0134] In the formula, represents the joint loss function; represents the binary cross entropy loss function; a loss function representing sensor-image; a loss function representing sensor-audio; and a tunable weight hyper-parameter.

[0135] In the embodiment, the tunable weight hyper-parameter is used to balance the contribution of the classification task and the contrastive learning task in joint training.

[0136] S2063: predicting the computer room state according to the loss function of the computer room state prediction, to obtain a computer room state monitoring result.

[0137] In the embodiment, the computer room in a normal state is represented as: label=0; and the computer room in an abnormal state is represented as: label=1.

[0138] From the above embodiment, it can be seen that the computer room state prediction probability is calculated by the pre-trained classifier, the loss function is calculated according to the computer room state prediction probability, the prediction error of the model on the computer room state classification task is measured according to the loss function, the data is jointly fused, the consistency of the multi-modal feature representation is improved, and the robustness of the complex computer room environment detection is improved.

[0139] Figure 3 The structure schematic diagram of the computer room monitoring device provided in the present application is shown in FIG. 1. Figure 3 As shown in FIG. 1, the computer room monitoring device 30 provided in the embodiment comprises an acquisition module 301, a fusion module 302, a first output module 303, a second output module 304, a contrastive learning module 305, and a third output module 306.

[0140] The acquisition module 301 is configured to acquire multi-modal computer room data to be processed, wherein the multi-modal computer room data comprises sensor data, image data, and audio data.

[0141] The fusion module 302 is configured to perform weighted fusion on the sensor data to obtain fused sensor feature data.

[0142] The first output module 303 is configured to input the image data into a pre-trained convolutional neural network model to obtain feature-extracted image data.

[0143] The second output module 304 is configured to input the audio data into a pre-trained long short-term memory network model to obtain feature-extracted audio data.

[0144] The contrastive learning module 305 is configured to analyze the semantic consistency of the sensor feature data, the feature-extracted image data, and the feature-extracted audio data by a pre-trained contrastive learning model to obtain fusion data.

[0145] The third output module 306 is configured to input the fused data into a pre-trained machine room state prediction model to obtain a machine room state monitoring result.

[0146] In a possible implementation, the fusion module 302 includes:

[0147] The encoding unit is configured to perform encoding processing on the sensor data to obtain mapped sensor data.

[0148] The first calculation unit is configured to calculate a weight score of the sensor data according to an attention mechanism.

[0149] The weighted fusion unit is configured to perform weighted fusion on the mapped sensor data according to the weight score of the sensor data to obtain fused sensor feature data.

[0150] In a possible implementation, the first output module 303 includes:

[0151] The convolution unit is configured to perform convolution and pooling processing on the image data to obtain low-level features.

[0152] The generation unit is configured to perform stacking processing on the low-level features to generate image semantic information.

[0153] The global fusion unit is configured to perform global fusion on the image semantic information to obtain feature-extracted image data.

[0154] In a possible implementation, the second output module 304 includes:

[0155] The frame division unit is configured to perform frame division processing on the audio data to obtain an audio data feature sequence.

[0156] The screening unit is configured to screen the audio data feature sequence through a forget gate of a long short-term memory network model to obtain screened audio data feature sequence.

[0157] The fusion unit is configured to output the screened audio data feature sequence through an output gate of the long short-term memory network model to obtain feature-extracted audio data.

[0158] In a possible implementation, the contrast learning module 305 includes:

[0159] The mapping unit is configured to perform mapping processing on the sensor feature data, the feature-extracted image data, and the feature-extracted audio data to obtain mapped semantic data.

[0160] The establishment unit is configured to establish a sensor-image-audio loss function.

[0161] The analysis unit is configured to analyze semantic consistency of the semantic data according to a loss function of the mapped semantic data and the sensor-image-audio, and obtain fusion data.

[0162] In a possible implementation, the third output module 306 includes:

[0163] The second calculation unit is configured to calculate a machine room state prediction probability of the fusion data by using the pre-trained classifier.

[0164] The third calculation unit is configured to calculate a loss function of the machine room state prediction according to the machine room state prediction probability.

[0165] The prediction unit is configured to predict the machine room state according to the loss function of the machine room state prediction, and obtain a machine room state monitoring result.

[0166] The machine room monitoring device provided in this embodiment can execute the method provided in the method embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0167] Figure 4 A structural schematic diagram of the machine room monitoring device provided in this embodiment is shown in FIG. 4. Figure 4 As shown in FIG. 4, the machine room monitoring device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the machine room monitoring device 40 further includes a communication component 403. The processor 401, the memory 402 and the communication component 403 are connected through a bus.

[0168] In the specific implementation process, the at least one processor 401 executes the computer execution instructions stored in the memory 402, so that the at least one processor 401 executes the machine room monitoring method described above.

[0169] The specific implementation process of the processor 401 can refer to the method embodiments described above, and has similar implementation principles and technical effects, which will not be described here again.

[0170] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0171] The memory can include a Random Access Memory (RAM) and can also include a Non-volatile Memory (NVM), such as at least one disk memory.

[0172] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0173] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the machine room monitoring method described above.

[0174] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the machine room monitoring method described above is implemented.

[0175] The readable storage medium described above can be implemented by any type of volatile or non-volatile storage device or their combination, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0176] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0177] The division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0178] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0179] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0180] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0181] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various program code storage media.

[0182] It should be understood that many of the materials and devices exemplified in this disclosure are articles of manufacture (i.e., articles of manufacture) according to this disclosure. The articles of manufacture can be manufactured as such or can be manufactured by combining the materials and devices exemplified in this disclosure. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It should be understood that, in some embodiments, equivalents to the specific electrode structures and / or methods described herein can be employed without departing from the scope of the application. Accordingly, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized into," and variations thereof herein, is meant to encompass the items listed thereafter, and equivalents thereof as well as additional items. Although the foregoing application has been described in some detail by way of illustration and example, it is not to be limited thereby, but rather, only by the scope of the appended claims.

Claims

1. A computer room monitoring method, characterized in that: include: Acquire multimodal computer room data to be processed, wherein the multimodal computer room data includes sensor data, image data, and audio data; Performing weighted fusion on the sensor data to obtain fused sensor feature data; Inputting the image data into a pre-trained convolutional neural network model to obtain image data after feature extraction; Inputting the audio data into a pre-trained long short-term memory network model to obtain audio data after feature extraction; Analyzing the semantic consistency of the sensor feature data, the image data after feature extraction, and the audio data after feature extraction through a pre-trained contrastive learning model to obtain fused data; The fused data is input into a pre-trained computer room status prediction model to obtain a computer room status monitoring result.

2. The method according to claim 1, characterized in that The weighted fusion of the sensor data to obtain fused sensor feature data includes: performing encoding processing on the sensor data to obtain mapped sensor data; Calculating a weight score of the sensor data according to an attention mechanism; The mapped sensor data are weightedly fused according to the weight scores of the sensor data to obtain fused sensor feature data.

3. The method according to claim 1, characterized in that Inputting the image data into a pre-trained convolutional neural network model to obtain image data after feature extraction includes: Performing convolution pooling processing on the image data to obtain low-level features; The low-level features are stacked to generate image semantic information; The image semantic information is globally fused to obtain image data after feature extraction.

4. The method according to claim 1, wherein Inputting the audio data into a pre-trained long short-term memory network model to obtain feature-extracted audio data includes: Performing frame processing on the audio data to obtain an audio data feature sequence; Filtering the audio data feature sequence through the forget gate of the long short-term memory network model to obtain a filtered audio data feature sequence; The filtered audio data feature sequence is fused through the output gate output of the long short-term memory network model to obtain audio data after feature extraction.

5. The method according to claim 1, wherein The method of analyzing the semantic consistency of the sensor feature data, the image data after feature extraction, and the audio data after feature extraction by a pre-trained contrastive learning model to obtain fused data includes: Mapping the sensor feature data, the image data after feature extraction, and the audio data after feature extraction to obtain mapped semantic data; Establish sensor-image-audio loss function; According to the mapped semantic data and the sensor-image-audio loss function, the semantic consistency of the semantic data is analyzed to obtain fused data.

6. The method according to any one of claims 1 to 5, characterized in that Inputting the fused data into a pre-trained computer room status prediction model to obtain a computer room status monitoring result includes: Calculate the prediction probability of the computer room state of the fused data through a pre-trained classifier; Calculating a loss function for predicting the state of the computer room according to the prediction probability of the state of the computer room; The computer room state is predicted according to the loss function of the computer room state prediction to obtain a computer room state monitoring result.

7. A computer room monitoring device, characterized in that: include: An acquisition module, configured to acquire multimodal computer room data to be processed, wherein the multimodal computer room data includes sensor data, image data, and audio data; A fusion module, configured to perform weighted fusion on the sensor data to obtain fused sensor feature data; A first output module is used to input the image data into a pre-trained convolutional neural network model to obtain image data after feature extraction; A second output module is used to input the audio data into a pre-trained long short-term memory network model to obtain audio data after feature extraction; a contrastive learning module, configured to analyze the semantic consistency of the sensor feature data, the feature-extracted image data, and the feature-extracted audio data using a pre-trained contrastive learning model to obtain fused data; The third output module is used to input the fusion data into a pre-trained computer room status prediction model to obtain a computer room status monitoring result.

8. A computer room monitoring device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the computer room monitoring method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the computer room monitoring method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the computer room monitoring method according to any one of claims 1 to 6 when executed by a processor.

Images