An artificial intelligence mechanical equipment monitoring system and monitoring method

Abstract

The present invention proposes an artificial intelligence mechanical equipment monitoring system and monitoring method, which receives audio data generated by an audio acquisition device, a first image acquisition device and a second image acquisition device when the mechanical equipment is in a working state, first image data of the overall or local motion state of the mechanical equipment in the working state and second image data of the output workpiece of the mechanical equipment, extracts the motion characteristics of the mechanical equipment from the first image data, extracts the appearance characteristics of the output workpiece of the mechanical equipment from the second image data, classifies the audio data based on the motion characteristics of the mechanical equipment and / or the appearance characteristics of the output workpiece, trains the sound model of the mechanical equipment using the audio data as training sample data, and monitors the sound of the mechanical equipment using the sound model of the mechanical equipment, so as to timely and accurately predict the potential failure risks of the mechanical equipment.

Description

Technical Field

[0001] The present invention relates to the field of industrial manufacturing technology, and in particular to an artificial intelligence mechanical equipment monitoring system and a monitoring method. Background Art

[0002] With the introduction of the concept of Industry 4.0, industrial manufacturing has made a great stride from the information age to the intelligent age. In the Industry 4.0 era, while seamlessly integrating production equipment with production equipment and production equipment with control equipment through sensing devices and information networks, artificial intelligence is also introduced into the industrial manufacturing system, realizing real-time perception, identification, tracking, detection and intelligent management of the industrial manufacturing system. The existing mechanical equipment monitoring system uses monitoring devices to monitor the working parameters of mechanical equipment such as power, voltage, temperature or appearance to determine whether the mechanical equipment has faults. This type of detection solution cannot timely and accurately predict the potential failure risks of mechanical equipment. Summary of the invention

[0003] Based on the above problems, the present invention proposes an artificial intelligence mechanical equipment monitoring system and monitoring method, which can timely and accurately predict the potential failure risks of mechanical equipment.

[0004] In view of this, one aspect of the present invention proposes an artificial intelligence mechanical equipment monitoring system, comprising a data processing device and an audio acquisition device, a first image acquisition device, and a second image acquisition device, which are communicatively connected to the data processing device and installed on a side adjacent to the mechanical equipment, wherein the audio acquisition device is configured to acquire audio data generated when the mechanical equipment is in a working state and send it to the data processing device, the first image acquisition device is configured to acquire first image data of the overall or local motion state of the mechanical equipment in a working state and send it to the data processing device, the second image acquisition device is configured to acquire second image data of a workpiece output by the mechanical equipment and send it to the data processing device, and the data processing device is configured to:

[0005] S100: receiving the audio data, the first image data and the second image data from the audio acquisition device, the first image acquisition device and the second image acquisition device;

[0006] S200: extracting the motion characteristics of the mechanical device from the first image data;

[0007] S300: extracting the appearance features of the output workpiece from the second image data;

[0008] S400: Classifying the audio data based on the motion characteristics of the mechanical device and / or the appearance characteristics of the output workpiece;

[0009] S500: training the sound model of the mechanical equipment using the audio data as training sample data;

[0010] S600: Using the sound model of the mechanical equipment to monitor the sound of the mechanical equipment.

[0011] Furthermore, in the above-mentioned mechanical equipment monitoring system, before the step of extracting the motion characteristics of the mechanical equipment from the first image data, the data processing device is configured as follows:

[0012] S110: Segment the audio data, the first image data, and the second image data based on the production cycle of one or a group of workpieces of the same mechanical equipment, and save the association between the audio data, the first image data, and the second image data based on the generation time.

[0013] Furthermore, in the above-mentioned mechanical equipment monitoring system, in the step of extracting the motion characteristics of the mechanical equipment from the first image data, the data processing device is configured as follows:

[0014] S210: Identify the main body of the mechanical device or a local position of the mechanical device from the first image data;

[0015] S220: Obtaining an offset value of the mechanical device or a body of a local position of the mechanical device relative to a standard position at a corresponding time point in a current production cycle;

[0016] S230: Obtaining an offset curve of the body of the mechanical device or a local position of the mechanical device in the current production cycle based on the offset value;

[0017] S240: Extracting motion characteristics of the mechanical device from the offset curve.

[0018] Furthermore, in the above-mentioned mechanical equipment monitoring system, in the step of extracting the appearance features of the output workpiece from the second image data, the data processing device is configured as follows:

[0019] S310: Extracting one or more of the overall or local shape, size, texture, and color parameters of the output workpiece from the second image data as the appearance feature of the output workpiece.

[0020] Furthermore, in the above-mentioned mechanical equipment monitoring system, in the step of training the sound model of the mechanical equipment using the audio data and the classification information of the audio data as training sample data, the data processing device is configured as follows:

[0021] S510: performing constant edge value padding on the audio data so that the audio data under the same category have the same data length to form normalized audio sample data;

[0022] S520: Inputting the normalized audio sample data into a preset neural network model for training;

[0023] S530: Calculating the training error of the normalized audio sample data;

[0024] S540: When the training error is greater than a preset value, the connection weights between the neurons in each layer of the neural network model are adjusted and the process returns to the step of inputting the normalized audio sample data into the preset neural network model for training.

[0025] Another aspect of the present invention provides an artificial intelligence mechanical equipment monitoring method, comprising:

[0026] S100: receiving audio data generated by an audio acquisition device, a first image acquisition device, and a second image acquisition device in a mechanical device working state, first image data of an overall or partial motion state of the mechanical device in a working state, and second image data of a workpiece output by the mechanical device;

[0027] S200: extracting the motion characteristics of the mechanical device from the first image data;

[0028] S300: extracting the appearance features of the output workpiece from the second image data;

[0029] S400: Classifying the audio data based on the motion characteristics of the mechanical device and / or the appearance characteristics of the output workpiece;

[0030] S500: training the sound model of the mechanical equipment using the audio data as training sample data;

[0031] S600: Using the sound model of the mechanical equipment to monitor the sound of the mechanical equipment.

[0032] Furthermore, in the above-mentioned mechanical equipment monitoring method, before the step of extracting the motion characteristics of the mechanical equipment from the first image data, it also includes:

[0033] S110: Segment the audio data, the first image data, and the second image data based on the production cycle of one or a group of workpieces of the same mechanical equipment, and save the association between the audio data, the first image data, and the second image data based on the generation time.

[0034] Furthermore, in the above-mentioned mechanical equipment monitoring method, the step of extracting the motion characteristics of the mechanical equipment from the first image data specifically includes:

[0035] S210: Identify the main body of the mechanical device or a local position of the mechanical device from the first image data;

[0036] S220: Obtaining an offset value of the mechanical device or a body of a local position of the mechanical device relative to a standard position at a corresponding time point in a current production cycle;

[0037] S230: Obtaining an offset curve of the body of the mechanical device or a local position of the mechanical device in the current production cycle based on the offset value;

[0038] S240: Extracting motion characteristics of the mechanical device from the offset curve.

[0039] Furthermore, in the above-mentioned mechanical equipment monitoring method, the step of extracting the appearance features of the output workpiece from the second image data specifically includes:

[0040] S310: Extracting one or more of the overall or local shape, size, texture, and color parameters of the output workpiece from the second image data as the appearance feature of the output workpiece.

[0041] Furthermore, in the above-mentioned mechanical equipment monitoring method, the step of training the sound model of the mechanical equipment using the audio data and the classification information of the audio data as training sample data specifically includes:

[0042] S510: performing constant edge value padding on the audio data so that the audio data under the same category have the same data length to form normalized audio sample data;

[0043] S520: Inputting the normalized audio sample data into a preset neural network model for training;

[0044] S530: Calculating the training error of the normalized audio sample data;

[0045] S540: When the training error is greater than a preset value, the connection weights between the neurons in each layer of the neural network model are adjusted and the process returns to the step of inputting the normalized audio sample data into the preset neural network model for training.

[0046] The present invention proposes an artificial intelligence mechanical equipment monitoring system and monitoring method, which receives audio data generated by an audio acquisition device, a first image acquisition device and a second image acquisition device when the mechanical equipment is in a working state, first image data of the overall or local motion state of the mechanical equipment in the working state and second image data of the output workpiece of the mechanical equipment, extracts the motion characteristics of the mechanical equipment from the first image data, extracts the appearance characteristics of the output workpiece of the mechanical equipment from the second image data, classifies the audio data based on the motion characteristics of the mechanical equipment and / or the appearance characteristics of the output workpiece, trains the sound model of the mechanical equipment using the audio data as training sample data, and monitors the sound of the mechanical equipment using the sound model of the mechanical equipment, so as to timely and accurately predict the potential failure risks of the mechanical equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a schematic block diagram of an artificial intelligence mechanical equipment monitoring system provided by an embodiment of the present invention;

[0048] Figure 2 is a schematic flow chart of an artificial intelligence mechanical equipment monitoring method provided by an embodiment of the present invention;

[0049] Figure 3 is a schematic flow chart of a motion feature extraction method for a mechanical device provided by an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of motion feature extraction of a mechanical device provided by an embodiment of the present invention;

[0051] Figure 5 It is a schematic flow chart of a mechanical equipment sound model training method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0053] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0054] In the description of the present invention, the term "multiple" refers to two or more. Unless otherwise clearly defined, the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention. The terms "connection", "installation", "fixation", etc. should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more.

[0055] In the description of this specification, the description of the terms "one embodiment", "some implementations", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0056] Refer to the following Figures 1 to 5 To describe an artificial intelligence mechanical equipment monitoring system and monitoring method provided according to some embodiments of the present invention.

[0057] like Figure 1 As shown, one aspect of the present invention proposes an artificial intelligence mechanical equipment monitoring system, including a data processing device and an audio acquisition device, a first image acquisition device and a second image acquisition device which are installed on a side adjacent to the mechanical equipment and are communicatively connected to the data processing device. The audio acquisition device is configured to acquire audio data generated when the mechanical equipment is in a working state and send it to the data processing device. The first image acquisition device is configured to acquire first image data of the overall or local motion state of the mechanical equipment in a working state and send it to the data processing device. The second image acquisition device is configured to acquire second image data of a workpiece output by the mechanical equipment and send it to the data processing device.

[0058] In the technical solution of the present invention, the mechanical equipment is the mechanical equipment of each production link of the manufacturing system. The audio acquisition device is installed on a side adjacent to the mechanical equipment, and is used to collect the audio data generated by the mechanical equipment in the working state. In the industrial manufacturing system, each manufacturing / processing equipment will emit various sounds of different sizes, frequencies, and durations due to the friction or collision between the parts or with other surrounding objects such as the ground, conveyor belts, and even space as it moves in its entirety or in part in the working state. In a manufacturing / processing cycle of the same or the same group of workpieces, some sounds are synchronized with the manufacturing / processing cycle of the workpiece and appear once or multiple times in each cycle; some sounds also appear periodically, but they are not related to the manufacturing / processing cycle of the workpiece, but are caused by the physical structure properties or electrical properties of the mechanical equipment itself; some sounds appear sporadically rather than periodically, which may be caused by various internal or external reasons of the mechanical equipment, such as aging or even rupture of the parts of the mechanical equipment, or impurities in the workpiece material, or abnormal displacement of the operating parts of the mechanical equipment. By collecting, processing and analyzing these periodic or non-periodic sounds, it is possible to monitor the working status of the mechanical equipment and predict the potential failure risks of the mechanical equipment in a timely and accurate manner.

[0059] As mentioned above, mechanical equipment will emit various sounds due to different reasons during operation. After these sounds are mixed together, it is difficult to extract their sound features. Simply relying on the sound features extracted from the audio data collected by the audio acquisition device is not enough to obtain the reason why the mechanical equipment emits the sound, and it is difficult to monitor the mechanical equipment based on this to predict the risk of failure. In the technical solution of the present invention, by setting the first image acquisition device to collect the first image data of the overall or local motion state of the mechanical equipment in the working state, and setting the second image acquisition device to collect the second image data of the workpiece output by the mechanical equipment, combined with the audio data, the first image data and the second image data, it is possible to monitor the working state of the mechanical equipment, detect abnormal conditions in time, and accurately predict the potential failure risks of the mechanical equipment.

[0060] In the technical solution of the present invention, the first image acquisition device is installed on a side adjacent to the mechanical equipment, and its image acquisition range covers the overall or local key components of the mechanical equipment. The first image acquisition device can be an independent image acquisition device or a combination of multiple image acquisition devices. For example, multiple image acquisition devices can be set to respectively acquire image data of multiple different components. Through the first image acquisition device, the first image data such as the overall or local vibration state, vibration amplitude or motion path of the mechanical equipment can be obtained, and from the first image data, it can be analyzed whether there is any abnormality in the overall or local vibration or movement of the mechanical equipment.

[0061] In the technical solution of the present invention, a second image acquisition device is installed at the workpiece outlet position of the mechanical equipment to collect the second image data of the workpiece output by the mechanical equipment, so that the data processing device can monitor the workpiece output conditions of each production link in the manufacturing system, especially the appearance of the workpiece. Mechanical equipment and their unloading methods are different, such as different types of CNC machine tools, their workpiece outlet refers to the unloading position of the workpiece after its manufacture and processing. For example, for CNC machine tools that use conveyor belts for feeding and unloading, the image acquisition device can be installed on both sides of the conveyor belt at the position of the discharge port of the CNC machine tool; and for manufacturing systems such as robotic arms that use clamping or magnetic suction to unload materials, the second image acquisition device can be installed at the initial position or target position of the robotic arm moving the workpiece. In the technical solutions of some embodiments of the present invention, the second image acquisition device can be an independent image acquisition device or a combination of multiple image acquisition devices. For example, in some possible embodiments of the present invention, three or more fixed-position image acquisition devices are arranged at the workpiece exit position of each mechanical device, and the workpiece image data collected by the three or more fixed-position image acquisition devices are combined to obtain a stereoscopic image of the workpiece. Although the arrangement cost of this embodiment is higher, it can obtain a stereoscopic image of the workpiece in a shorter time and in a more efficient manner. In some possible embodiments of the present invention, an image acquisition device that can be moved in a small range is arranged at the workpiece exit position of each mechanical device, and the stereoscopic image of the workpiece is obtained by moving the image acquisition device. In some possible embodiments of the present invention, a fixed image acquisition device is arranged at the workpiece exit position of each mechanical device, and the workpiece is moved or rotated by using a gripping device such as a mechanical arm so that the image acquisition device obtains a stereoscopic image of the workpiece. For ease of understanding, unless otherwise specified, the embodiments of the present invention proposed later are all based on the case where the image acquisition device and the workpiece are both in fixed positions to collect workpiece image data as an example, so as to illustrate the technical solution of the present invention to monitor the mechanical equipment using these data after acquiring the appearance characteristics of the workpiece in this scenario, but it does not mean that the protection scope of the present invention is limited to the case where the image acquisition device and the workpiece are both in fixed positions. A person skilled in the art can obtain a three-dimensional image of the workpiece by moving the image acquisition device or the workpiece, or by setting a plurality of image acquisition devices around the workpiece. Through the second image acquisition device, the appearance characteristics of the workpiece output by the mechanical equipment, including the second image data of shape, size, texture, and color, can be obtained, and from the second image data, it can be analyzed whether there are abnormalities in the workpiece manufacturing and processing process of the mechanical equipment.

[0062] In the technical solution of the present invention, the data processing device can be a cloud server connected to the manufacturing system via the Internet, or a control terminal or workstation installed in a production environment. In some embodiments of the present invention, the data processing device is connected to a control device. Similarly, the control device can also be a cloud server connected to the manufacturing system via the Internet, or a control terminal or workstation installed in a production environment. The data processing device sends the monitoring results of the mechanical equipment to the control device, and the control device controls the mechanical equipment in response to the monitoring results. In other embodiments of the present invention, the data processing device can be directly connected to the mechanical equipment to control the mechanical equipment.

[0063] In the mechanical equipment monitoring system provided by the present invention, the data processing device includes a processor and a memory. Figure 2 As shown, the data processing device is configured to execute the following steps by running the program stored in the memory through the processor:

[0064] S100: receiving the audio data, the first image data and the second image data from the audio acquisition device, the first image acquisition device and the second image acquisition device;

[0065] S200: extracting the motion characteristics of the mechanical device from the first image data;

[0066] S300: extracting the appearance features of the output workpiece from the second image data;

[0067] S400: Classifying the audio data based on the motion characteristics of the mechanical device and / or the appearance characteristics of the output workpiece;

[0068] S500: training the sound model of the mechanical equipment using the audio data as training sample data;

[0069] S600: Using the sound model of the mechanical equipment to monitor the sound of the mechanical equipment.

[0070] The present invention proposes an artificial intelligence mechanical equipment monitoring system and monitoring method, which receives audio data generated by an audio acquisition device, a first image acquisition device and a second image acquisition device when the mechanical equipment is in a working state, first image data of the overall or local motion state of the mechanical equipment in the working state and second image data of the output workpiece of the mechanical equipment, extracts the motion characteristics of the mechanical equipment from the first image data, extracts the appearance characteristics of the output workpiece of the mechanical equipment from the second image data, classifies the audio data based on the motion characteristics of the mechanical equipment and / or the appearance characteristics of the output workpiece, trains the sound model of the mechanical equipment using the audio data as training sample data, and monitors the sound of the mechanical equipment using the sound model of the mechanical equipment, so as to timely and accurately predict the potential failure risks of the mechanical equipment.

[0071] In the above-mentioned mechanical equipment monitoring system, before the step of extracting the motion characteristics of the mechanical equipment from the first image data, the data processing device is configured as follows:

[0072] S110: Segment the audio data, the first image data, and the second image data based on the production cycle of one or a group of workpieces of the same mechanical equipment, and save the association between the audio data, the first image data, and the second image data based on the generation time.

[0073] Some mechanical equipment can only manufacture and process one workpiece at a time, while some mechanical equipment can manufacture and process a group of multiple identical workpieces at the same time. The minimum cycle of one or a group of workpiece output is taken as the production cycle, and the audio data, the first image data and the second image data are segmented to obtain the audio data generated in the working state of the mechanical equipment in each production cycle, the first image data of the overall or partial motion state, and the second image data of the workpiece output by the mechanical equipment. The association relationship between the audio data, the first image data and the second image data is saved based on the generation time, so that when the abnormal situation of the mechanical equipment is analyzed in the first image data and the second image data, the audio data of the corresponding time can be obtained through the corresponding image generation time, so that the training sample data of the sound model of the mechanical equipment is more targeted.

[0074] like Figure 3 As shown, in the above-mentioned mechanical equipment monitoring system, in the step of extracting the motion characteristics of the mechanical equipment from the first image data, the data processing device is configured as follows:

[0075] S210: Identify the body of the mechanical equipment or the local position of the mechanical equipment from the first image data. In some embodiments of the present invention, the first image acquisition device is installed in a fixed position and a fixed viewing angle, and its shooting range covers the movement range of the mechanical equipment or the local position of the mechanical equipment in each production cycle. By performing edge recognition and comparison on multiple frames of images in the first image data, the environmental image can be distinguished from the body image of the mechanical equipment or the local position of the mechanical equipment, thereby eliminating the first image data in the environmental image to avoid interference in the data processing process.

[0076] S220: Obtain the offset value of the main body of the mechanical device or the local position of the mechanical device relative to the standard position at the corresponding time point of the current production cycle. In the processing process of step S210, the edge image of the main body of the mechanical device or the local position of the mechanical device has been obtained. In the case where the first image acquisition device adopts a fixed position and fixed viewing angle installation method, the corresponding offset value can be obtained by comparing the edge image of the main body of the mechanical device or the local position of the mechanical device in the first image data with the standard position at the corresponding time point of the current production cycle. For the mechanical device or the local position of the mechanical device that is stationary in the working state (excluding the vibration in the working state), its standard position does not change during the production cycle. The standard position of the mechanical device can be calculated by obtaining the image data of the non-working state by the first image acquisition device. For the mechanical device or the local position of the mechanical device that needs to actively move in the working state, its standard position for each production cycle is a periodic motion path, and the standard position at the corresponding time point in different production cycles is the same. Figure 4The corresponding relationship between the motion path of a cylindrical mechanical equipment component (the local position of the mechanical equipment) and the production cycle under an embodiment of the present invention is shown, wherein point A is the starting position of the motion path of the center point of the cylindrical component, point B is the farthest point position of the motion path of the center point of the cylindrical component, the solid line between points AB is the standard motion path of the center point of the cylindrical component, each point on the standard motion path is the standard position at each time point of the corresponding production cycle, and the direction of the arrow is the motion direction of the center point. The dotted line between points AB is the actual motion path of the center point of the cylindrical component in the working state acquired by the first image acquisition device. T0->T1->T2 is a complete production cycle, wherein T0 is the starting time point of a production cycle, T1 is the time point when the cylindrical component moves to the farthest point position B and starts to reset, and T2 is the end time point of a production cycle. It should be noted that the smooth curve in the figure is only for illustration. Since the vibration amplitude and vibration frequency of the mechanical equipment or the local position of the mechanical equipment in the working state have great randomness, its actual motion path will be more complicated.

[0077] S230: Based on the offset value, an offset curve of the main body of the mechanical device or the local position of the mechanical device in the current production cycle is obtained. In each complete production cycle, with time as the x value and the offset value corresponding to each time point as the y value, a curve corresponding to the function y(x) can be fitted as the offset curve of the main body of the mechanical device or the local position of the mechanical device in the production cycle.

[0078] S240: Extract the motion characteristics of the mechanical equipment from the offset curve. Specifically, compare and analyze the offset curves of multiple production cycles, determine the time period with similar offset characteristics in the multiple offset curves, and extract the offset curves in the time period as the motion characteristics of the mechanical equipment. The similar offset characteristics include that the local part of the offset curve has similar frequency, peak-to-valley amplitude, etc. Figure 4 For example, after comparing and analyzing the offset curves of multiple production cycles, it is determined that the offset curves between time periods (T3, T4) have the same offset characteristics, and the offset curves between time periods (T3, T4) are extracted as the motion characteristics of the mechanical equipment. When the mechanical equipment has abnormalities, such as aging or damage of components, and when these abnormalities can be reflected in the vibration conditions of the mechanical equipment in the working state, it means that these abnormalities have reached a relatively serious level that requires attention.

[0079] In the above-mentioned mechanical equipment monitoring system, in the step of extracting the appearance features of the output workpiece from the second image data, the data processing device is configured as follows:

[0080] S310: Extract one or more of the overall or local shape, size, texture, and color parameters of the output workpiece from the second image data as the appearance features of the output workpiece. The appearance features of the output workpiece refer to the features that the appearance of the current output workpiece is different from that of the standard output workpiece, for example, the current output workpiece is different from the standard output workpiece in shape or size, or the current output workpiece has textures such as scratches that are not present on the standard output workpiece, or has different surface roughness. Similarly, when the mechanical equipment has abnormalities, such as aging or damage of components, and when these abnormalities can be reflected in the appearance features of the output workpiece of the mechanical equipment, it means that these abnormalities have reached a serious level that requires attention.

[0081] In the above-mentioned mechanical equipment monitoring system, in the step of classifying the audio data based on the motion characteristics of the mechanical equipment and / or the appearance characteristics of the output workpiece, the data processing device is configured as follows:

[0082] S410: Determine a plurality of the first image data having similar offset features or a plurality of the second image data having similar appearance features, classify a plurality of the audio data corresponding to the plurality of the first image data or the plurality of the second image data into the same category and generate corresponding category labels. Classify the audio data based on the offset features or the appearance features, so as to train the sound model of the mechanical equipment using these categories and the audio data as training samples.

[0083] like Figure 5 As shown, in the above-mentioned mechanical equipment monitoring system, in the step of training the sound model of the mechanical equipment using the audio data and the classification information of the audio data as training sample data, the data processing device is configured as follows:

[0084] S510: performing constant edge value padding on the audio data so that the audio data under the same category have the same data length to form normalized audio sample data;

[0085] S520: Inputting the normalized audio sample data into a preset neural network model for training;

[0086] S530: Calculating the training error of the normalized audio sample data;

[0087] S540: When the training error is greater than a preset value, the connection weights between the neurons in each layer of the neural network model are adjusted and the process returns to the step of inputting the normalized audio sample data into the preset neural network model for training.

[0088] Due to differences in the acquisition environment and the audio clipping algorithm, the length of the audio data of each production cycle obtained after segmentation will not be completely consistent. In order to enable the subsequent training process to use a unified weight matrix for calculation, it is necessary to fill the audio data with constant edge values ​​so that the audio data under the same category have the same data length. In the technical solutions of some embodiments of the present invention, the audio data corresponding to the time period with the same offset characteristics can be used as training samples. Due to the particularity of audio data, the sampling rate used by the audio acquisition device, or when the audio data is processed by the processing device to reduce the sampling rate, in order to obtain sufficient sound features, its sampling rate still needs to be higher than a specific value. After converting the audio data into an input vector, it is still a relatively large volume of data. Using the audio data of a complete production cycle as training sample data has high requirements on the configuration and performance of the processing device, while intercepting the audio data corresponding to the time period with the same offset characteristics as training samples can reduce the configuration and performance requirements of the processing device.

[0089] like Figure 2 As shown, another aspect of the present invention provides an artificial intelligence mechanical equipment monitoring method, comprising:

[0090] S100: receiving audio data generated by an audio acquisition device, a first image acquisition device, and a second image acquisition device in a mechanical device working state, first image data of an overall or partial motion state of the mechanical device in a working state, and second image data of a workpiece output by the mechanical device;

[0091] S200: extracting the motion characteristics of the mechanical device from the first image data;

[0092] S300: extracting the appearance features of the workpiece output by the mechanical equipment from the second image data;

[0093] S400: Classifying the audio data based on the motion characteristics of the mechanical device and / or the appearance characteristics of the output workpiece;

[0094] S500: training the sound model of the mechanical equipment using the audio data as training sample data;

[0095] S600: Using the sound model of the mechanical equipment to monitor the sound of the mechanical equipment.

[0096] The present invention proposes an artificial intelligence mechanical equipment monitoring system and monitoring method, which receives audio data generated by an audio acquisition device, a first image acquisition device and a second image acquisition device when the mechanical equipment is in a working state, first image data of the overall or local motion state of the mechanical equipment in the working state and second image data of the output workpiece of the mechanical equipment, extracts the motion characteristics of the mechanical equipment from the first image data, extracts the appearance characteristics of the output workpiece of the mechanical equipment from the second image data, classifies the audio data based on the motion characteristics of the mechanical equipment and / or the appearance characteristics of the output workpiece, trains the sound model of the mechanical equipment using the audio data as training sample data, and monitors the sound of the mechanical equipment using the sound model of the mechanical equipment, so as to timely and accurately predict the potential failure risks of the mechanical equipment.

[0097] In the above-mentioned mechanical equipment monitoring method, before the step of extracting the motion characteristics of the mechanical equipment from the first image data, the method further includes:

[0098] S110: Segment the audio data, the first image data, and the second image data based on the production cycle of one or a group of workpieces of the same mechanical equipment, and save the association between the audio data, the first image data, and the second image data based on the generation time.

[0099] Some mechanical equipment can only manufacture and process one workpiece at a time, while some mechanical equipment can manufacture and process a group of multiple identical workpieces at the same time. The minimum cycle of one or a group of workpiece output is taken as the production cycle, and the audio data, the first image data and the second image data are segmented to obtain the audio data generated in the working state of the mechanical equipment in each production cycle, the first image data of the overall or partial motion state, and the second image data of the workpiece output by the mechanical equipment. The association relationship between the audio data, the first image data and the second image data is saved based on the generation time, so that when the abnormal situation of the mechanical equipment is analyzed in the first image data and the second image data, the audio data of the corresponding time can be obtained through the corresponding image generation time, so that the training sample data of the sound model of the mechanical equipment is more targeted.

[0100] like Figure 3 As shown, in the above-mentioned mechanical equipment monitoring method, the step of extracting the motion characteristics of the mechanical equipment from the first image data specifically includes:

[0101] S210: Identify the body of the mechanical equipment or the local position of the mechanical equipment from the first image data. In some embodiments of the present invention, the first image acquisition device is installed in a fixed position and a fixed viewing angle, and its shooting range covers the movement range of the mechanical equipment or the local position of the mechanical equipment in each production cycle. By performing edge recognition and comparison on multiple frames of images in the first image data, the environmental image can be distinguished from the body image of the mechanical equipment or the local position of the mechanical equipment, thereby eliminating the first image data in the environmental image to avoid interference in the data processing process.

[0102] S220: Obtain the offset value of the main body of the mechanical device or the local position of the mechanical device relative to the standard position at the corresponding time point of the current production cycle. In the processing process of step S210, the edge image of the main body of the mechanical device or the local position of the mechanical device has been obtained. In the case where the first image acquisition device adopts a fixed position and fixed viewing angle installation method, the corresponding offset value can be obtained by comparing the edge image of the main body of the mechanical device or the local position of the mechanical device in the first image data with the standard position at the corresponding time point of the current production cycle. For the mechanical device or the local position of the mechanical device that is stationary in the working state (excluding the vibration in the working state), its standard position does not change during the production cycle. The standard position of the mechanical device can be calculated by obtaining the image data of the non-working state by the first image acquisition device. For the mechanical device or the local position of the mechanical device that needs to actively move in the working state, its standard position for each production cycle is a periodic motion path, and the standard position at the corresponding time point in different production cycles is the same. Figure 4The corresponding relationship between the motion path of a cylindrical mechanical equipment component (the local position of the mechanical equipment) and the production cycle under an embodiment of the present invention is shown, wherein point A is the starting position of the motion path of the center point of the cylindrical component, point B is the farthest point position of the motion path of the center point of the cylindrical component, the solid line between points AB is the standard motion path of the center point of the cylindrical component, each point on the standard motion path is the standard position at each time point of the corresponding production cycle, and the direction of the arrow is the motion direction of the center point. The dotted line between points AB is the actual motion path of the center point of the cylindrical component in the working state acquired by the first image acquisition device. T0->T1->T2 is a complete production cycle, wherein T0 is the starting time point of a production cycle, T1 is the time point when the cylindrical component moves to the farthest point position B and starts to reset, and T2 is the end time point of a production cycle. It should be noted that the smooth curve in the figure is only for illustration. Since the vibration amplitude and vibration frequency of the mechanical equipment or the local position of the mechanical equipment in the working state have great randomness, its actual motion path will be more complicated.

[0103] S230: Based on the offset value, an offset curve of the main body of the mechanical device or the local position of the mechanical device in the current production cycle is obtained. In each complete production cycle, with time as the x value and the offset value corresponding to each time point as the y value, a curve corresponding to the function y(x) can be fitted as the offset curve of the main body of the mechanical device or the local position of the mechanical device in the production cycle.

[0104] S240: Extract the motion characteristics of the mechanical equipment from the offset curve. Specifically, compare and analyze the offset curves of multiple production cycles, determine the time period with similar offset characteristics in the multiple offset curves, and extract the offset curves in the time period as the motion characteristics of the mechanical equipment. The similar offset characteristics include that the local part of the offset curve has similar frequency, peak-to-valley amplitude, etc. Figure 4 For example, after comparing and analyzing the offset curves of multiple production cycles, it is determined that the offset curves between time periods (T3, T4) have the same offset characteristics, and the offset curves between time periods (T3, T4) are extracted as the motion characteristics of the mechanical equipment. When the mechanical equipment has abnormalities, such as aging or damage of components, and when these abnormalities can be reflected in the vibration conditions of the mechanical equipment in the working state, it means that these abnormalities have reached a relatively serious level that requires attention.

[0105] In the above-mentioned mechanical equipment monitoring method, the step of extracting the appearance features of the output workpiece from the second image data specifically includes:

[0106] S310: Extract one or more of the overall or local shape, size, texture, and color parameters of the output workpiece from the second image data as the appearance features of the output workpiece. The appearance features of the output workpiece refer to the features that the appearance of the current output workpiece is different from that of the standard output workpiece, for example, the current output workpiece is different from the standard output workpiece in shape or size, or the current output workpiece has textures such as scratches that are not present on the standard output workpiece, or has different surface roughness. Similarly, when the mechanical equipment has abnormalities, such as aging or damage of components, and when these abnormalities can be reflected in the appearance features of the output workpiece of the mechanical equipment, it means that these abnormalities have reached a serious level that requires attention.

[0107] In the above-mentioned mechanical equipment monitoring method, in the step of classifying the audio data based on the motion characteristics of the mechanical equipment and / or the appearance characteristics of the output workpiece, the data processing device is configured as follows:

[0108] S410: Determine a plurality of the first image data having similar offset features or a plurality of the second image data having similar appearance features, classify a plurality of the audio data corresponding to the plurality of the first image data or the plurality of the second image data into the same category and generate corresponding category labels. Classify the audio data based on the offset features or the appearance features, so as to train the sound model of the mechanical equipment using these categories and the audio data as training samples.

[0109] like Figure 5 As shown, in the above-mentioned mechanical equipment monitoring method, the step of training the sound model of the mechanical equipment using the audio data and the classification information of the audio data as training sample data specifically includes:

[0110] S510: performing constant edge value padding on the audio data so that the audio data under the same category have the same data length to form normalized audio sample data;

[0111] S520: Inputting the normalized audio sample data into a preset neural network model for training;

[0112] S530: Calculating the training error of the normalized audio sample data;

[0113] S540: When the training error is greater than a preset value, the connection weights between the neurons in each layer of the neural network model are adjusted and the process returns to the step of inputting the normalized audio sample data into the preset neural network model for training.

[0114] Due to differences in the acquisition environment and the audio clipping algorithm, the length of the audio data of each production cycle obtained after segmentation will not be completely consistent. In order to enable the subsequent training process to use a unified weight matrix for calculation, it is necessary to fill the audio data with constant edge values ​​so that the audio data under the same category have the same data length. In the technical solutions of some embodiments of the present invention, the audio data corresponding to the time period with the same offset characteristics can be used as training samples. Due to the particularity of audio data, the sampling rate used by the audio acquisition device, or when the audio data is processed by the processing device to reduce the sampling rate, in order to obtain sufficient sound features, its sampling rate still needs to be higher than a specific value. After converting the audio data into an input vector, it is still a relatively large volume of data. Using the audio data of a complete production cycle as training sample data has high requirements on the configuration and performance of the processing device, while intercepting the audio data corresponding to the time period with the same offset characteristics as training samples can reduce the configuration and performance requirements of the processing device.

[0115] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0116] According to the embodiments of the present invention as described above, these embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and changes can be made based on the above description. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can make good use of the present invention and the modified use based on the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An artificial intelligence mechanical equipment monitoring system, characterized in that: The invention comprises a data processing device and an audio acquisition device, a first image acquisition device and a second image acquisition device which are communicatively connected to the data processing device and installed on a side adjacent to the mechanical equipment. The audio acquisition device is configured to acquire audio data generated when the mechanical equipment is in a working state and send it to the data processing device. The first image acquisition device is configured to acquire first image data of the overall or partial motion state of the mechanical equipment in a working state and send it to the data processing device. The second image acquisition device is configured to acquire second image data of a workpiece output by the mechanical equipment and send it to the data processing device. The data processing device is configured to: S100: receiving the audio data, the first image data and the second image data from the audio acquisition device, the first image acquisition device and the second image acquisition device; S200: extracting the motion characteristics of the mechanical device from the first image data; S300: extracting the appearance features of the output workpiece from the second image data; S400: Classifying the audio data based on the motion characteristics of the mechanical device and the appearance characteristics of the output workpiece; S500: training the sound model of the mechanical equipment using the audio data as training sample data; S600: Using the sound model of the mechanical equipment to monitor the sound of the mechanical equipment.

2. The mechanical equipment monitoring system according to claim 1, characterized in that: Before the step of extracting the motion feature of the mechanical device from the first image data, the data processing device is configured to: S110: Segment the audio data, the first image data, and the second image data based on the production cycle of one or a group of workpieces of the same mechanical equipment, and save the association between the audio data, the first image data, and the second image data based on the generation time.

3. The mechanical equipment monitoring system according to claim 2, characterized in that: In the step of extracting the motion feature of the mechanical device from the first image data, the data processing device is configured to: S210: Identify the main body of the mechanical device or a local position of the mechanical device from the first image data; S220: Obtaining an offset value of the mechanical device or a body of a local position of the mechanical device relative to a standard position at a corresponding time point in a current production cycle; S230: Obtaining an offset curve of the body of the mechanical device or a local position of the mechanical device in the current production cycle based on the offset value; S240: Extracting motion characteristics of the mechanical device from the offset curve.

4. The mechanical equipment monitoring system according to claim 2, characterized in that: In the step of extracting the appearance features of the output workpiece from the second image data, the data processing device is configured to: S310: Extracting one or more of the overall or local shape, size, texture, and color parameters of the output workpiece from the second image data as the appearance feature of the output workpiece.

5. The mechanical equipment monitoring system according to any one of claims 1 to 4, characterized in that: In the step of training the sound model of the mechanical equipment using the audio data and the classification information of the audio data as training sample data, the data processing device is configured to: S510: performing constant edge value padding on the audio data so that the audio data under the same category have the same data length to form normalized audio sample data; S520: Inputting the normalized audio sample data into a preset neural network model for training; S530: Calculating the training error of the normalized audio sample data; S540: When the training error is greater than a preset value, the connection weights between the neurons in each layer of the neural network model are adjusted and the process returns to the step of inputting the normalized audio sample data into the preset neural network model for training.

6. An artificial intelligence mechanical equipment monitoring method, characterized in that: include: S100: receiving audio data generated by an audio acquisition device, a first image acquisition device, and a second image acquisition device in a mechanical device working state, first image data of an overall or partial motion state of the mechanical device in a working state, and second image data of a workpiece output by the mechanical device; S200: extracting the motion characteristics of the mechanical device from the first image data; S300: extracting the appearance features of the output workpiece from the second image data; S400: Classifying the audio data based on the motion characteristics of the mechanical device and the appearance characteristics of the output workpiece; S500: training the sound model of the mechanical equipment using the audio data as training sample data; S600: Using the sound model of the mechanical equipment to monitor the sound of the mechanical equipment.

7. The mechanical equipment monitoring method according to claim 6, characterized in that: Before the step of extracting the motion feature of the mechanical device from the first image data, the method further includes: S110: Segment the audio data, the first image data, and the second image data based on the production cycle of one or a group of workpieces of the same mechanical equipment, and save the association between the audio data, the first image data, and the second image data based on the generation time.

8. The mechanical equipment monitoring method according to claim 7, characterized in that: The step of extracting the motion characteristics of the mechanical device from the first image data specifically includes: S210: Identify the main body of the mechanical device or a local position of the mechanical device from the first image data; S220: Obtaining an offset value of the mechanical device or a body of a local position of the mechanical device relative to a standard position at a corresponding time point in a current production cycle; S230: Obtaining an offset curve of the body of the mechanical device or a local position of the mechanical device in the current production cycle based on the offset value; S240: Extracting motion characteristics of the mechanical device from the offset curve.

9. The mechanical equipment monitoring method according to claim 8, characterized in that: The step of extracting the appearance features of the output workpiece from the second image data specifically includes: S310: Extracting one or more of the overall or local shape, size, texture, and color parameters of the output workpiece from the second image data as the appearance feature of the output workpiece.

10. The mechanical equipment monitoring method according to any one of claims 6 to 9, characterized in that: The step of training the sound model of the mechanical equipment using the audio data and the classification information of the audio data as training sample data specifically includes: S510: performing constant edge value padding on the audio data so that the audio data under the same category have the same data length to form normalized audio sample data; S520: Inputting the normalized audio sample data into a preset neural network model for training; S530: Calculating the training error of the normalized audio sample data; S540: When the training error is greater than a preset value, the connection weights between the neurons in each layer of the neural network model are adjusted and the process returns to the step of inputting the normalized audio sample data into the preset neural network model for training.

Images