An elevator fault accident identification method, system, device and medium based on intelligent image recognition

Abstract

The application provides an elevator fault accident identification method, system, device and medium based on intelligent image recognition, and belongs to the technical field of elevator fault diagnosis. The method steps are as follows: an intelligent image sensor is assembled in advance on the roof of an elevator car and an elevator shaft; image data in elevator operation data collected by the intelligent image sensor is preprocessed, and preliminary feature extraction is performed; a data set is constructed by combining the image data in the elevator operation data, the preliminary extracted features and elevator operation data of known fault types; a deep learning model is constructed and trained using the data set to obtain an elevator fault identification model; the image data in the elevator operation data collected in real time and the preliminary extracted features are input into the elevator fault identification model to identify elevator operation faults and alarm. The application identifies elevator faults by combining intelligent image sensors with a deep learning model, thereby improving the fault identification efficiency and accuracy during elevator operation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of elevator fault diagnosis, and particularly relates to an elevator fault accident identification method, system, device and medium based on intelligent image recognition. BACKGROUND

[0002] As an important vertical transportation tool in modern cities, the safety, reliability and operation efficiency of elevators are increasingly concerned. The complexity and diversity of elevator systems make elevator fault diagnosis extremely complex. Traditional elevator fault diagnosis mainly relies on manual inspection and regular maintenance, which is not only inefficient, but also difficult to cover all potential fault points comprehensively. With the development of Internet of Things, big data and artificial intelligence technologies, a new generation of elevator fault diagnosis has emerged and is developing towards intelligence, automation and remote.

[0003] The new generation of elevator fault diagnosis is realized through sensors, which requires deploying a large number of sensors such as speed sensors, temperature sensors and pressure sensors in elevators. First, these sensors not only complicate the networking, increase the complexity and maintenance difficulty of the elevator system, but also have relatively high procurement and maintenance costs, ultimately increasing the economic burden of elevator manufacturers and operators, and limiting the widespread application of elevator fault diagnosis technology. Second, the performance of these sensors is greatly affected by environmental factors. In the elevator operating environment, changes in temperature, humidity and other environmental factors may cause the performance of the sensors to decline, thereby affecting the accuracy of fault identification. In addition, problems such as sensor aging and damage may also cause inaccurate or missing data, further affecting the reliability of diagnosis.

[0004] Therefore, it is necessary to provide an elevator fault accident identification method, system, device and medium based on intelligent image recognition to solve the above technical problems. SUMMARY

[0005] In view of the above defects in the prior art that traditional elevator fault diagnosis relies on manual work and is inefficient, and the new sensor-based method increases the operating cost of elevators and the sensors are greatly affected by environmental factors, affecting the accuracy of elevator fault identification, the application provides an elevator fault accident identification method, system, device and medium based on intelligent image recognition to solve the above technical problems.

[0006] In a first aspect, the application provides an elevator fault accident identification method based on intelligent image recognition, comprising the following steps:

[0007] S1. Pre-installing intelligent image sensors on the roof of the elevator car and the elevator shaft, respectively, to collect elevator operation data;

[0008] S2. Preprocess the image data in the elevator operation data collected by the intelligent image sensor, and perform preliminary feature extraction, combine the image data in the elevator operation data and the preliminary extracted features with the elevator operation data of known fault types to construct a data set;

[0009] S3. Construct a deep learning model and train it using the data set to obtain an elevator fault identification model;

[0010] S4. Input the image data and preliminary extracted features in the elevator operation data collected in real time by the elevator into the elevator fault identification model, identify whether there is a fault in the elevator operation and the corresponding fault type, and alarm according to the corresponding fault type.

[0011] Further, the specific steps of step S1 are as follows:

[0012] S11. An intelligent camera device as an intelligent image sensor is installed on the roof of the elevator car and the elevator pit, respectively, and the intelligent camera device is provided with a microphone module;

[0013] S12. Network connection between the intelligent camera device and the back-end monitoring server is established;

[0014] S13. The intelligent camera device collects elevator operation data in real time and sends it to the back-end monitoring server; the elevator operation data includes image data and sound data.

[0015] Further, the specific steps of step S2 are as follows:

[0016] S21. The back-end monitoring server performs noise reduction processing, contrast enhancement processing and grayscale processing on the image data in the elevator operation data, and uses a feature extraction algorithm to perform preliminary feature extraction to obtain the grayscale features of the image and the contour features of the objects in the image;

[0017] S22. Obtain elevator fault image data from a public data set, including elevator fire image and elevator personnel fall image;

[0018] S23. Mark the image data and grayscale features in the elevator operation data as elevator reverse fire data, and extract the grayscale features after preprocessing the elevator fire image in the public data set, and mark it as elevator forward fire data, to generate an elevator fire state data set;

[0019] S24. Mark the elevator fault operation data and the elevator normal operation data according to the image data in the elevator operation data and the contour features of the objects in the image, to generate an elevator operation recognition data set;

[0020] S25. Marking normal images and personnel falling images of personnel according to the image data in the elevator operation data and the contour feature of the object in the image, and generating an elevator personnel state data set.

[0021] Further, the step S21 specifically comprises the following steps:

[0022] S211. Determining the noise reduction precision, selecting a filter window according to the noise reduction precision, and sequentially positioning the image data of the image using the filter window;

[0023] S212. Sorting the pixel values in the positioning area according to the size, taking the pixel value of the middle value as the pixel value of the entire positioning area, and completing the noise reduction processing;

[0024] S213. Calculating a histogram for the image data of the elevator operation data, determining a mapping function according to the histogram, and mapping the brightness value of the image data according to the mapping function, to complete the contrast enhancement processing;

[0025] S214. Converting the image data of the elevator operation data from a color RGB space to a YUV space, and performing weighted averaging on the red component, green component and blue component according to a preset weight component to obtain a gray value, to complete the grayscale processing;

[0026] S215. Using the SIFT feature extraction algorithm to construct a Gaussian kernel tower, detecting the key points in the image by comparing the pixel differences at different scales, and calculating the gradient direction histogram in the specified area range for each key point to generate a vector descriptor of a specified dimension to represent the gray feature of the image;

[0027] S216. Using the HOG feature extraction algorithm to calculate the gradient direction and size of each element in the image, dividing the image according to a specified block size, counting the gradient direction histogram of the pixels inside each block, and then aggregating a specified number of blocks into a cluster, normalizing the gradient direction histogram in each cluster, and concatenating them to form a one-dimensional vector image feature descriptor to represent the contour feature of the image.

[0028] Further, the step S3 specifically comprises the following steps:

[0029] S31. Constructing an elevator fire identification model, an elevator operation failure identification model and an elevator personnel safety accident identification model, training the elevator fire identification model through the elevator fire state data set, training the elevator operation failure identification model through the elevator operation identification data set, and training the elevator personnel safety accident identification model through the elevator personnel state data set;

[0030] S32. Integrate the trained elevator fire identification model, elevator operation fault identification model and elevator personnel safety accident identification model into an elevator fault identification model set, and deploy the elevator fault identification model set to the backend server.

[0031] Further, the step S31 includes the following specific steps:

[0032] S311. Construct the elevator fire identification model based on the convolutional neural network as follows:

[0033] Y fire = f fire ( W fire ⋅[ X fire , G fire ]+ b fire );

[0034] wherein, f fire represents the activation function of the model and the processing of the output layer, X fire represents the image data, G fire represents the gray feature, Y fire represents the fire occurrence identification value, W fire represents the weight of all layers, b fire represents the bias of all layers;[ X fire , G fire ] represents the vector of the image data and the gray feature spliced;

[0035] S312. Construct the elevator operation fault identification model based on the convolutional neural network as follows:

[0036] Y run =f run (W run ⋅ [X run ,C run ]+b run )

[0037] wherein, f run represents the activation function of the model and the processing of the output layer, image data X run and its contour feature C run , weight parameter W run , bias parameter b run , Y runrepresents the recognition value of the elevator operation failure;[X run , C run ] represents the vector spliced with the image data and the contour feature;

[0038] S313. The elevator personnel safety accident recognition model is constructed based on the convolutional neural network as follows:

[0039] Y person = f person (W person · [X person , C person ]+b person )

[0040] wherein f person represents the activation function and the processing of the output layer of the model, W person is the weight parameter, X person is the image data, b person is the bias parameter, and Y person represents the recognition value of the personnel safety accident;[X person , C person ] represents the vector spliced with the image data and the contour feature;

[0041] S314. The elevator fire state data set is divided into training data and test data, and the elevator fire recognition model is trained using the training data, the model parameters are updated through the back propagation algorithm, and after the loss is minimized or maximized iteration, the learning rate and the block size of the batch processing are adjusted, and then the accuracy and recall rate of the model are evaluated through the test data, and the final elevator fire recognition model is obtained;

[0042] S315. The elevator operation recognition data set is divided into training data and test data according to a set proportion, the elevator operation failure recognition model is trained using the training data, and the model parameters are updated through the back propagation algorithm, and the learning rate and the batch size are adjusted after the test set is verified, the model is evaluated in combination with the evaluation index, and the elevator operation failure recognition model is adjusted when the evaluation result does not meet the requirements, and the optimal elevator operation failure recognition model is obtained;

[0043] S316. The elevator personnel state data set is divided into training data and test data, and the elevator personnel safety accident recognition model is trained using the training data, the model parameters are updated through the back propagation algorithm, and after the loss is minimized or maximized iteration, the learning rate and the block size of the batch processing are adjusted, and then the accuracy and recall rate of the model are evaluated through the test data, and the final elevator personnel safety accident recognition model is obtained.

[0044] Further, the specific steps of step S4 are as follows:

[0045] S41. The back-end monitoring server inputs the image data and the gray scale features of the image in the real-time collected elevator operation data into an elevator fire identification model in the elevator fault identification model set, outputs a fire occurrence identification value, and issues a fire alarm when the fire occurrence identification value exceeds a set threshold value;

[0046] S42. The back-end monitoring server inputs the image data and the image contour features in the real-time collected elevator operation data into an elevator operation fault identification model in the elevator fault identification model set, outputs identification values of various types of elevator operation faults, and issues a fault alarm of a corresponding type when an identification value of a certain type of elevator fault exceeds a threshold value;

[0047] S43. The back-end monitoring server inputs the image data and the image contour features in the real-time collected elevator operation data into an elevator personnel safety accident identification model in the elevator fault identification model set, outputs identification values of various types of personnel safety accidents, and issues a personnel safety alarm of a corresponding type in combination with the collected elevator operation sound data when an identification value of a certain type of personnel safety accident exceeds a threshold value.

[0048] In a second aspect, the present application provides an elevator fault accident identification system based on intelligent image recognition, comprising:

[0049] An elevator operation data collection unit is configured to pre-assemble intelligent image sensors on the roof of the elevator car and the elevator shaft, respectively, and collect elevator operation data.

[0050] An elevator operation data preprocessing and feature extraction unit is configured to preprocess the image data collected by the intelligent image sensors and perform preliminary feature extraction, and construct a data set in combination with the image data and the preliminary extracted features in the elevator operation data and the known elevator operation data of fault types.

[0051] An elevator fault identification model construction and training unit is configured to construct a deep learning model and train the model using the data set to obtain an elevator fault identification model.

[0052] An elevator fault identification and alarm unit is configured to input the image data and the preliminary extracted features in the real-time collected elevator operation data into the elevator fault identification model, identify whether there is a fault in the elevator operation and the corresponding fault type, and issue an alarm according to the corresponding fault type.

[0053] In a third aspect, the present application provides an electronic device comprising a processor and a memory.

[0054] The memory is configured to store a computer program, and the processor is configured to call and run the computer program from the memory, so that the electronic device executes the method of the first aspect.

[0055] In a fourth aspect, the present application provides a storage medium,

[0056] The storage medium has stored therein instructions which, when executed on a computer, cause the computer to perform the method of the first aspect.

[0057] The present application has the following beneficial effects:

[0058] The elevator fault accident identification method, system, device and medium based on intelligent image recognition provided by the present application can identify elevator faults through intelligent image sensors combined with a deep learning model, without the need for manual inspection, thereby improving the fault identification efficiency and accuracy during the operation of the elevator, avoiding the problems of high cost and low identification accuracy of traditional sensors, and improving the safety and reliability of the elevator. The present application can realize remote monitoring and diagnosis of elevator faults and can quickly respond to and handle elevator faults.

[0059] As can be seen, compared with the prior art, the present application has outstanding substantial characteristics and significant progress, and the beneficial effects of its implementation are also obvious. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0061] Figure 1 is a flowchart of the elevator fault accident identification method based on intelligent image recognition of the present application.

[0062] Figure 2 is a schematic diagram of the elevator fault accident identification system based on intelligent image recognition of the present application. DETAILED DESCRIPTION

[0063] The elevator fault accident identification method based on intelligent image recognition can be applied to one or more electronic devices, which is an electronic device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions. The hardware thereof includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0064] The electronic device can be any electronic product capable of human-computer interaction with a user, for example, a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), an interactive Internet Protocol Television (IPTV), and the like. The network in which the electronic device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), and the like.

[0065] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor should fall within the protection scope of the present application.

[0066] Please refer to Figure 1 As shown in the figure, the elevator fault accident identification method based on intelligent image recognition in the embodiment includes the following steps:

[0067] S1. An intelligent image sensor is pre-installed on the roof of the elevator car and the elevator shaft, respectively, to collect elevator operation data;

[0068] S2. The image data in the elevator operation data collected by the intelligent image sensor is pre-processed, and preliminary feature extraction is performed. The image data and the preliminary extracted features in the elevator operation data are combined to construct a data set with known elevator operation data of fault types;

[0069] S3. A deep learning model is constructed and trained using the data set to obtain an elevator fault identification model;

[0070] S4. The image data and the preliminary extracted features in the real-time collected elevator operation data of the elevator are input into the elevator fault identification model to identify whether there is a fault in the elevator operation and the corresponding fault type, and an alarm is given according to the corresponding fault type.

[0071] In the embodiment, the specific steps of step S1 are as follows:

[0072] S11. An intelligent camera device serving as an intelligent image sensor is installed on the roof of the elevator car and the elevator shaft, respectively. The intelligent camera device is provided with a microphone module;

[0073] S12. Network connection between the intelligent camera device and a back-end monitoring server is established;

[0074] S13. The intelligent camera device collects elevator operation data in real time and sends it to the backend monitoring server; the elevator operation data includes image data and sound data;

[0075] The specific steps of step S2 are as follows:

[0076] S21. The backend monitoring server performs noise reduction processing, contrast enhancement processing and grayscale processing on the image data in the elevator operation data, and uses a feature extraction algorithm to perform preliminary feature extraction to obtain the grayscale features of the image and the contour features of the objects in the image;

[0077] The specific steps of step S21 are as follows:

[0078] S211. Determine the noise reduction accuracy, select a filter window according to the noise reduction accuracy, and use the filter window to sequentially locate the image data of the image;

[0079] S212. Sort the pixel values in the positioning area according to size, and take the pixel value of the middle value as the pixel value of the entire positioning area to complete the noise reduction processing;

[0080] S213. Calculate the histogram of the image data of the elevator operation data, determine the mapping function according to the histogram, and then map the brightness value of the image data according to the mapping function to complete the contrast enhancement processing;

[0081] S214. Convert the image data of the elevator operation data from the color RGB space to the YUV space, and perform weighted averaging on the red, green and blue components according to the preset weight components to obtain the grayscale value, and complete the grayscale processing;

[0082] S215. Use the SIFT feature extraction algorithm to construct a Gaussian kernel tower, detect the key points in the image by comparing the pixel differences at different scales, and calculate the gradient direction histogram in the specified area range for each key point to generate a vector descriptor of a specified dimension to represent the grayscale features of the image;

[0083] S216. Use the HOG feature extraction algorithm to calculate the gradient direction and size of each element in the image, divide the image according to the specified block size, and calculate the gradient direction histogram of the pixels inside each block, then aggregate a specified number of blocks into a cluster, and concatenate the gradient direction histograms in each cluster after normalization to form a one-dimensional vector image feature descriptor to represent the contour features of the image;

[0084] S22. Obtain elevator failure image data from a public data set, the elevator failure image data including elevator fire images and elevator personnel fall images;

[0085] S23. Mark the image data and grayscale features in the elevator operation data as elevator reverse fire data. After preprocessing the elevator fire images in the public dataset, extract the grayscale features and mark them as elevator forward fire data to generate an elevator fire status dataset.

[0086] S24. Based on the image data and the contour features of objects in the elevator operation data, label the elevator fault operation data and the elevator normal operation data to generate an elevator operation recognition dataset; for example, label the contour features of the elevator door opening or closing state, so as to identify the elevator door failure to close fault based on the contour features.

[0087] S25. Based on the image data and the contour features of objects in the elevator operation data, label normal images and images of people falling, and generate an elevator personnel status dataset; for example, label the contour features of different personnel states, and then identify safety accidents of people falling based on the contour features;

[0088] The specific steps of step S3 are as follows:

[0089] S31. Construct an elevator fire identification model, an elevator operation fault identification model, and an elevator personnel safety accident identification model. Train the elevator fire identification model using an elevator fire status dataset, train the elevator operation fault identification model using an elevator operation identification dataset, and train the elevator personnel safety accident identification model using an elevator personnel status dataset.

[0090] The specific steps of step S31 are as follows:

[0091] S311. An elevator fire identification model based on a convolutional neural network is constructed as follows:

[0092] Y fire = f fire ( W fire ⋅[ X fire , G fire ]+ b fire );

[0093] in, f fire This represents the activation function and output layer processing of the model. X fire Represents image data, G fire Representing grayscale features, Y fire Indicates the fire occurrence identification value. W firerepresents the weights of all layers, b fire represents the bias of all layers, X fire , G fire represents the vector spliced by the image data and the contour feature;

[0094] wherein all layers include convolutional layers, pooling layers, fully connected layers and output layers except for the input layer;

[0095] S312. The elevator operation fault identification model is constructed based on the convolutional neural network as follows:

[0096] Y run =f run (W run ⋅ [X run ,C run ]+b run )

[0097] wherein f run represents the activation function of the model and the processing of the output layer, X run is the image data and its contour feature C run , W run is the weight parameter, b run is the bias parameter, and Y run represents the identification value of the elevator operation fault; [X run , C run ] represents the vector spliced by the image data and the contour feature;

[0098] S313. The elevator personnel safety accident identification model is constructed based on the convolutional neural network as follows:

[0099] Y person =f person (W person ⋅ [X person ,C person ]+b person )

[0100] wherein f person represents the activation function of the model and the processing of the output layer, W person is the weight parameter, X person is the image data, b person is the bias parameter, and Y person represents the identification value of the personnel safety accident; [X person , C person ] represents the vector spliced by the image data and the contour feature;

[0101] S314. The elevator fire state data set is divided into training data and test data, and the training data is used to train the elevator fire identification model, the model parameters are updated through the back propagation algorithm, and after the minimum or maximum iteration of the loss, the learning rate and the block size of the batch processing are adjusted, and then the accuracy and recall rate of the model are evaluated through the test data, and the final elevator fire identification model is obtained;

[0102] S315. The elevator operation recognition data set is divided into training data and test data according to a set proportion, the training data is used to train the elevator operation fault identification model, and the model parameters are updated through the back propagation algorithm, and the learning rate and the batch size are adjusted after the test set is verified, and the model is evaluated in combination with the evaluation index, and the elevator operation fault identification model is adjusted when the evaluation result does not meet the requirements, and the optimal elevator operation fault identification model is obtained;

[0103] S316. The elevator personnel state data set is divided into training data and test data, and the training data is used to train the elevator personnel safety accident identification model, the model parameters are updated through the back propagation algorithm, and after the minimum or maximum iteration of the loss, the learning rate and the block size of the batch processing are adjusted, and then the accuracy and recall rate of the model are evaluated through the test data, and the final elevator personnel safety accident identification model is obtained;

[0104] S32. The elevator fire identification model, the elevator operation fault identification model and the elevator personnel safety accident identification model trained are integrated into an elevator fault identification model set, and the elevator fault identification model set is deployed to a backend server;

[0105] Step S4 includes the following specific steps:

[0106] S41. The backend monitoring server inputs the image data and the gray scale features of the image in the real-time collected elevator operation data into the elevator fire identification model in the elevator fault identification model set, outputs a fire occurrence identification value, and issues a fire alarm when the fire occurrence identification value exceeds a set threshold value;

[0107] S42. The backend monitoring server inputs the image data and the contour features of the image in the real-time collected elevator operation data into the elevator operation fault identification model in the elevator fault identification model set, outputs identification values of various types of elevator operation faults, and issues a fault alarm of a corresponding type when the identification value of a certain type of elevator fault exceeds a threshold value;

[0108] S43. The back-end monitoring server inputs the image data and image contour features in the real-time collected elevator operation data into the elevator personnel safety accident identification model in the elevator fault identification set, outputs the identification values of each type of personnel safety accident, and when the identification value of a certain type of personnel safety accident exceeds a threshold value, issues a personnel safety alarm of the corresponding type in combination with the collected elevator operation sound data.

[0109] It should be understood that the size of the serial number of each step in the above embodiments does not mean the execution order of the processes, and the execution order of the processes should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0110] The following is an embodiment of an elevator fault accident identification system based on intelligent image recognition provided by the embodiments of the present disclosure. The system and the above-mentioned elevator fault accident identification method based on intelligent image recognition belong to the same inventive concept. Details not described in the embodiment of the elevator fault accident identification system based on intelligent image recognition can be referred to the above-mentioned embodiment of the elevator fault accident identification method based on intelligent image recognition.

[0111] As shown in Figure 2 , the system comprises:

[0112] an elevator operation data collection unit configured to pre-assemble intelligent image sensors on the roof of the elevator car and the elevator pit, respectively, and collect elevator operation data;

[0113] an elevator operation data preprocessing and feature extraction unit configured to preprocess image data in the elevator operation data collected by the intelligent image sensors, and perform preliminary feature extraction, and construct a data set in combination with the image data in the elevator operation data and the preliminary extracted features and known elevator operation data of fault types;

[0114] an elevator fault identification model construction and training unit configured to construct a deep learning model and train the deep learning model using the data set to obtain an elevator fault identification model;

[0115] an elevator fault identification and alarm unit configured to input the image data and the preliminary extracted features in the elevator operation data collected in real time into the elevator fault identification model, identify whether there is a fault in the elevator operation and the corresponding fault type, and alarm according to the corresponding fault type.

[0116] The intelligent image recognition based elevator fault accident identification method provided by the present application is described in combination with the modules and algorithm steps of each example disclosed in the present application, and can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0117] Those skilled in the art can understand that each aspect of the intelligent image recognition based elevator fault accident identification method provided by the present application can be realized as a system, a method or a program product. Therefore, each aspect of the present disclosure can be embodied as a whole hardware embodiment, a whole software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" herein.

[0118] In the non-transitory computer readable storage medium storing the intelligent image recognition based elevator fault accident identification method, it can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0119] Although the present application has been described in detail by referring to the preferred embodiments thereof, it is not limited to the preferred embodiments. Various equivalent modifications or replacements can be made to the embodiments of the present application by those skilled in the art without departing from the spirit and essence of the present application, and such modifications or replacements should be within the scope of the present application. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, and such changes or replacements should be within the protection scope of the present application.

Claims

1. A method for identifying elevator malfunctions and accidents based on intelligent image recognition, characterized in that, Includes the following steps: S1. Smart image sensors are pre-installed on the top of the elevator car and in the elevator shaft to collect elevator operation data; S2. Preprocess the image data in the elevator operation data collected by the intelligent image sensor and perform preliminary feature extraction. Combine the image data in the elevator operation data and the preliminary extracted features with the elevator operation data of known fault types to construct a dataset. S3. Construct a deep learning model and train it using a dataset to obtain an elevator fault identification model; S4. Input the image data and preliminary extracted features from the real-time elevator operation data into the elevator fault identification model to identify whether there is a fault in the elevator operation and the corresponding fault type, and issue an alarm according to the corresponding fault type; The specific steps of step S2 are as follows: S21. The backend monitoring server performs noise reduction, contrast enhancement, and grayscale processing on the image data in the elevator operation data, and uses a feature extraction algorithm to perform preliminary feature extraction to obtain the grayscale features of the image and the contour features of the objects in the image. S22. Obtain elevator malfunction image data from a publicly available dataset, the elevator malfunction image data including elevator fire images and elevator people falling images; S23. Mark the image data and grayscale features in the elevator operation data as elevator reverse fire data. After preprocessing the elevator fire images in the public dataset, extract the grayscale features and mark them as elevator forward fire data to generate an elevator fire status dataset. S24. Based on the image data and the contour features of objects in the elevator operation data, label them as elevator fault operation data and elevator normal operation data, and generate an elevator operation recognition dataset; S25. Based on the image data and the contour features of objects in the elevator operation data, label normal images and images of people falling, and generate an elevator personnel status dataset; The specific steps of step S3 are as follows: S31. Construct an elevator fire identification model, an elevator operation fault identification model, and an elevator personnel safety accident identification model. Train the elevator fire identification model using an elevator fire status dataset, train the elevator operation fault identification model using an elevator operation identification dataset, and train the elevator personnel safety accident identification model using an elevator personnel status dataset. S32. Integrate the trained elevator fire identification model, elevator operation fault identification model, and elevator personnel safety accident identification model into an elevator fault identification model set, and deploy the elevator fault identification model set to the backend server.

2. The elevator fault accident identification method based on intelligent image recognition as described in claim 1, characterized in that, The specific steps of step S1 are as follows: S11. An intelligent camera device serving as an intelligent image sensor is installed on the top of the elevator car and in the elevator shaft, respectively. The intelligent camera device is equipped with a microphone module. S12. Establish a network connection between the smart camera device and the backend monitoring server; S13. The intelligent camera device collects elevator operation data in real time and sends it to the backend monitoring server; the elevator operation data includes image data and sound data.

3. The elevator fault accident identification method based on intelligent image recognition as described in claim 1, characterized in that, The specific steps of step S21 are as follows: S211. Determine the noise reduction accuracy, select the filtering window according to the noise reduction accuracy, and use the filtering window to sequentially locate the image data of the image process; S212. Sort the pixel values ​​within the positioning area by size, and use the middle pixel value as the pixel value of the entire positioning area to complete the noise reduction process; S213. Calculate the histogram of the image data of elevator operation data, determine the mapping function based on the histogram, and then map the brightness values ​​of the image data according to the mapping function to complete the contrast enhancement process; S214. Convert the image data of elevator operation data from the color RGB space to the YUV space, and perform a weighted average of the red, green and blue components according to the preset weight components to obtain the grayscale value, thus completing the grayscale processing. S215. Construct a Gaussian pyramid using the SIFT feature extraction algorithm, detect key points in the image by comparing pixel differences at different scales, calculate the gradient direction histogram within a set region for each key point, and generate a vector descriptor of a specified dimension to represent the grayscale features of the image. S216. Use the HOG feature extraction algorithm to calculate the gradient direction and magnitude of each element in the image, divide the image into blocks of a set size, count the gradient direction histogram of pixels within each block, then aggregate a set number of blocks into clusters, normalize the gradient direction histograms within each cluster, and concatenate them to form a one-dimensional vector image feature descriptor to represent the contour features of the image.

4. The elevator fault accident identification method based on intelligent image recognition as described in claim 1, characterized in that, The specific steps of step S31 are as follows: S311. An elevator fire identification model based on a convolutional neural network is constructed as follows: Yfire=ffire(Wfire•[Xfire, Gfire]+bfire); Where ffire represents the activation function and output layer processing of the model, Xfire represents the image data, Gfire represents the grayscale features, Yfire represents the fire occurrence identification value, Wfire represents the weights of all layers, and bfire represents the biases of all layers; [Xfire, Gfire] represents the vector concatenated with the image data and grayscale features. S312. An elevator operation fault identification model based on a convolutional neural network is constructed as follows: Yrun=frun(Wrun·[Xrun, Crun]+brun) Where frun represents the activation function and output layer processing of the model, image data Xrun and its contour features Crun, weight parameter Wrun, bias parameter brun, and Yrun represents the identification value of elevator operation fault; [Xrun, Crun] represents the vector concatenated with image data and contour features. S313. An elevator occupant safety accident identification model based on a convolutional neural network is constructed as follows: Yperson=fperson(Wperson•[Xperson, Cperson]+bperson) Where fperson represents the activation function and output layer processing of the model, Wperson is the weight parameter, Xperson is the image data, bperson is the bias parameter, Yperson represents the identification value of personnel safety accidents; [Xperson, Cperson] represents the vector concatenated from the image data and contour features. S314. Divide the elevator fire status dataset into training data and test data, use the training data to train the elevator fire identification model, update the model parameters through the backpropagation algorithm, and adjust the learning rate and batch block size after the minimum or maximum loss iteration. Then evaluate the accuracy and recall of the model through the test data to obtain the final elevator fire identification model. S315. Divide the elevator operation recognition dataset into training data and test data according to a set ratio. Use the training data to train the elevator operation fault recognition model and update the model parameters through the backpropagation algorithm. Use the test set to verify the model, adjust the learning rate and batch size, and then evaluate the model in combination with the evaluation index. If the evaluation result does not meet the requirements, adjust the elevator operation fault recognition model to obtain the optimal elevator operation fault recognition model. S316. Divide the elevator personnel status dataset into training data and test data, and use the training data to train the elevator personnel safety accident identification model. Update the model parameters through the backpropagation algorithm, and adjust the learning rate and batch size after the minimum or maximum loss iteration. Then evaluate the accuracy and recall of the model through the test data to obtain the final elevator personnel safety accident identification model.

5. The elevator fault accident identification method based on intelligent image recognition as described in claim 1, characterized in that, The specific steps of step S4 are as follows: S41. The backend monitoring server inputs the image data and grayscale features of the real-time collected elevator operation data into the elevator fire identification model in the elevator fault identification model set, outputs the fire occurrence identification value, and issues a fire alarm when the fire occurrence identification value exceeds the set threshold. S42. The backend monitoring server inputs the image data and image contour features from the real-time collected elevator operation data into the elevator operation fault identification model in the elevator fault identification model set, outputs the identification value of each type of elevator operation fault, and issues the corresponding type of fault alarm when the identification value of a certain type of elevator fault exceeds the threshold. S43. The backend monitoring server inputs the image data and image contour features from the real-time collected elevator operation data into the elevator personnel safety accident identification model in the elevator fault identification set, outputs the identification value of each type of personnel safety accident, and when the identification value of a certain type of personnel safety accident exceeds the threshold, it issues the corresponding type of personnel safety alarm in combination with the collected elevator operation sound data.

6. An elevator malfunction and accident identification system based on intelligent image recognition, characterized in that, include: The elevator operation data acquisition unit is used to pre-install intelligent image sensors on the top of the elevator car and in the elevator shaft to collect elevator operation data. The elevator operation data preprocessing and feature extraction unit is used to preprocess the image data in the elevator operation data collected by the intelligent image sensor and perform preliminary feature extraction. The data set is constructed by combining the image data in the elevator operation data and the preliminary extracted features with the elevator operation data of known fault types. The elevator fault identification model construction and training unit is used to build a deep learning model and train it using a dataset to obtain the elevator fault identification model. The elevator fault identification and alarm unit is used to input the image data and preliminary extracted features from the real-time elevator operation data into the elevator fault identification model to identify whether there is a fault in the elevator operation and the corresponding fault type, and to issue an alarm according to the corresponding fault type. The elevator operation data preprocessing and feature extraction unit includes: The backend monitoring server performs noise reduction, contrast enhancement, and grayscale processing on the image data in the elevator operation data, and uses a feature extraction algorithm to perform preliminary feature extraction to obtain the grayscale features of the image and the contour features of the objects in the image. Elevator malfunction image data was obtained from a publicly available dataset, including elevator fire images and images of people falling in elevators. Image data and grayscale features in elevator operation data are labeled as elevator reverse fire data. After preprocessing elevator fire images in the public dataset, grayscale features are extracted and labeled as elevator forward fire data to generate elevator fire status dataset. Based on the image data and the contour features of objects in the elevator operation data, the elevator operation identification dataset is generated by labeling them as elevator fault operation data and elevator normal operation data. Based on the image data and the contour features of objects in the elevator operation data, normal images of people and images of people falling are labeled to generate an elevator personnel status dataset; The elevator fault identification model construction and training unit includes: An elevator fire identification model, an elevator operation fault identification model, and an elevator personnel safety accident identification model are constructed. The elevator fire identification model is trained using an elevator fire status dataset, the elevator operation fault identification model is trained using an elevator operation identification dataset, and the elevator personnel safety accident identification model is trained using an elevator personnel status dataset. The trained elevator fire identification model, elevator operation fault identification model, and elevator personnel safety accident identification model are integrated into an elevator fault identification model set, and the elevator fault identification model set is deployed to the backend server.

7. An electronic device, characterized in that, Including processor and memory; The memory is used to store a computer program, and the processor is used to retrieve and run the computer program from the memory, causing the electronic device to perform the method described in any one of claims 1-5.

8. A storage medium, characterized in that, The storage medium stores instructions that, when run on a computer, cause the computer to perform the method described in any one of claims 1-5.

Images