An elevator cloud platform management system based on image recognition

Abstract

The present invention discloses an elevator cloud platform management system based on image recognition, comprising an image acquisition module, elevator safety segmentation modules, elevator safety microservice processing modules and an integrated management module, wherein: each microservice module independently processes its own image recognition business based on a neural network of a YOLOv7 network model; the integrated management module performs secondary recognition of image features for objects generated by different microservice modules based on a deep learning model, and performs comprehensive dispatch management of elevator safety according to the secondary recognition results; the present invention accurately and quickly locates safety problems existing or occurring in elevators based on the correlation of sound features in image recognition, comprehensively manages secondary recognition of neural network model output results corresponding to different microservice module tasks through a deep learning model, realizes load balancing of cloud platforms for different microservice image processing tasks, and further accurately and quickly identifies and solves elevator safety problems.

Description

Technical Field

[0001] The present invention belongs to the field of elevator cloud platform management based on image recognition, and specifically refers to an elevator cloud platform management system based on image recognition. Background Art

[0002] The elevator management system is a modern elevator monitoring solution that integrates technologies such as the Internet of Things and big data analysis. This system aims to improve the operating efficiency, safety and maintenance convenience of elevators, while enhancing the passenger experience.

[0003] At present, people can bring electric vehicles and other equipment into elevators at will. However, due to the many uncertain factors of people and electric vehicles in elevators, and the long running time of elevators will also cause uncertain vibrations or abnormal speeds, there are safety hazards in the elevators. The elevator management system lacks control over the safety in the elevator, resulting in technical problems such as low safety in the elevator. In addition, the current elevator management system based on image recognition cannot accurately control the safety hazards in the elevator and cannot quickly deal with safety issues. Summary of the invention

[0004] In order to solve the problems in the above-mentioned prior art such as low safety in elevators, inability to accurately control safety hazards in elevators, and inability to quickly handle safety issues, the present invention proposes an elevator cloud platform management system based on image recognition to improve the above-mentioned problems.

[0005] The specific application is as follows:

[0006] An elevator cloud platform management system based on image recognition, comprising an image acquisition module, an electric vehicle prohibited entry segmentation module, an elevator hand clamping segmentation module, an elevator vibration abnormality segmentation module, an elevator speed abnormality segmentation module, an electric vehicle prohibited entry elevator management microservice module, an elevator hand clamping management microservice module, an elevator vibration abnormality management microservice module, an elevator speed abnormality management microservice module and a comprehensive management module;

[0007] The image acquisition module is used to acquire the video frame image in the target elevator and upload the video frame image in the elevator to the cloud platform, wherein the cloud platform is embedded with a target segmentation model, and the target segmentation model is used to segment the objects in the video frame image in the elevator. The cloud platform is based on the development architecture of Nodejs microservices;

[0008] The electric vehicle prohibited entry segmentation module is used to segment the in-elevator video frame image into an electric vehicle object according to the target segmentation model embedded in the cloud platform, and determine the electric vehicle object and the first identification object, wherein the first identification object is the sound object associated with the electric vehicle object;

[0009] The elevator hand-caught object segmentation module is used to segment the elevator hand-caught object on the elevator video frame image according to the target segmentation model embedded in the cloud platform, and determine the elevator hand-caught object and the second identification object, wherein the second identification object is the sound object associated with the elevator hand-caught object;

[0010] The elevator vibration abnormality segmentation module is used to segment the elevator vibration abnormality object on the elevator video frame according to the target segmentation model embedded in the cloud platform, and determine the elevator vibration abnormality object and the third identification object, wherein the third identification object is the sound object associated with the elevator vibration abnormality object;

[0011] The elevator speed abnormality segmentation module is used to segment the elevator speed abnormality object on the elevator video frame according to the target segmentation model embedded in the cloud platform, and determine the elevator speed abnormality object and the fourth identification object, wherein the fourth identification object is the sound object associated with the elevator speed abnormality object;

[0012] The electric vehicle prohibited from entering the elevator management microservice module is used to perform visual training on electric vehicle objects and related sound objects. The trained model is an electric vehicle prohibited from entering the elevator model, which is used for electric vehicle prohibited from entering the elevator management. The training adopts a neural network model for training;

[0013] The elevator hand-entrapment management microservice module is used to perform visual training on elevator hand-entrapment objects and associated sound objects. The trained model is an elevator hand-entrapment model, which is used for elevator hand-entrapment management. The training adopts a neural network model for training;

[0014] The elevator vibration abnormality management microservice module is used to perform visual training on the elevator vibration abnormality object and the associated sound object. The trained model is the elevator vibration abnormality model, which is used for elevator vibration abnormality management. The training adopts a neural network model for training;

[0015] The elevator speed abnormality management microservice module is used to perform visual training on the elevator speed abnormality object and the associated sound object. The trained model is used for elevator speed abnormality management. The training is performed using a neural network model.

[0016] The comprehensive management module performs secondary recognition of image features for objects generated by different microservice modules based on a deep learning model, and performs comprehensive dispatch management of elevator safety according to the secondary recognition results;

[0017] The different microservice modules correspond to different neural network models, a total of 4 neural network models and one deep learning model. The models will be deployed in a distributed manner on the cloud platform. The cloud platform controls the number of models through load balancing. The cloud platform uses asynchronous communication to uniformly manage the training of each model and real-time data testing.

[0018] Furthermore, the target segmentation model is used to segment the objects in the ladder video frame image, and the specific implementation process is as follows:

[0019] S1: Edit the collected historical surveillance videos to generate elevator operation clips;

[0020] S2: Classify the elevator operation clips in step S1, and mark the target objects of the video frame to form object data set labels, where the objects include electric vehicle objects, elevator hand clamp objects, elevator vibration abnormal objects, and elevator speed abnormal objects;

[0021] S3: Performing a fast Fourier transform on the image frame corresponding to the target object to obtain the spectrum value of each corresponding speech frame, generating a spectrogram using the spectrum value of each speech frame, and extracting features from the spectrogram using the Inception convolution structure in the image recognition algorithm to obtain a sound feature label set associated with the target object;

[0022] S4: filtering the sound feature tags with a correlation less than a threshold value according to the correlation between the object and the sound feature tags associated with the object, thereby forming a tag of the target object and a tag of the associated sound;

[0023] S5: construct training data according to the video frame images with object labels and sound feature labels associated with the objects, and use the training data to train the target segmentation model to obtain a trained target segmentation model.

[0024] Furthermore, the specific implementation process of the correlation degree in step S4 is:

[0025] L1: According to each target object label and the degree of sound association in each video frame, obtain the feature parameters of each target object label in each video frame;

[0026] L2: The target object label in the video frame is taken as the first label, and the label of the same target object in the adjacent next video frame is taken as the second label. The absolute value of the difference between the feature parameters of the sound label corresponding to the first label and the sound label corresponding to the second label is taken as the denominator;

[0027] L3: There are multiple video frames in the video frame sequence, and a video frame with the target object label in step L2. If the video frame also has a sound label corresponding to the target object label, the feature parameters of the target object label and the corresponding sound label are subtracted, the absolute value of the difference is taken, and the absolute value of all differences is taken to construct the numerator;

[0028] L4: Subtract the preset positive comparison parameter from the ratio of the denominator to the numerator and perform positive correlation mapping to obtain the correlation between the target object in each video frame and the sound label in the video frame.

[0029] Furthermore, the SIFT algorithm is used for the segmentation of electric vehicle objects and elevator hand-pinching objects to obtain whether there is an electric vehicle object or an elevator hand-pinching object in each video frame. If there is an electric vehicle object or an elevator hand-pinching object, the SIFT algorithm is used to perform secondary confirmation of the electric vehicle object or the elevator hand-pinching object between adjacent frames.

[0030] Furthermore, the elevator speed abnormal object segmentation and elevator vibration abnormality segmentation adopt a feature matching method of the SIFT algorithm combined with the streamer method to obtain whether there is an elevator speed abnormal object or an elevator speed abnormal object in each video frame.

[0031] Furthermore, the specific implementation process of the electric vehicle prohibition elevator management microservice module is as follows:

[0032] M1: Use the images of the electric vehicle object and the first identification object as training data for the electric vehicle prohibition model;

[0033] M2: The condition for completing the training of the electric vehicle prohibition elevator model is that the correlation degree of the electric vehicle object or the sound associated with the electric vehicle object can meet the set threshold. The training is terminated as long as one of the two conditions is met. At the same time, the accuracy of the training model recognition must reach 90%;

[0034] M3: A trained electric vehicle prohibition elevator model, which is used to determine whether an electric vehicle enters the elevator. If so, a warning instruction is sent to the elevator control module to keep the elevator door open. The elevator control module is used to control the elevator switch to close.

[0035] M4: The camera in the elevator takes pictures in real time and transmits the pictures to the electric vehicle prohibition elevator management microservice module. The electric vehicle prohibition elevator management microservice fails to identify the electric vehicle and sends a warning cancellation instruction to the elevator control module to close the elevator door normally.

[0036] The specific implementation process of the elevator hand clamp management microservice module is as follows:

[0037] K1: Use the images of the elevator hand-caught object and the second identification object as training data for the elevator hand-caught model;

[0038] K2: The condition for completing the training of the elevator hand-pinching model is that the elevator hand-pinching object or the sound associated with the elevator hand-pinching object can be detected and the correlation degree meets the set threshold. As long as one of the two conditions is met, the training is terminated. At the same time, the accuracy of the training model recognition must reach 90%;

[0039] K3: A trained elevator hand-pinching model, which is used to identify the elevator hand-pinching object. If it exists, a warning command is sent to the elevator control module to keep the elevator door open until the hand-pinching object disappears. Then, a warning release command is sent to the elevator control module to close the elevator door normally.

[0040] The specific implementation process of the elevator vibration abnormality management microservice module is as follows:

[0041] R1: Use the images of the elevator vibration abnormal object and the third identification object as training data for the elevator vibration abnormality model;

[0042] R2: The condition for completing the elevator vibration abnormality training is that the elevator vibration abnormality object can be detected or the correlation of the sound associated with the elevator vibration abnormality object meets the set threshold. As long as one of the two conditions is met, the training is terminated, and the accuracy of the training model recognition must reach 90%;

[0043] R3: The trained elevator vibration anomaly model is used to determine the elevator vibration anomaly. If it exists, a warning instruction is sent to the operation and maintenance personnel, and the first safety hazard coefficient is generated;

[0044] The specific implementation process of the elevator speed abnormality management microservice module is as follows:

[0045] P1: using the images of the elevator speed abnormality object and the fourth identification object as training data for the elevator speed abnormality model;

[0046] P2: The condition for completing the elevator speed abnormality training is that the elevator speed abnormality object can be detected or the correlation of the sound associated with the elevator speed abnormality object meets the set threshold. As long as one of the two conditions is met, the training is terminated, and the accuracy of the training model recognition must reach 90%;

[0047] P3: The trained elevator speed abnormality model is used to judge the elevator speed abnormality. If it exists, a warning instruction is sent to the operation and maintenance personnel, and a second safety hazard coefficient is generated.

[0048] Furthermore, the specific implementation process of the secondary identification of the comprehensive management module is:

[0049] N1: Build a deep learning model, and the training data is the output feature object data set of the electric vehicle prohibition elevator management microservice module, the elevator hand clamp management microservice module, the elevator vibration abnormality management microservice module, and the elevator speed abnormality management microservice module;

[0050] N2: The deep learning model has a shared layer that is a public data set, and the other four layers are set, including an electric vehicle prohibition elevator management microservice layer, an elevator hand clamp management microservice layer, an elevator vibration abnormality management microservice layer, and an elevator speed abnormality management microservice layer;

[0051] N3: Perform multi-task learning training on the four layers, and the trained deep learning model is used for actual deployment;

[0052] N4: The dynamic tracking method adopted for the first safety hazard coefficient triggered by the elevator vibration abnormality management microservice layer and the second safety hazard coefficient triggered by the elevator speed abnormality management microservice layer is as follows:

[0053] N41: Establish the sequence N(b,a) corresponding to the first safety hazard coefficient i ), where a i The sound correlation value corresponding to the abnormal elevator vibration object in different image frames is constructed to construct y=k*f(a i ), k is the slope of the linear function. When the slope does not change within T time, continue tracking. When the absolute value of the change in k is greater than the threshold, stop tracking, stop the elevator operation, and send an elevator stop command to the operation and maintenance personnel for operation and maintenance;

[0054] N42: Establish the sequence N(b,a) corresponding to the second safety hazard coefficient i ), where a i The sound correlation degree values ​​corresponding to the abnormal elevator speed object in different image frames are constructed to construct y=k*f(a i ), k is the slope of the linear function. When the slope does not change within T time, continue tracking. When the absolute value of the change in k is greater than the threshold, stop tracking, stop the elevator operation, and send an elevator stop command to the operation and maintenance personnel for operation and maintenance;

[0055] N5: A deep learning model is deployed for N elevators. The cloud platform designs a visual interface and pre-configures the threshold of the maximum number of elevators to be monitored by a deep learning model. If the system finds that the maximum detection capability value of a deep learning model does not reach the threshold, it decreases by 1 and adjusts the number of elevators monitored by a deep learning model until the system load is balanced.

[0056] N6: For a deep learning model, deploy M corresponding neural networks to correspond to it, and pre-configure the threshold of the maximum number of control neural networks for a deep learning model. If the system finds that the maximum detection capability value of a deep learning model does not reach the threshold, it decreases by 1 in sequence and adjusts the number of neural networks corresponding to a deep learning model until the system load is balanced.

[0057] Furthermore, the neural network model is obtained by the following method: on the basis of the YOLOv7 network model, the number of classifications of the model is set to 4, corresponding to the electric vehicle prohibition elevator management microservice module, the elevator hand clamping management microservice module, the elevator vibration abnormality management microservice module and the elevator speed abnormality management microservice module respectively, the feature extraction network is set to the ShuffleNet network, and the enhanced feature network FPN is set to the depthwise separable convolution.

[0058] A computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, is used to implement the functions of any one of the elevator cloud platform management systems based on image recognition.

[0059] The beneficial effects of an elevator cloud platform management system based on image recognition of the present invention are as follows:

[0060] The present invention collects video frames in the target elevator and uploads the video frames to the cloud platform. The cloud platform is embedded with a target segmentation model. The target segmentation model is used to segment objects in the video frames in the elevator. The segmented objects include electric vehicle objects and associated sound objects, elevator hand clamping objects and associated sound objects, elevator vibration abnormal objects and associated sound objects, and elevator speed abnormal objects and associated sound objects. The sound objects are filtered by correlation. Each microservice module independently processes the image recognition business of segmenting image objects and associated sound objects based on the neural network of the YOLOv7 network model. The comprehensive management module is based on a deep learning model. Secondary recognition of image features is performed on objects generated by different microservice modules, and comprehensive dispatch management of elevator safety is performed based on the secondary recognition results. The present invention accurately and quickly locates safety issues that exist or occur in elevators based on the correlation of sound features in image recognition, and comprehensively manages the secondary recognition of neural network model output results corresponding to different microservice module tasks through a deep learning model. At the same time, load balancing of different microservice image processing tasks on the cloud platform is achieved, and elevator safety issues are further accurately and quickly identified and resolved. The powerful processing capabilities of the cloud platform are used to deploy multiple neural network models and deep learning models to quickly and accurately identify safety issues in the services of each module of the elevator. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1This is a system architecture diagram of an elevator cloud platform management system based on image recognition in the present invention.

[0062] Among them, 1-image acquisition module, 2-electric vehicle no-entry segmentation module, 3-elevator hand-pinching segmentation module, 4-elevator vibration abnormal segmentation module, 5-elevator speed abnormal segmentation module, 6-electric vehicle no-entry elevator management microservice module, 7-elevator hand-pinching management microservice module, 8-elevator vibration abnormal management microservice module, 9-elevator speed abnormal management microservice module, 10-comprehensive management module. DETAILED DESCRIPTION

[0063] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Example 1

[0065] First, the technical terms involved in the embodiments of the present application are explained.

[0066] (1) Nodejs microservices

[0067] Nodejs microservice is a microservice architecture implementation based on the Nodejs platform. Microservice architecture is an architectural style that decomposes an application into a set of small, loosely coupled services. Each service is built around a specific business function and can be independently deployed, expanded, and updated.

[0068] Figure 1 This is a system architecture diagram of an elevator cloud platform management system based on image recognition in the present invention.

[0069] The system includes an image acquisition module 1, an electric vehicle prohibited entry segmentation module 2, an elevator hand clamp segmentation module 3, an elevator vibration abnormal segmentation module 4, an elevator speed abnormal segmentation module 5, an electric vehicle prohibited entry elevator management microservice module 6, an elevator hand clamp management microservice module 7, an elevator vibration abnormal management microservice module 8, an elevator speed abnormal management microservice module 9 and a comprehensive management module 10;

[0070] The image acquisition module 1 is used to acquire a video frame image in a target elevator and upload the video frame image in the elevator to a cloud platform, wherein a target segmentation model is embedded in the cloud platform, and the target segmentation model is used to segment objects in the video frame image in the elevator;

[0071] The electric vehicle prohibited entry segmentation module 2 is used to segment the in-elevator video frame image into an electric vehicle object according to the target segmentation model embedded in the cloud platform, and determine the electric vehicle object and the first identification object, wherein the first identification object is the sound object associated with the electric vehicle object;

[0072] The elevator hand-caught object segmentation module 3 is used to segment the elevator hand-caught object on the elevator video frame image according to the target segmentation model embedded in the cloud platform, and determine the elevator hand-caught object and the second identification object, wherein the second identification object is the sound object associated with the elevator hand-caught object;

[0073] The elevator vibration abnormality segmentation module 4 is used to segment the elevator vibration abnormality object on the elevator video frame according to the target segmentation model embedded in the cloud platform, and determine the elevator vibration abnormality object and the third identification object, wherein the third identification object is the sound object associated with the elevator vibration abnormality object;

[0074] The elevator speed abnormality segmentation module 5 is used to segment the elevator speed abnormality object on the elevator video frame according to the target segmentation model embedded in the cloud platform, and determine the elevator speed abnormality object and the fourth identification object, wherein the fourth identification object is the sound object associated with the elevator speed abnormality object;

[0075] The electric vehicle prohibited from entering the elevator management microservice module 6 is used to perform visual training on electric vehicle objects and related sound objects. The trained model is an electric vehicle prohibited from entering the elevator model, which is used for electric vehicle prohibited from entering the elevator management. The training adopts a neural network model for training;

[0076] The elevator hand-entrapment management microservice module 7 is used to perform visual training on elevator hand-entrapment objects and associated sound objects. The trained model is an elevator hand-entrapment model, which is used for elevator hand-entrapment management. The training adopts a neural network model for training;

[0077] The elevator vibration abnormality management microservice module 8 is used to perform visual training on the elevator vibration abnormality object and the associated sound object. The trained model is the elevator vibration abnormality model, which is used for elevator vibration abnormality management. The training adopts a neural network model for training;

[0078] The elevator speed abnormality management microservice module 9 is used to perform visual training on the elevator speed abnormality object and the associated sound object. The trained model is used for elevator speed abnormality management. The training is performed using a neural network model.

[0079] The comprehensive management module 10 performs secondary recognition of image features for objects generated by different microservice modules based on a deep learning model, and performs comprehensive dispatch management of elevator safety according to the secondary recognition results;

[0080] The different microservice modules correspond to different neural network models, a total of 4 neural network models and one deep learning model. The models will be deployed in a distributed manner on the cloud platform. The cloud platform controls the number of models through load balancing. The cloud platform uses asynchronous communication to uniformly manage the training of each model and real-time data testing.

[0081] It should be noted that since the various microservice modules of the elevator cloud platform management system adopt the above-mentioned process to perform elevator safety management, electric vehicles will be detected once they enter the elevator. In some cases, electric vehicles are blocked by objects, but the electric vehicle prohibition elevator management microservice module can quickly locate that it is an electric vehicle based on the sound characteristics associated with the electric vehicle. The system will automatically prompt the owner to push the electric vehicle out until the electric vehicle is no longer in the elevator and the elevator resumes normal operation. Other elevator hand pinching management microservice modules, elevator vibration abnormality management microservice modules and elevator speed abnormality management microservice modules can all combine image recognition technology and the sound correlation in image recognition technology to accurately identify various abnormal problems of the elevator. There is no need to add detection equipment. Just add sound correlation recognition on the basis of image recognition to quickly locate various abnormal problems of the elevator.

[0082] Embodiment 2, this embodiment is based on the above embodiment,

[0083] The target segmentation model is used to segment the objects in the ladder video frame image. The specific implementation process is:

[0084] S1: Edit the collected historical surveillance videos to generate elevator operation clips;

[0085] S2: Classify the elevator operation clips in step S1, and mark the target objects of the video frame to form object data set labels, where the objects include electric vehicle objects, elevator hand clamp objects, elevator vibration abnormal objects, and elevator speed abnormal objects;

[0086] S3: Performing a fast Fourier transform on the image frame corresponding to the target object to obtain the spectrum value of each corresponding speech frame, generating a spectrogram using the spectrum value of each speech frame, and extracting features from the spectrogram using the Inception convolution structure in the image recognition algorithm to obtain a sound feature label set associated with the target object;

[0087] S4: filtering the sound feature tags with a correlation less than a threshold value according to the correlation between the object and the sound feature tags associated with the object, thereby forming a tag of the target object and a tag of the associated sound;

[0088] S5: construct training data according to the video frame images with object labels and sound feature labels associated with the objects, and use the training data to train the target segmentation model to obtain a trained target segmentation model.

[0089] Furthermore, the specific implementation process of the correlation degree in step S4 is:

[0090] L1: According to each target object label and the degree of sound association in each video frame, obtain the feature parameters of each target object label in each video frame;

[0091] L2: The target object label in the video frame is taken as the first label, and the label of the same target object in the adjacent next video frame is taken as the second label. The absolute value of the difference between the feature parameters of the sound label corresponding to the first label and the sound label corresponding to the second label is taken as the denominator;

[0092] L3: There are multiple video frames in the video frame sequence, and a video frame with the target object label in step L2. If the video frame also has a sound label corresponding to the target object label, the feature parameters of the target object label and the corresponding sound label are subtracted, the absolute value of the difference is taken, and the absolute value of all differences is taken to construct the numerator;

[0093] L4: Subtract the preset positive comparison parameter from the ratio of the denominator to the numerator and perform positive correlation mapping to obtain the correlation between the target object in each video frame and the sound label in the video frame.

[0094] Furthermore, the SIFT algorithm is used for the segmentation of electric vehicle objects and elevator hand-pinching objects to obtain whether there is an electric vehicle object or an elevator hand-pinching object in each video frame. If there is an electric vehicle object or an elevator hand-pinching object, the SIFT algorithm is used to perform secondary confirmation of the electric vehicle object or the elevator hand-pinching object between adjacent frames.

[0095] Furthermore, the elevator speed abnormal object segmentation and elevator vibration abnormality segmentation adopt a feature matching method of the SIFT algorithm combined with the streamer method to obtain whether there is an elevator speed abnormal object or an elevator speed abnormal object in each video frame.

[0096] Furthermore, the specific implementation process of the electric vehicle prohibition elevator management microservice module 6 is as follows:

[0097] M1: Use the images of the electric vehicle object and the first identification object as training data for the electric vehicle prohibition model;

[0098] M2: The condition for completing the training of the electric vehicle prohibition elevator model is that the correlation degree of the electric vehicle object or the sound associated with the electric vehicle object can meet the set threshold. The training is terminated as long as one of the two conditions is met. At the same time, the accuracy of the training model recognition must reach 90%;

[0099] M3: A trained electric vehicle prohibition elevator model, which is used to determine whether an electric vehicle enters the elevator. If so, a warning instruction is sent to the elevator control module to keep the elevator door open. The elevator control module is used to control the elevator switch to close.

[0100] M4: The camera in the elevator takes pictures in real time and transmits the pictures to the electric vehicle prohibition elevator management microservice module. The electric vehicle prohibition elevator management microservice fails to identify the electric vehicle and sends a warning cancellation instruction to the elevator control module to close the elevator door normally.

[0101] The specific implementation process of the elevator hand clamping management microservice module 7 is as follows:

[0102] K1: Use the images of the elevator hand-caught object and the second identification object as training data for the elevator hand-caught model;

[0103] K2: The condition for completing the training of the elevator hand-pinching model is that the elevator hand-pinching object or the sound associated with the elevator hand-pinching object can be detected and the correlation degree meets the set threshold. As long as one of the two conditions is met, the training is terminated. At the same time, the accuracy of the training model recognition must reach 90%;

[0104] K3: A trained elevator hand-pinching model, which is used to identify the elevator hand-pinching object. If it exists, a warning command is sent to the elevator control module to keep the elevator door open until the hand-pinching object disappears. Then, a warning release command is sent to the elevator control module to close the elevator door normally.

[0105] The specific implementation process of the elevator vibration abnormality management microservice module 8 is as follows:

[0106] R1: Use the images of the elevator vibration abnormal object and the third identification object as training data for the elevator vibration abnormality model;

[0107] R2: The condition for completing the elevator vibration abnormality training is that the elevator vibration abnormality object can be detected or the correlation of the sound associated with the elevator vibration abnormality object meets the set threshold. As long as one of the two conditions is met, the training is terminated, and the accuracy of the training model recognition must reach 90%;

[0108] R3: The trained elevator vibration anomaly model is used to determine the elevator vibration anomaly. If it exists, a warning instruction is sent to the operation and maintenance personnel, and the first safety hazard coefficient is generated;

[0109] The specific implementation process of the elevator speed abnormality management microservice module 9 is as follows:

[0110] P1: using the images of the elevator speed abnormality object and the fourth identification object as training data for the elevator speed abnormality model;

[0111] P2: The condition for completing the elevator speed abnormality training is that the elevator speed abnormality object can be detected or the correlation of the sound associated with the elevator speed abnormality object meets the set threshold. As long as one of the two conditions is met, the training is terminated, and the accuracy of the training model recognition must reach 90%;

[0112] P3: The trained elevator speed abnormality model is used to judge the elevator speed abnormality. If it exists, a warning instruction is sent to the operation and maintenance personnel, and a second safety hazard coefficient is generated.

[0113] Furthermore, the specific implementation process of the secondary identification of the comprehensive management module 10 is as follows:

[0114] N1: Build a deep learning model, and the training data is the output feature object data set of the electric vehicle prohibition elevator management microservice module, the elevator hand clamp management microservice module, the elevator vibration abnormality management microservice module, and the elevator speed abnormality management microservice module;

[0115] N2: The deep learning model has a shared layer that is a public data set, and the other four layers are set, including an electric vehicle prohibition elevator management microservice layer, an elevator hand clamp management microservice layer, an elevator vibration abnormality management microservice layer, and an elevator speed abnormality management microservice layer;

[0116] N3: Perform multi-task learning training on the four layers, and the trained deep learning model is used for actual deployment;

[0117] N4: The dynamic tracking method adopted for the first safety hazard coefficient triggered by the elevator vibration abnormality management microservice layer and the second safety hazard coefficient triggered by the elevator speed abnormality management microservice layer is as follows:

[0118] N41: Establish the sequence N(b,a) corresponding to the first safety hazard coefficient i ), where a i The sound correlation value corresponding to the abnormal elevator vibration object in different image frames is constructed to construct y=k*f(a i ), k is the slope of the linear function. When the slope does not change within T time, continue tracking. When the absolute value of the change in k is greater than the threshold, stop tracking, stop the elevator operation, and send an elevator stop command to the operation and maintenance personnel for operation and maintenance;

[0119] N42: Establish the sequence N(b,a) corresponding to the second safety hazard coefficient i), where a i The sound correlation degree values ​​corresponding to the abnormal elevator speed object in different image frames are constructed to construct y=k*f(a i ), k is the slope of the linear function. When the slope does not change within T time, continue tracking. When the absolute value of the change in k is greater than the threshold, stop tracking, stop the elevator operation, and send an elevator stop command to the operation and maintenance personnel for operation and maintenance;

[0120] N5: A deep learning model is deployed for N elevators. The cloud platform designs a visual interface and pre-configures the threshold of the maximum number of elevators to be monitored by a deep learning model. If the system finds that the maximum detection capability value of a deep learning model does not reach the threshold, it decreases by 1 and adjusts the number of elevators monitored by a deep learning model until the system load is balanced.

[0121] N6: For a deep learning model, deploy M corresponding neural networks to correspond to it, and pre-configure the threshold of the maximum number of control neural networks for a deep learning model. If the system finds that the maximum detection capability value of a deep learning model does not reach the threshold, it decreases by 1 in sequence and adjusts the number of neural networks corresponding to a deep learning model until the system load is balanced.

[0122] Furthermore, the neural network model is obtained by the following method: on the basis of the YOLOv7 network model, the number of classifications of the model is set to 4, corresponding to the electric vehicle prohibition elevator management microservice module, the elevator hand clamping management microservice module, the elevator vibration abnormality management microservice module and the elevator speed abnormality management microservice module respectively, the feature extraction network is set to the ShuffleNet network, and the enhanced feature network is set to the depthwise separable convolution.

[0123] It should be noted that due to the elevator cloud platform management system's super-strong multi-service processing capabilities through the cloud platform, multiple neural network models and deep learning models can be deployed on the cloud platform to quickly and accurately identify the security issues of each module business of the elevator. At the same time, the cloud platform realizes load balancing processing of multiple neural network models and deep learning models, and can stably realize the problem that multiple neural network models and deep learning models consume a lot of resources for image processing. The cloud platform can just solve the problem of using multiple training models in one system. This embodiment adopts the Nodejs development architecture system. Nodejs is based on the microservice architecture. Multiple microservices can run independently and stably on the cloud platform. Nodejs has a huge ecosystem. npm, as the world's largest open source library collection, provides developers with a rich selection of modules and speeds up development. At the same time, the deployment of Nodejs applications is relatively simple, and services can be quickly started and restarted. Nodejs is cross-platform and can run on multiple operating systems. This provides flexibility for the microservice architecture and is easy to deploy and maintain in different environments. Nodejs microservices can be well combined with container technologies such as Docker, support containerized deployment of microservices, and facilitate expansion and management in the cloud platform environment. The microservice architecture allows services to be independently developed and tested. The modular characteristics of Nodejs make it easier to write unit tests and integration tests.

[0124] Nodejs combined with neural network model development can be implemented through a variety of machine learning libraries. TensorFlow.js is a JavaScript library developed by Google that allows developers to use machine learning models directly in a browser or Nodejs environment. It supports the development of machine learning models using JavaScript and provides a series of pre-trained models that can be used for common use cases. Using the TensorFlow.js library, this embodiment builds, trains, and deploys a neural network model for image recognition based on elevator safety, including a model for machine learning tasks for image recognition.

[0125] The beneficial effects of an elevator cloud platform management system based on image recognition of the present invention are as follows:

[0126] The present invention collects video frames in the target elevator and uploads the video frames to the cloud platform. The cloud platform is embedded with a target segmentation model. The target segmentation model is used to segment objects in the video frames in the elevator. The segmented objects include electric vehicle objects and associated sound objects, elevator hand clamping objects and associated sound objects, elevator vibration abnormal objects and associated sound objects, and elevator speed abnormal objects and associated sound objects. The sound objects are filtered by correlation. Each microservice module independently processes the image recognition business of segmenting image objects and associated sound objects based on the neural network of the YOLOv7 network model. The comprehensive management module is based on a deep learning model. Secondary recognition of image features is performed on objects generated by different microservice modules, and comprehensive dispatch management of elevator safety is performed based on the secondary recognition results. The present invention accurately and quickly locates safety issues that exist or occur in elevators based on the correlation of sound features in image recognition, and comprehensively manages the secondary recognition of neural network model output results corresponding to different microservice module tasks through a deep learning model. At the same time, load balancing of different microservice image processing tasks on the cloud platform is achieved, and elevator safety issues are further accurately and quickly identified and resolved. The powerful processing capabilities of the cloud platform are used to deploy multiple neural network models and deep learning models to quickly and accurately identify safety issues in the services of each module of the elevator.

[0127] The present invention and its embodiments are described above, which is not restrictive. What is shown in the accompanying drawings is only one of the embodiments of the present invention, and the actual content is not limited to this. In short, if ordinary technicians in this field are inspired by it and do not deviate from the purpose of the invention, they can creatively design structural methods and embodiments similar to the technical scheme, which should all fall within the scope of protection of the present invention.

Claims

1. An elevator cloud platform management system based on image recognition, characterized in that: It comprises an image acquisition module (1), an electric vehicle entry ban segmentation module (2), an elevator hand clamp segmentation module (3), an elevator vibration abnormality segmentation module (4), an elevator speed abnormality segmentation module (5), an electric vehicle entry ban elevator management microservice module (6), an elevator hand clamp management microservice module (7), an elevator vibration abnormality management microservice module (8), an elevator speed abnormality management microservice module (9) and a comprehensive management module (10); The image acquisition module (1) is used to acquire a video frame image in a target elevator and upload the video frame image to a cloud platform, wherein the cloud platform is embedded with a target segmentation model, and the target segmentation model is used to segment objects in the video frame image in the elevator. The cloud platform is based on a Nodejs microservice development architecture; The target segmentation model is used to segment the objects in the ladder video frame image. The specific implementation process is: S1: Edit the collected historical surveillance videos to generate elevator operation clips; S2: Classify the elevator operation clips in step S1, and mark the target objects of the video frame to form object data set labels, where the objects include electric vehicle objects, elevator hand clamp objects, elevator vibration abnormal objects, and elevator speed abnormal objects; S3: Performing a fast Fourier transform on the image frame corresponding to the target object to obtain the spectrum value of each corresponding speech frame, generating a spectrogram using the spectrum value of each speech frame, and extracting features from the spectrogram using the Inception convolution structure in the image recognition algorithm to obtain a sound feature label set associated with the target object; S4: filtering the sound feature tags with a correlation less than a threshold value according to the correlation between the object and the sound feature tags associated with the object, thereby forming a tag of the target object and a tag of the associated sound; S5: constructing training data according to the video frame images with object labels and sound feature labels associated with the objects, and using the training data to train the target segmentation model to obtain a trained target segmentation model; The electric vehicle prohibited entry segmentation module (2) is used to perform electric vehicle object segmentation on the in-elevator video frame image according to the target segmentation model embedded in the cloud platform, and determine the electric vehicle object and the first identification object, wherein the first identification object is the sound object associated with the electric vehicle object; The elevator hand-caught object segmentation module (3) is used to segment the elevator hand-caught object on the elevator video frame image according to the target segmentation model embedded in the cloud platform, and determine the elevator hand-caught object and a second identification object, wherein the second identification object is a sound object associated with the elevator hand-caught object; The SIFT algorithm is used for electric vehicle object segmentation and elevator hand-claw object segmentation to obtain whether there is an electric vehicle object or an elevator hand-claw object in each video frame. If there is an electric vehicle object or an elevator hand-claw object, the SIFT algorithm is used to perform secondary object confirmation on the electric vehicle object or the elevator hand-claw object between adjacent frames; The elevator vibration abnormality segmentation module (4) is used to segment the elevator vibration abnormality object on the elevator video frame image according to the target segmentation model embedded in the cloud platform, and determine the elevator vibration abnormality object and a third identification object, wherein the third identification object is a sound object associated with the elevator vibration abnormality object; The elevator speed abnormality segmentation module (5) is used to segment the elevator speed abnormality object on the elevator video frame according to the target segmentation model embedded in the cloud platform, and determine the elevator speed abnormality object and a fourth identification object, wherein the fourth identification object is a sound object associated with the elevator speed abnormality object; The elevator speed abnormal object segmentation and elevator vibration abnormality segmentation adopt the feature matching method of SIFT algorithm combined with the streamer method to obtain whether there is an elevator speed abnormal object or an elevator speed abnormal object in each video frame; The electric vehicle prohibited from entering the elevator management microservice module (6) is used to perform visual training on electric vehicle objects and related sound objects. The trained model is an electric vehicle prohibited from entering the elevator model, which is used for electric vehicle prohibited from entering the elevator management. The training adopts a neural network model for training; The elevator hand-entrapment management microservice module (7) is used to perform visual training on elevator hand-entrapment objects and associated sound objects. The trained model is an elevator hand-entrapment model, which is used for elevator hand-entrapment management. The training is performed using a neural network model. The elevator vibration abnormality management microservice module (8) is used to perform visual training on the elevator vibration abnormality object and the associated sound object, the trained model is the elevator vibration abnormality model, the model is used for elevator vibration abnormality management, and the training adopts a neural network model for training; The elevator speed abnormality management microservice module (9) is used to perform visual training on the elevator speed abnormality object and the associated sound object, and the trained model is used for elevator speed abnormality management, and the training is performed using a neural network model; The comprehensive management module (10) performs secondary recognition of image features on objects generated by different microservice modules based on a deep learning model, and performs comprehensive dispatch management of elevator safety according to the secondary recognition results; The different microservice modules correspond to different neural network models, a total of 4 neural network models and one deep learning model. The models will be deployed in a distributed manner on the cloud platform. The cloud platform controls the number of models through load balancing. The cloud platform uses asynchronous communication to uniformly manage the training of each model and real-time data testing.

2. The elevator cloud platform management system based on image recognition according to claim 1 is characterized in that: The specific implementation process of the correlation degree in step S4 is: L1: According to each target object label and the degree of sound association in each video frame, obtain the feature parameters of each target object label in each video frame; L2: The target object label in the video frame is taken as the first label, and the label of the same target object in the adjacent next video frame is taken as the second label. The absolute value of the difference between the feature parameters of the sound label corresponding to the first label and the sound label corresponding to the second label is taken as the denominator; L3: There are multiple video frames in the video frame sequence, and a video frame with the target object label in step L2. If the video frame also has a sound label corresponding to the target object label, the feature parameters of the target object label and the corresponding sound label are subtracted, the absolute value of the difference is taken, and the absolute value of all differences is taken to construct the numerator; L4: Subtract the preset positive comparison parameter from the ratio of the denominator to the numerator and perform positive correlation mapping to obtain the correlation between the target object in each video frame and the sound label in the video frame.

3. The elevator cloud platform management system based on image recognition according to claim 1, characterized in that: The specific implementation process of the electric vehicle prohibition elevator management microservice module (6) is as follows: M1: Use the images of the electric vehicle object and the first identification object as training data for the electric vehicle prohibition model; M2: The condition for completing the training of the electric vehicle prohibition elevator model is that the correlation degree of the electric vehicle object or the sound associated with the electric vehicle object can meet the set threshold. The training is terminated as long as one of the two conditions is met. At the same time, the accuracy of the training model recognition must reach 90%; M3: A trained electric vehicle prohibition elevator model, which is used to determine whether an electric vehicle enters the elevator. If so, a warning instruction is sent to the elevator control module to keep the elevator door open. The elevator control module is used to control the elevator switch to close. M4: The camera in the elevator takes pictures in real time and transmits the pictures to the electric vehicle prohibition elevator management microservice module. The electric vehicle prohibition elevator management microservice fails to identify the electric vehicle and sends a warning cancellation instruction to the elevator control module to close the elevator door normally. The specific implementation process of the elevator hand-entrapment management microservice module (7) is as follows: K1: Use the images of the elevator hand-caught object and the second identification object as training data for the elevator hand-caught model; K2: The condition for completing the training of the elevator hand-pinching model is that the elevator hand-pinching object or the sound associated with the elevator hand-pinching object can be detected and the correlation degree meets the set threshold. As long as one of the two conditions is met, the training is terminated. At the same time, the accuracy of the training model recognition must reach 90%; K3: A trained elevator hand-pinching model, which is used to identify the elevator hand-pinching object. If it exists, a warning command is sent to the elevator control module to keep the elevator door open until the hand-pinching object disappears. Then, a warning release command is sent to the elevator control module to close the elevator door normally. The specific implementation process of the elevator vibration abnormality management microservice module (8) is as follows: R1: Use the images of the elevator vibration abnormal object and the third identification object as training data for the elevator vibration abnormality model; R2: The condition for completing the elevator vibration abnormality training is that the elevator vibration abnormality object can be detected or the correlation of the sound associated with the elevator vibration abnormality object meets the set threshold. As long as one of the two conditions is met, the training is terminated, and the accuracy of the training model recognition must reach 90%; R3: The trained elevator vibration anomaly model is used to determine the elevator vibration anomaly. If it exists, a warning instruction is sent to the operation and maintenance personnel, and the first safety hazard coefficient is generated; The specific implementation process of the elevator speed abnormality management microservice module (9) is as follows: P1: using the images of the elevator speed abnormality object and the fourth identification object as training data for the elevator speed abnormality model; P2: The condition for completing the elevator speed abnormality training is that the elevator speed abnormality object can be detected or the correlation of the sound associated with the elevator speed abnormality object meets the set threshold. The training is terminated as long as one of the two conditions is met. At the same time, the accuracy of the training model recognition must reach 90%; P3: The trained elevator speed abnormality model is used to judge the elevator speed abnormality. If it exists, a warning instruction is sent to the operation and maintenance personnel, and a second safety hazard coefficient is generated.

4. The elevator cloud platform management system based on image recognition according to claim 1, characterized in that: The specific implementation process of the secondary identification of the comprehensive management module (10) is as follows: N1: Build a deep learning model, and the training data is the output feature object data set of the electric vehicle prohibition elevator management microservice module, the elevator hand clamp management microservice module, the elevator vibration abnormality management microservice module, and the elevator speed abnormality management microservice module; N2: The deep learning model has a shared layer that is a public data set, and the other four layers are set, including an electric vehicle prohibition elevator management microservice layer, an elevator hand clamp management microservice layer, an elevator vibration abnormality management microservice layer, and an elevator speed abnormality management microservice layer; N3: Perform multi-task learning training on the four layers, and the trained deep learning model is used for actual deployment; N4: The dynamic tracking method adopted for the first safety hazard coefficient triggered by the elevator vibration abnormality management microservice layer and the second safety hazard coefficient triggered by the elevator speed abnormality management microservice layer is as follows: N41: Establish the sequence N(b,a) corresponding to the first safety hazard coefficient i ), where a i The sound correlation value corresponding to the abnormal elevator vibration object in different image frames is constructed to construct y=k*f(a i ), k is the slope of the linear function. When the slope does not change within T time, continue tracking. When the absolute value of the change in k is greater than the threshold, stop tracking, stop the elevator operation, and send an elevator stop command to the operation and maintenance personnel for operation and maintenance; N42: Establish the sequence N(b,a) corresponding to the second safety hazard coefficient i ), where a i The sound correlation degree values ​​corresponding to the abnormal elevator speed object in different image frames are constructed to construct y=k*f(a i ), k is the slope of the linear function. When the slope does not change within T time, continue tracking. When the absolute value of the change in k is greater than the threshold, stop tracking, stop the elevator operation, and send an elevator stop command to the operation and maintenance personnel for operation and maintenance; N5: A deep learning model is deployed for N elevators. The cloud platform designs a visual interface and pre-configures the threshold of the maximum number of elevators to be monitored by a deep learning model. If the system finds that the maximum detection capability value of a deep learning model does not reach the threshold, it decreases by 1 and adjusts the number of elevators monitored by a deep learning model until the system load is balanced. N6: For a deep learning model, deploy M corresponding neural networks to correspond to it, and pre-configure the threshold of the maximum number of control neural networks for a deep learning model. If the system finds that the maximum detection capability value of a deep learning model does not reach the threshold, it decreases by 1 in sequence and adjusts the number of neural networks corresponding to a deep learning model until the system load is balanced.

5. The elevator cloud platform management system based on image recognition according to claim 1, characterized in that: The neural network model is obtained by the following method: on the basis of the YOLOv7 network model, the number of classifications of the model is set to 4, corresponding to the electric vehicle prohibition elevator management microservice module, the elevator hand clamping management microservice module, the elevator vibration abnormality management microservice module and the elevator speed abnormality management microservice module respectively, the feature extraction network is set to the ShuffleNet network, and the enhanced feature network FPN is set to the depth separable convolution.

6. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement the functions of an elevator cloud platform management system based on image recognition as described in any one of claims 1 to 5.

Images