Carriage cleaning abnormity identification method and device, electronic equipment and storage medium

Abstract

The invention provides a carriage cleaning abnormity identification method and device, electronic equipment and a storage medium, and relates to the technical field of rail transit, a target detection model and a target classification model are utilized to identify the number and category of personnel in a current carriage, and when the requirements are both met, the carriage cleaning abnormity is identified. The cleaning action of the cleaner is analyzed through the posture evaluation model, whether the cleaning action meets the standard or not is judged, whether the cleaning date recorded by the cleaner is abnormal or not is verified through an optical character recognition (OCR) algorithm, and reminding intervention is further conducted when it is monitored that the cleaning action of the cleaner does not meet the standard or the recorded cleaning date is abnormal. Through cooperative work of multiple models, the accuracy of carriage cleaning abnormity recognition is greatly improved, automation and intelligence of carriage cleaning detection are achieved, abnormal conditions found in the cleaning process can be handled in time, and the efficiency of carriage cleaning management is remarkably improved.

Description

Technical Field

[0001] This application relates to the technical field of rail transit. Specifically, it relates to a method, device, electronic device, and storage medium for identifying abnormal carriage cleaning. Background Art

[0002] In the field of rail transit, such as subways and light rails, the cleaning work of carriages is crucial, which is directly related to the riding experience and health of passengers. However, there are many deficiencies in the existing carriage cleaning management methods. Traditional cleaning detection often relies on manual inspections, which are not only inefficient but also prone to omissions, and it is impossible to timely detect abnormal situations during the cleaning process. For example, problems such as cleaners not cleaning on time, perfunctory cleaning actions, or incorrect recording of cleaning dates may occur, and these problems are difficult to accurately and quickly identify and handle. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a method, device, electronic device, and storage medium for identifying abnormal carriage cleaning, which can accurately judge whether there are abnormalities during the carriage cleaning process through the collaborative work of multiple models, and improve the intelligent level and efficiency of carriage cleaning management.

[0004] In a first aspect, a method for identifying abnormal carriage cleaning provided by an embodiment of this application includes the following steps: Collect first images of the personnel in each carriage of multiple carriages through a camera at a set cleaning time, and detect the first images based on a trained object detection model to obtain the number of personnel in all current carriages; If the number of personnel in all current carriages is less than a set threshold, classify the work uniforms of the first images based on a trained target classification model to determine the personnel categories in all current carriages; Judge whether there is a cleaner and no passengers based on the personnel categories in all current carriages. If so, determine that the carriage cleaning operation conditions are met, obtain the video stream of the cleaner in the cleaning area of each carriage through the camera, and evaluate the video stream based on a trained posture evaluation model to judge whether the cleaning actions of the cleaner are normal; Verify whether the cleaning date recorded by the cleaner is abnormal based on the OCR recognition result and the time of the carriage management system.

[0005] In some embodiments, a trained target classification model is obtained in the following manner: Obtain a first image sample set of the personnel in the carriage during train operation and non-operation, and perform corresponding labeling of the work uniform categories on the first image sample set to obtain a first image labeled sample set; the work uniform categories include cleaners, passengers, and others; The data augmentation strategy based on image masking generates a larger number of second image annotation sample sets from the first image annotation sample set, aggregates the first image annotation sample set and the second image annotation sample set, and divides them into first image annotation training samples and first image annotation test samples; Train a pre-constructed target classification model based on the first image annotation training samples to obtain an initial target classification model; and input the first image annotation test samples into the initial target classification model to verify its probability accuracy; If the probability accuracy of the initial target classification model meets the preset requirements, use the initial target classification model as the trained target classification model.

[0006] In some embodiments, the data augmentation strategy based on image masking generates a larger number of second image annotation sample sets from the first image annotation sample set, including the following steps: Take each first image annotation sample in the first image annotation sample set as the original image one by one, and randomly select one from other first image annotation samples as the replacement source image; Randomly determine the mask area for the original image, and extract the target area corresponding to the mask area from the replacement source image; Replace the target area into the mask area of the original image to obtain a new image, and re-annotate the new image to obtain a second image annotation sample, so as to generate a second image annotation sample set.

[0007] In some embodiments, the trained pose evaluation model is used to evaluate the video stream to determine whether the cleaning actions of the cleaner are normal, including the following steps: Pre-divide different cleaning areas for each carriage, and set corresponding cleaning action standards for each cleaning area to characterize whether the cleaning action features of the cleaner meet the requirements; the cleaning action features include the bending angle of the arm joints and the swinging amplitude of the arms; the cleaning areas include the ground, seats, doors, and ventilation openings; Extract the first video stream of the carriage where the cleaner is located, and input the video frame images of the first video stream into the trained pose evaluation model according to the time series to obtain the human key point coordinates of the cleaner; Calculate the cleaning action features of the cleaner in different cleaning areas according to the obtained human key point coordinates, and compare the calculated cleaning action features with the set cleaning action standards to determine whether the cleaning actions of the cleaner in the corresponding cleaning areas are normal.

[0008] In some embodiments, verifying whether the cleaning date recorded by the cleaner is abnormal based on the OCR recognition result and the time of the carriage management system includes the following steps: Collect a second image of the identification area recording the cleaning date inside the carriage through a camera; Based on the optical character recognition (OCR) algorithm, recognize the second image to determine whether the cleaning date recorded by the cleaner has changed; if it has changed, obtain the time when the cleaner modified the cleaning date and the cleaning date recorded by the cleaner; If the cleaning date recorded by the cleaner is inconsistent with the time of the carriage management system, or the time when the cleaner modified the cleaning date is before the time of the carriage cleaning work, it is determined that the cleaning date recorded by the cleaner is abnormal.

[0009] In some embodiments, the method further includes the following steps: If the cleaning action of the cleaner is abnormal, play voice guidance information about the cleaning action to the cleaner through the in-car voice prompt device; if the recorded cleaning date is abnormal, send a reminder message including the carriage number and the abnormal date to the management personnel through the carriage management system and use it as a performance evaluation indicator.

[0010] In a second aspect, the present application provides a carriage cleaning abnormality recognition device, and the device includes: A detection module, configured to collect a first image of the personnel inside the carriage through a camera, and detect the first image based on a trained target detection model to obtain the number of personnel currently inside the carriage; A classification module, configured to, if the number of personnel currently inside the carriage is less than a set threshold, classify the first image based on a trained target classification model to obtain the work uniform category of the personnel currently inside the carriage; A first abnormality recognition module, configured to determine whether the carriage cleaning operation conditions are met according to the work uniform category of the personnel currently inside the carriage. If so, obtain the video stream of the cleaner in the cleaning area inside the carriage through a camera, and evaluate the video stream based on a trained posture evaluation model to determine whether the cleaning action of the cleaner is normal; A second abnormality recognition module, configured to verify whether the cleaning date recorded by the cleaner is abnormal based on the optical character recognition (OCR) algorithm.

[0011] In a third aspect, the present application provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, the steps of the carriage cleaning abnormality recognition method according to any one of the first aspects are executed.

[0012] Fourthly, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the carriage cleaning anomaly recognition method described in any one of the first aspects.

[0013] For the carriage cleaning anomaly recognition method, device, electronic device and storage medium provided by the present application, at a set cleaning time, first images of people in each carriage of multiple carriages are collected through a camera, and based on a trained object detection model, the first images are detected to obtain the number of people in all current carriages; if the number of people in all current carriages is less than a set threshold, based on a trained target classification model, the work uniforms in the first images are classified to determine the types of people in all current carriages; according to the types of people in all current carriages, it is judged whether there is a cleaner and no passengers. If so, it is determined that the carriage cleaning operation conditions are met, the video stream of the cleaner in the cleaning area of each carriage is obtained through the camera, and based on a trained pose evaluation model, the video stream is evaluated to judge whether the cleaning actions of the cleaner are normal; based on the OCR recognition result and the carriage management system time, it is verified whether the cleaning date recorded by the cleaner is abnormal. Thus, through the collaborative work of multiple models, accurate judgments are made in each link from carriage crowd detection, work uniform classification and recognition, cleaning operation detection to cleaning date recognition, effectively reducing false alarms and missed alarms, greatly improving the accuracy of carriage cleaning anomaly recognition, and realizing the automation and intelligence of carriage cleaning detection.

[0014] In some embodiments, through various data collection methods and image augmentation techniques, the target classification model has stronger generalization ability and can better adapt to the carriage cleaning detection requirements in different scenarios. Description of the Drawings

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 Shows the flowchart of the carriage cleaning anomaly recognition method described in the embodiments of the present application; Figure 2 Shows the flowchart of generating a larger number of second image annotation sample sets from the first image annotation sample set based on the data augmentation strategy of image masking in the embodiments of the present application; Figure 3The flowchart shows that in an embodiment of the present application, the trained pose evaluation model is used to evaluate the video stream to determine whether the cleaning actions of the cleaner are normal; Figure 4 The schematic structural diagram of the carriage cleaning anomaly recognition device in an embodiment of the present application is shown; Figure 5 The schematic structural diagram of the electronic device in an embodiment of the present application is shown. Detailed implementation manners

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It should be understood that the accompanying drawings in the present application are only for the purposes of illustration and description, and are not used to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn to actual scale. The flowcharts used in the present application show operations implemented according to some embodiments of the present application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical context relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present application.

[0018] In addition, the described embodiments are only some embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application usually described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application claimed, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts fall within the protection scope of the present application.

[0019] It should be noted that the term "including" will be used in the embodiments of the present application to indicate the existence of the features stated hereinafter, but does not exclude the addition of other features.

[0020] In view of the technical problems proposed in the background art, the present application provides a carriage cleaning anomaly recognition method, device, electronic device, and storage medium, which can accurately determine whether there are anomalies during the carriage cleaning process through the collaborative work of multiple models, and improve the intelligent level and efficiency of carriage cleaning management.

[0021] In one embodiment, referring to the accompanying specification Figure 1 , a carriage cleaning anomaly recognition method provided by the present application includes the following steps: S1. At the set cleaning time, collect the first images of the personnel in each carriage of multiple carriages through a camera, and detect the first images based on the trained object detection model to obtain the number of personnel in all current carriages; S2. If the number of personnel in all current carriages is less than the set threshold, classify the work uniforms of the first images based on the trained object classification model to determine the personnel categories in all current carriages; S3. Judge whether there is a cleaner and no passengers according to the personnel categories in all current carriages. If so, determine that the carriage cleaning operation conditions are met, obtain the video stream of the cleaner in the cleaning area of each carriage through the camera, and evaluate the video stream based on the trained pose evaluation model to judge whether the cleaning actions of the cleaner are normal; S4. Verify whether the cleaning date recorded by the cleaner is abnormal based on the OCR recognition result and the time of the carriage management system.

[0022] Specifically, in step S1, the yolov8 object detection model based on the convolutional neural network (CNN) architecture can be used to detect the number of personnel in the carriage. This model is mainly composed of a backbone network, a neck network, and a detection head. Among them, the backbone network adopts the CSPDarknet (Cross Stage Partial Darknet) structure, which is responsible for extracting rich feature information from the first images of the personnel in the carriage collected through the camera; the neck network uses the PANet (Path Aggregation Network), which fuses feature maps of different scales through bottom-up and top-down path aggregation to enhance the model's detection ability for targets of different sizes; the detection head adopts the decoupled head design of the YOLO series to predict the category, bounding box position, and confidence of the target respectively. When training the yolov8 object detection model, collect a large amount of carriage image data under different scenarios and lighting conditions, and use annotation tools to annotate the bounding boxes of the personnel in the images. Divide the annotated data into a training set and a test set, use the stochastic gradient descent (SGD) optimization algorithm, set appropriate hyperparameters such as learning rate, momentum, and weight decay, and perform iterative training on the model. During the training process, continuously adjust the parameters of the model to gradually reduce the loss function of the model (such as CIoU Loss for bounding box regression loss and cross-entropy loss for category classification loss) until the model converges. When the real-time collected carriage images are input into the trained yolov8 object detection model, the model will output the detected number of personnel.

[0023] Among them, how to construct and train the object detection model should be well-known technical means to those skilled in the art, and specific embodiments will not be elaborated here.

[0024] Preferably, the object detection model only starts running at the start time of the cleaning period (e.g., reasonably set according to the time point of the last train). This can reduce energy consumption and operating costs.

[0025] In step S2, the reason for restricting the number of people in the current carriage is that in the rail transit operation scenario of this application, the behavior habits of passengers will affect the cleaning operation. For example, passengers often walk around and place luggage in the carriage. Even if the number of people is small, it may still interfere with the cleaning work. At the same time, there is a risk of collision between cleaning personnel / cleaning equipment and passengers. For safety reasons, the cleaning work is limited to the condition without passengers. Therefore, in this application, a threshold is set for the number of people in the current carriage (e.g., reasonably set according to the total number of cleaning personnel). If the number of people in the current carriage exceeds this threshold, the subsequent object classification model, pose evaluation model, and optical character recognition OCR algorithm will not run.

[0026] On the one hand, the object classification model, pose evaluation model, and optical character recognition OCR algorithm require a large amount of computing resources when running, such as resources like CPU and GPU. When the train is running, the number of people in the carriage is often greater than the set threshold. If the object classification model, pose evaluation model, and optical character recognition OCR algorithm run simultaneously, it will keep the computing resources in a high-load state, resulting in a large amount of energy consumption. In these cases, not running the object classification model, pose evaluation model, and optical character recognition OCR algorithm can avoid overload and also allow the hardware resources to be more allocated to other necessary tasks, such as carriage system monitoring, management, etc., improving the stability of the carriage system operation. On the other hand, if the number of people in the current carriage is too large, for example, the working scenarios of non-cleaning personnel (passengers, maintenance workers, or drivers, etc.) are very different from the cleaning actions of cleaners. If the pose evaluation model runs, it is easy to misjudge their normal working behaviors as cleaning abnormalities, increasing the misjudgment rate.

[0027] The target classification model can also adopt the YOLOv8 convolutional neural network architecture. When training it, first, use multiple high-definition cameras installed at different positions in the carriage to obtain the first image sample set of people in the carriage when the train is running and not running, and mark the first image sample set with the corresponding work clothes category to obtain the first image annotation sample set; the work clothes category includes cleaners, security officers, passengers, drivers, maintenance personnel, etc., among which the first image sample set obtained when the train is running corresponds to the work clothes category of passengers and security officers, and the first image sample set obtained when the train is not running corresponds to the work clothes category of cleaners, drivers, and maintenance personnel.

[0028] Furthermore, in order to improve the diversity of the first image sample set, the present application generates a larger number of second image annotation sample sets from the first image annotation sample set based on the data augmentation strategy of the image mask, and summarizes the first image annotation sample set and the second image annotation sample set, and divides them into first image annotation training samples and first image annotation test samples; then the pre-constructed target classification model is trained based on the first image annotation training samples to obtain an initial target classification model; and the first image annotation test sample is input into the initial target classification model to verify its probability accuracy; if the probability accuracy of the initial target classification model meets the preset requirements, the initial target classification model is used as the trained target classification model. Furthermore, the real-time collected images of people in the car are input into the trained target classification model, and the work clothes categories of the people in the current car are output. For example, the work clothes categories are divided into five categories, namely 0: maintenance personnel, 1: driver, 2: security personnel, 3: cleaning, and 4: passenger.

[0029] See the attached manual for details. Figure 2 The data augmentation strategy based on image mask generates a larger number of second image annotated sample sets from the first image annotated sample set, including the following steps: S201, taking each first image annotated sample in the first image annotated sample set as an original image one by one, and randomly selecting one from other first image annotated samples as a replacement source image; S202, randomly determining a mask area of the original image, and extracting a target area corresponding to the mask area from the replacement source image; S203: Replace the target area into the mask area of the original image to obtain a new image, and re-label the new image to obtain a second image labeled sample to generate a second image labeled sample set.

[0030] That is, a random mask replacement strategy is adopted. First, a random number generator is used to generate a random mask area for each first image annotation sample. The size, shape, and position of the mask area are randomly determined. Then, another first image annotation sample is randomly selected from the first image annotation sample set, and the corresponding area is extracted. The extracted area is replaced into the original image, or the brightness, contrast, color, etc. of the replaced mask area are further adjusted. Finally, a new synthetic image is generated based on the enhanced mask, and the work uniform category is labeled for the synthetic image to obtain an augmented second image annotation sample. Thus, a large number of second image annotation sample sets are generated according to the limited first image annotation sample set, making full use of the existing annotation resources, improving the diversity and richness of the data, and enhancing the generalization ability of the target classification model.

[0031] In other embodiments, data augmentation can also be performed in other ways. For example, methods such as horizontal / vertical flipping, rotation, scaling, cropping, shearing, translation, contrast, color jitter, and noise are used to augment the existing annotation resources, and it is not limited to the data augmentation strategy based on image masks proposed in this application.

[0032] In step S3, the defined carriage cleaning operation conditions are essentially the operating conditions of the pose evaluation model, which are further limitations on the personnel category on the basis that the number of people in the current carriage meets the requirements. In one embodiment, the set carriage cleaning operation conditions are: there is a cleaner and no passengers in the current carriage. For example, when the number of people in the current carriage is less than the set threshold but there are passengers, the pose evaluation model does not run; another example is that when the number of people in the current carriage is less than the set threshold, there are no passengers, but there are only maintenance workers and drivers and no cleaner, the pose evaluation model does not run.

[0033] Then, when the number and category of people in the current carriage both meet the requirements, the pose evaluation model is run to perform the cleaning action evaluation step. Among them, refer to the attached Figure 3 specification, evaluating the video stream based on the trained pose evaluation model to determine whether the cleaning actions of the cleaner are normal includes the following steps: S301. Different cleaning areas are pre-divided for each carriage, and corresponding cleaning action standards are set for each cleaning area to characterize whether the cleaning action features of the cleaner meet the requirements. The cleaning action features include the bending angle of the arm joints and the swinging amplitude of the arms. The cleaning areas include the ground, seats, doors, and ventilation openings. S302. Extract the first video stream of the carriage where the cleaner is located, and input the video frame images of the first video stream into the trained pose evaluation model according to the time series to obtain the body key point coordinates of the cleaner. S303, calculating the cleaning action characteristics of the cleaner in different cleaning areas according to the obtained coordinates of the key points of the human body, and comparing the calculated cleaning action characteristics with the set cleaning action standards to determine whether the cleaning action of the cleaner in the corresponding cleaning area is normal.

[0034] Specifically, in step S301, cleaning action standards are mainly set for different cleaning areas in the car. For example, to clean the floor, the cleaning tool needs to be swung widely; to clean the seat, it is necessary to bend over and wipe with a smaller movement amplitude; when cleaning the vents, the arm is raised at a larger angle, etc. That is, different cleaning areas have unique cleaning action characteristics. In one embodiment, a two-dimensional table of cleaning areas, corresponding cleaning action standards, and human body key point coordinates required to calculate each cleaning action feature can be pre-constructed for subsequent direct call and comparison.

[0035] In step S302 and step S303, the main thing is how to obtain the coordinates of the key points of the cleaning staff to calculate the cleaning action characteristics of the cleaning staff in different cleaning areas. Specifically, first obtain the first video stream of the car where the cleaning staff is located, and extract video frames from the obtained continuous first video stream at a certain time interval. For example, 1-5 frames are extracted per second, and the specific interval can be determined according to the complexity of the cleaning action and the processing capacity of the posture evaluation model. The extracted video frames are stored in image formats, such as JPEG or PNG, and preprocessed, such as resizing, normalization, and color space conversion; for subsequent processing. Then the preprocessed video frame image is input into the posture evaluation model for reasoning. In one embodiment, the posture evaluation model adopts the MM action2 RTMO model, which can quickly and accurately locate the human posture information of the cleaning staff in the video stream or image sequence, that is, the coordinates of the key points of the human body, such as the positions of the shoulders, elbows, wrists, hips, knees, ankles, etc., and filter out the key point coordinates related to the cleaning action, such as arm joints. Finally, the cleaning action characteristics of the cleaner are calculated based on the filtered key point coordinates. For example, the angle features between key points are calculated to obtain the bending angle of the arm joint; the amplitude features of the arm swing are determined by comparing the position changes of key points in adjacent video frames. Then, according to the preset cleaning action standards, it is judged whether the cleaning action of the cleaner in the corresponding cleaning area is abnormal. For example, when cleaning the vents, the angle of the arm should be between 120°-180°; when cleaning the floor, the amplitude of the arm swing should be no less than 60°.

[0036] Among them, when an abnormality is detected in the cleaning action of the cleaner, timely reminder and recording should be carried out. For example, when it is detected that the swing amplitude of the cleaner's arm does not meet the standard, a reminder is immediately issued through the voice prompt device in the carriage, informing the cleaner to increase the action amplitude to ensure thorough cleaning, and the voice guidance information of the cleaning action is played; when it is detected that the bending angle of the cleaner's arm joint does not meet the standard, for example, when cleaning the ventilation opening at the top of the carriage, the raising angle of the arm is too small to reach all areas of the ventilation opening, a reminder is immediately issued through the voice prompt device in the carriage, explaining the characteristics of the cleaning part and the correct cleaning angle.

[0037] In addition, it should be noted that before comparing the calculated cleaning action characteristics with the set cleaning action standards, it is also necessary to verify whether the cleaning action characteristics of all cleaning areas have been calculated; if not, the second video stream when the cleaner is not in the carriage is extracted, and coordinate unification and time synchronization are performed on the second video stream and the first video stream; the video frame images of the second video stream and the first video stream are input into the trained pose evaluation model in time series to re-obtain the human key point coordinates of the cleaner, so as to calculate the cleaning action characteristics of the cleaner in different cleaning areas. That is, when all human key point coordinates cannot be obtained through the video stream of the carriage where the cleaner is located, the video stream captured by other carriages needs to be combined. Among them, how to perform coordinate unification and time synchronization of the video stream should be a well-known technical means in the art and is not the key point of this solution, so it will not be elaborated here.

[0038] In other embodiments, after obtaining the human key point coordinates of the cleaner, the result of whether the cleaning action of the cleaner is abnormal can also be obtained by directly analyzing the human bone points by constructing a neural network. This is because constructing a neural network can more accurately capture the essential characteristics of the cleaning action, thereby realizing automatic cleaning quality evaluation. Specifically in implementation, the human key point coordinates can be first normalized to a proportional value relative to the image size to eliminate the influence of different image resolutions; then processed in time series, and bone features are constructed, such as calculating the angles formed by adjacent joint points to reflect the bending state of the arm; calculating the distances between non-adjacent joint points to judge the action amplitude; calculating the displacement vectors, speeds and accelerations of the key points between consecutive frames to capture the dynamic characteristics of the action. Furthermore, the extracted bone features are analyzed to directly obtain the result of whether the cleaning action of the cleaner is abnormal. Among them, the construction and training of the neural network should be well-known technical means in the art and will not be elaborated here.

[0039] In step S4, it is mainly to judge whether the cleaning date recorded by the cleaner is normal. Specifically, the second image of the identification area recording the cleaning date inside the carriage is collected. For example, a camera or other image acquisition device is used to photograph the cleaning record board, electronic display screen, etc. that record the cleaning date. Ensure that the collected image is clear and complete, contains accurate date information, and provides high-quality data for subsequent optical character recognition (OCR) algorithm processing. Then, the second image containing the cleaning date is input into the optical character recognition (OCR) algorithm to extract the features in the image, convert the features into a character sequence, and remove invalid characters such as extra spaces and punctuation marks to extract the cleaning date in the image.

[0040] Among them, it is stipulated that the cleaner can only modify the cleaning date on the record board after completing the carriage cleaning work. In an embodiment, through the recognition of the second image, it is judged whether the cleaning date recorded by the cleaner has changed; if it has changed, the time when the cleaner modified the cleaning date (change time) and the cleaning date recorded by the cleaner are obtained; if the cleaning date recorded by the cleaner is inconsistent with the carriage management system time, or the time when the cleaner modified the cleaning date is before the carriage cleaning work time, it is judged that the cleaning date recorded by the cleaner is abnormal. Among them, the time of the carriage cleaning work can be extracted during the process of judging whether the cleaning action of the cleaner is normal in step S3.

[0041] When it is monitored that the recorded cleaning date is abnormal, for example, the cleaner intentionally / unintentionally fills in a cleaning date that is inconsistent with the carriage system date or fills in the cleaning date in advance, a reminder message including the carriage number and the abnormal date is sent to the management personnel through the carriage management system, so that the management personnel can conduct further investigation and processing, and it is used as an assessment index for the cleaner.

[0042] It can be seen that a carriage cleaning anomaly recognition method provided by this application uses a target detection model and a target classification model to identify the number and category of current personnel inside the carriage. And when both meet the requirements, it uses a posture evaluation model to analyze the cleaning actions of the cleaner to judge whether the cleaning actions conform to the specifications, and also uses the optical character recognition (OCR) algorithm to verify whether the cleaning date recorded by the cleaner is abnormal. Further, when it is monitored that the cleaning actions of the cleaner do not conform to the specifications or the recorded cleaning date is abnormal, a reminder intervention is carried out. Then, through the collaborative work of multiple models, from carriage personnel detection, work uniform classification recognition, cleaning operation detection to cleaning date recognition, accurate judgment is made in each link, effectively reducing false alarms and missed alarms, greatly improving the accuracy of carriage cleaning anomaly recognition, realizing the automation and intelligence of carriage cleaning detection, being able to timely handle abnormal situations found during the cleaning process, and significantly improving the efficiency of carriage cleaning management.

[0043] Based on the same inventive concept, a vehicle compartment cleaning abnormality identification device is also provided in an embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned vehicle compartment cleaning abnormality identification method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0044] As the instruction manual Figure 4 As shown, an embodiment of the present application provides a vehicle compartment cleaning abnormality identification device, the device comprising: The detection module 401 is used to collect a first image of a person in each of the multiple carriages through a camera at a set cleaning time, and detect the first image based on a trained target detection model to obtain the number of people in all the carriages at present; A classification module 402 is used to classify the first image into work clothes based on the trained target classification model to determine the categories of the people in all the carriages if the number of people in all the carriages is less than a set threshold; The first abnormality recognition module 403 is used to determine whether there is a cleaner and no passengers according to the types of people in all the carriages at present. If so, it is determined that the carriage cleaning operation conditions are met, and the video stream of the cleaner in the cleaning area of each carriage is obtained through the camera, and the video stream is evaluated based on the trained posture evaluation model to determine whether the cleaning action of the cleaner is normal; The second abnormality identification module 404 is used to verify whether the cleaning date recorded by the cleaner is abnormal based on the OCR recognition result and the time of the carriage management system.

[0045] In one embodiment, the device further comprises: A training module is used to obtain a first image sample set of people in a carriage when a train is running and not running, and to annotate the first image sample set with corresponding work clothes categories to obtain a first image annotated sample set; the work clothes categories include cleaners, passengers and others; based on the data augmentation strategy of the image mask, the first image annotated sample set is used to generate a larger number of second image annotated sample sets, and the first image annotated sample set and the second image annotated sample set are summarized and divided into first image annotated training samples and first image annotated test samples; based on the first image annotated training samples, a pre-constructed target classification model is trained to obtain an initial target classification model; and the first image annotated test sample is input into the initial target classification model to verify its probability accuracy; if the probability accuracy of the initial target classification model meets the preset requirements, the initial target classification model is used as the trained target classification model.

[0046] In one embodiment, the training module generates a larger number of second image annotation sample sets based on the data augmentation strategy of the image mask for the first image annotation sample set, including: taking each first image annotation sample in the first image annotation sample set as the original image one by one, and randomly selecting one from other first image annotation samples as the replacement source image; randomly determining the mask area for the original image, and extracting the target area corresponding to the mask area from the replacement source image; replacing the target area into the mask area of the original image to obtain a new image, and re-annotating the new image to obtain a second image annotation sample, so as to generate a second image annotation sample set.

[0047] In one embodiment, the conditions for meeting the carriage cleaning operation are: there is a cleaner in the current carriage and no passengers.

[0048] In one embodiment, the first anomaly recognition module 403 evaluates the video stream based on the trained pose evaluation model to determine whether the cleaning actions of the cleaner are normal, including: pre-dividing different cleaning areas for each carriage, and setting corresponding cleaning action standards for each of the cleaning areas to characterize whether the cleaning action features of the cleaner meet the requirements; the cleaning action features include the bending angle of the arm joints and the swinging amplitude of the arms; the cleaning areas include the ground, seats, doors, and ventilation openings; extracting the first video stream of the carriage where the cleaner is located, and inputting the video frame images of the first video stream into the trained pose evaluation model according to the time series to obtain the human key point coordinates of the cleaner; calculating the cleaning action features of the cleaner in different cleaning areas according to the obtained human key point coordinates, and comparing the calculated cleaning action features with the set cleaning action standards to determine whether the cleaning actions of the cleaner in the corresponding cleaning areas are normal. Among them, before comparing the calculated cleaning action features with the set cleaning action standards, it further includes: verifying whether the cleaning action features of all cleaning areas are calculated; if not, extracting the second video stream when the cleaner is not in the carriage, and performing coordinate unification and time synchronization on the second video stream and the first video stream; inputting the video frame images of the second video stream and the first video stream into the trained pose evaluation model according to the time series to re-obtain the human key point coordinates of the cleaner, so as to calculate the cleaning action features of the cleaner in different cleaning areas.

[0049] In one embodiment, the second anomaly recognition module 404 determines whether the cleaning date recorded by the cleaner is abnormal based on the OCR recognition result and the time verification of the carriage management system, including: collecting a second image of the identification area recording the cleaning date in the carriage through a camera; recognizing the second image based on the optical character recognition (OCR) algorithm to determine whether the cleaning date recorded by the cleaner has changed; if it has changed, obtaining the time when the cleaner modified the cleaning date and the cleaning date recorded by the cleaner; if the cleaning date recorded by the cleaner is inconsistent with the time of the carriage management system, or the time when the cleaner modified the cleaning date is before the time of the carriage cleaning work, it is determined that the cleaning date recorded by the cleaner is abnormal.

[0050] In one embodiment, the device further includes: A processing module, configured to, if the cleaning action of the cleaner is abnormal, play voice guidance information about the cleaning action to the cleaner through a voice prompt device in the carriage; if the recorded cleaning date is abnormal, send a reminder message including the carriage number and the abnormal date to the management personnel through the carriage management system, and use it as a performance appraisal indicator.

[0051] A carriage cleaning anomaly recognition device provided by the present application, wherein a detection module collects a first image of the personnel in each carriage of multiple carriages through a camera at a set cleaning time, and based on a trained object detection model, detects the first image to obtain the number of personnel in all current carriages; if the number of personnel in all current carriages is less than a set threshold, a classification module classifies the work uniforms of the first image based on a trained object classification model to determine the personnel categories of the personnel in all current carriages; a first anomaly recognition module determines whether there is a cleaner and no passengers based on the personnel categories of the personnel in all current carriages. If so, it is determined that the carriage cleaning operation conditions are met, and the video stream of the cleaner in the cleaning area of each carriage is obtained through the camera, and based on a trained posture evaluation model, the video stream is evaluated to determine whether the cleaning action of the cleaner is normal; a second anomaly recognition module determines whether the cleaning date recorded by the cleaner is abnormal based on the OCR recognition result and the time verification of the carriage management system. Thus, through the collaborative work of multiple models, accurate judgments are made in each link from carriage crowd detection, work uniform classification recognition, cleaning operation detection to cleaning date recognition, effectively reducing false alarms and missed alarms, greatly improving the accuracy of carriage cleaning anomaly recognition, and realizing the automation and intelligence of carriage cleaning detection.

[0052] Based on the same concept of the present invention, the accompanying Figure 5As shown in the figure, the present application provides a structure of an electronic device 500. The electronic device 500 includes: at least one processor 501, at least one network interface 504 or other user interfaces 503, a memory 505, and at least one communication bus 502. The communication bus 502 is used to implement connection communication between these components. The electronic device 500 optionally includes a user interface 503, including a display (such as a touch screen, LCD, CRT, holographic imaging, or projector, etc.), a keyboard, or a pointing device (such as a mouse, trackball, touchpad, or touch screen, etc.).

[0053] The memory 505 may include a read-only memory and a random access memory, and provide instructions and data to the processor 501. A part of the memory 505 may further include a non-volatile random access memory (NVRAM).

[0054] In some embodiments, the memory 505 stores the following elements, executable modules, or data structures, or subsets thereof, or extended sets thereof: An operating system 5051, including various system programs, for implementing various basic services and processing hardware-based tasks; An application program module 5052, including various application programs, such as a launcher, a media player, a browser, etc., for implementing various application services.

[0055] In the embodiments of the present application, by calling the programs or instructions stored in the memory 505, the processor 501 is used to execute the steps in a method for identifying abnormal carriage cleaning, and can accurately determine whether there is an abnormality during the carriage cleaning process through multi-model collaborative work, improving the intelligent level and efficiency of carriage cleaning management.

[0056] The present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the steps in the method for identifying abnormal carriage cleaning.

[0057] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, it can execute the above-mentioned method for identifying abnormal carriage cleaning.

[0058] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some communication interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0059] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0060] In addition, each functional unit in the embodiments provided in this application can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0061] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical discs and other various media that can store program codes.

[0062] Finally, it should be noted that the above embodiments are only specific implementation manners of the present application, used to illustrate the technical solutions of the present application, rather than limiting it. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the art within the technical scope disclosed by the present application can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. All should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for identifying abnormal carriage cleaning, characterized in that, The method includes the following steps: Collect first images of the personnel in each carriage of multiple carriages through a camera at a set cleaning time, and detect the first images based on a trained object detection model to obtain the number of personnel in all current carriages; If the number of personnel in all current carriages is less than a set threshold, classify the work uniforms of the first images based on a trained object classification model to determine the personnel categories in all current carriages; Judge whether there is a cleaner and no passengers based on the personnel categories in all current carriages. If so, determine that the carriage cleaning operation conditions are met, obtain the video stream of the cleaner in the cleaning area of each carriage through the camera, and evaluate the video stream based on a trained pose evaluation model to judge whether the cleaning actions of the cleaner are normal; Verify whether the cleaning date recorded by the cleaner is abnormal based on the OCR recognition result and the carriage management system time.

2. The method for identifying abnormal carriage cleaning according to claim 1, wherein, Obtain a trained object classification model in the following way: Obtain a first image sample set of the personnel in the carriage during train operation and non-operation, and perform corresponding labeling of the work uniform categories on the first image sample set to obtain a first image labeled sample set; the work uniform categories include cleaner, passenger, and others; Generate a larger number of second image labeled sample sets from the first image labeled sample set based on the data augmentation strategy of image masking, summarize the first image labeled sample set and the second image labeled sample sets, and divide them into first image labeled training samples and first image labeled test samples; Train a pre-constructed object classification model based on the first image labeled training samples to obtain an initial object classification model; and input the first image labeled test samples into the initial object classification model to verify its probability accuracy; If the probability accuracy of the initial object classification model meets the preset requirements, use the initial object classification model as the trained object classification model.

3. The method for identifying abnormal carriage cleaning according to claim 2, wherein The data augmentation strategy of image masking generates a larger number of second image labeled sample sets from the first image labeled sample set, including the following steps: Take each first image labeled sample in the first image labeled sample set as the original image one by one, and randomly select one from other first image labeled samples as the replacement source image; Randomly determine its mask area for the original image, and extract the target area corresponding to the mask area from the replacement source image; Replace the target area into the mask area of the original image to obtain a new image, and re-label the new image to obtain a second image labeled sample to generate a second image labeled sample set.

4. The method for identifying abnormal carriage cleaning according to claim 1, wherein, The evaluation of the video stream based on a trained pose evaluation model to judge whether the cleaning actions of the cleaner are normal includes the following steps: Pre-divide different cleaning areas for each carriage, and set corresponding cleaning action standards for each of the said cleaning areas, which are used to characterize whether the cleaning action characteristics of the cleaner meet the requirements; the cleaning action characteristics include the bending angle of the arm joints and the swinging amplitude of the arms; the cleaning areas include the floor, seats, doors, and vents. Extract the first video stream of the carriage where the cleaner is located, and input the video frame images of the first video stream into the trained pose evaluation model according to the time series to obtain the body key point coordinates of the cleaner. Calculate the cleaning action characteristics of the cleaner in different cleaning areas according to the obtained body key point coordinates, and compare the calculated cleaning action characteristics with the set cleaning action standards to determine whether the cleaning actions of the cleaner in the corresponding cleaning areas are normal.

5. The method for identifying abnormal carriage cleaning according to claim 4, wherein Before comparing the calculated cleaning action characteristics with the set cleaning action standards, the following steps are also included: Verify whether the cleaning action characteristics of all cleaning areas are calculated; if not, extract the second video stream of the cleaner outside the carriage, and perform coordinate unification and time synchronization on the second video stream and the first video stream. Input the video frame images of the second video stream and the first video stream into the trained pose evaluation model according to the time series to re-obtain the body key point coordinates of the cleaner, so as to calculate the cleaning action characteristics of the cleaner in different cleaning areas.

6. The method for identifying abnormal carriage cleaning according to claim 5, wherein The verification of whether the cleaning date recorded by the cleaner is abnormal based on the OCR recognition result and the carriage management system time includes the following steps: Collect a second image of the identification area in the carriage where the cleaning date is recorded through a camera. Based on the optical character recognition OCR algorithm, identify the second image to determine whether the cleaning date recorded by the cleaner has changed; if it has changed, obtain the time when the cleaner modified the cleaning date and the cleaning date recorded by the cleaner. If the cleaning date recorded by the cleaner is inconsistent with the carriage management system time, or the time when the cleaner modified the cleaning date is before the carriage cleaning work time, it is determined that the cleaning date recorded by the cleaner is abnormal.

7. The method for identifying abnormal carriage cleaning according to claim 6, characterized in that, The method also includes the following steps: If the cleaning action of the cleaner is abnormal, play voice guidance information about the cleaning action to the cleaner through the in-car voice prompt device; if the recorded cleaning date is abnormal, send a reminder message including the carriage number and the abnormal date to the management personnel through the carriage management system, and use it as a performance evaluation indicator.

8. An abnormal recognition device for carriage cleaning, characterized in that The device includes: A detection module, which is used to collect the first images of the personnel in each carriage of multiple carriages through a camera at the set cleaning time, and detect the first images based on the trained target detection model to obtain the number of personnel in all current carriages. A classification module, which is used to classify the work uniforms of the first images based on the trained target classification model if the number of personnel in all current carriages is less than the set threshold, and determine the personnel categories in all current carriages. The first anomaly recognition module is used to determine whether there is a cleaner and no passengers according to the personnel categories in all current carriages. If so, it is determined that the carriage cleaning operation conditions are met, the video stream of the cleaner in the cleaning area of each carriage is obtained through a camera, and the trained pose evaluation model is used to evaluate the video stream to determine whether the cleaning actions of the cleaner are normal. The second anomaly recognition module is used to verify whether the cleaning date recorded by the cleaner is abnormal based on the OCR recognition result and the time of the carriage management system.

9. An electronic device, characterized in that, It includes: A processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, the steps of the carriage cleaning anomaly recognition method according to any one of claims 1 to 7 are executed.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is run by the processor, the steps of the carriage cleaning anomaly recognition method according to any one of claims 1 to 7 are executed.

Images