A method and system for detecting the normalization of workshop wheelset operation based on image segmentation

Abstract

The present invention belongs to the technical field of wheelset operation, and specifically relates to a method and system for detecting the standardization of wheelset operation actions in a workshop based on image segmentation. The present invention obtains an initial image set of on-site workers during operation, processes each image in the initial image set using an image segmentation method, and segments the workers, work areas, and work-related tools in each image. A screening is performed based on the segmentation results to retain images in the work area of ​​the initial image set where both workers and work-related tools exist, thereby obtaining a primary image set; extracts the human skeleton key point information of the workers in each image in the primary image set, and filters out images in which the workers' actions are obviously not work actions based on the human skeleton key point information, thereby obtaining a secondary image set; and uses each image in the secondary image set to perform action standardization detection of wheelset operations in the workshop. The problem of low detection accuracy of existing work standardization detection methods is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of wheelset operation, and in particular relates to a method and system for detecting the standardization of wheelset operation actions in a workshop based on image segmentation. Background Art

[0002] During the wheelset operation in the workshop, operators perform standardized operations according to the process, which has a great impact on production safety and production efficiency; according to the workshop wheelset assembly operation standards, operators need to complete the assembly operations in sequence according to the eight-step operation process. For process standardization detection, the operation steps such as releasing the wheel stopper after the wheel falls, manual torque review, and measuring the boss gap can be detected through action recognition. Therefore, it is very important to detect the standardization of the operator's operating actions. Traditionally, the detection of the operator's operating actions is carried out by professional technicians going to the site to assess the operators, or by irregular on-site inspections to detect the standardization of the operator's operating actions; although this can achieve the purpose of effective detection, it relies too much on the experience of technicians, and each inspection requires technicians to go to the site, which consumes too much manpower.

[0003] The existing Chinese invention professional application with application publication number CN118115935A discloses a method and device for identifying abnormal behavior. The method obtains a list of actions corresponding to the complete operation process of the operator by obtaining a video clip containing an operator performing an operation and a detection area; and identifies the operator's abnormal actions based on the action list and the standard operation process. Although this method can achieve on-site detection without relying on technical personnel, under normal working conditions, when the operator is performing an action, there are often other actions that do not belong to the operation action, and in the video clip of the operator performing the operation, there will also be interference from the actions of multiple people in the non-operation area. These influencing factors will reduce the accuracy of the judgment of the operator's actions, and the detection accuracy is low. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for detecting the standardization of wheelset operation in a workshop based on image segmentation, so as to solve the problem of low detection accuracy of existing methods for detecting standardization of operation.

[0005] In order to solve the above technical problems, the present invention provides a method for detecting the normalization of wheelset operation actions in a workshop based on image segmentation, the method comprising: An initial image set of on-site workers during work is obtained, and each image in the initial image set is processed using an image segmentation method to segment the workers, work areas, and work-related tools in each image, and a screening is performed based on the segmentation results to retain images in the work area of ​​the initial image set where both workers and work-related tools exist, thereby obtaining a primary image set; human skeleton key point information of the workers in each image in the primary image set is extracted, and images in which the workers' actions are obviously not work actions are screened out based on the human skeleton key point information, thereby obtaining a secondary image set; Each image in the secondary image set is used to perform a motion standardization inspection on the wheelset operation in the workshop.

[0006] Furthermore, the image segmentation method is used to process each image in the initial image set as follows: each image in the initial image set is input into a target segmentation model, and the segmentation results of the workers, work areas and work-related tools are output; wherein the target segmentation model is obtained by training a deep learning model using a data set, and the data set is an image set containing the segmentation results of the workers, work areas and work-related tools.

[0007] Furthermore, the image segmentation method is used to process each image in the initial image set in the following manner: input each image in the initial image set into a work area target segmentation model to identify the work area in each image in the initial image set, then input the initial image set into a worker target segmentation model to identify the worker in each image in the initial image set, and filter out images in the initial image set where the worker is not in the work area; then input each image in the filtered initial image set into a work tool target segmentation model to identify the work tool in each image in the initial image set, and filter out images in the initial image set where the work tool is not in the work area, thereby obtaining an image set; Among them, the work area target segmentation model is obtained by training the deep learning model using the work area image dataset, and the work area image dataset is an image set containing the work area segmentation results; the operator target segmentation model is obtained by training the deep learning model using the operator image dataset, and the operator image dataset is an image set containing the operator segmentation results; the work tool target segmentation model is obtained by training the deep learning model using the work tool image dataset, and the work tool image dataset is an image set containing the work tool segmentation results.

[0008] Furthermore, when the operation is to place the wheel stopper, the method of filtering out images in which the operator's actions obviously do not belong to the operation actions based on the human skeleton key point information is as follows: if the angle between the operator's waist bone key points and the leg bone key points in a certain image in the extracted image set is different from the angle between the operator's waist bone key points and the leg bone key points during the operator's standard operation by more than a set degree threshold, then the image is determined to be an image in which the operator's actions obviously do not belong to the operation actions.

[0009] Furthermore, when the operation is manual torque review, the method of filtering out images in which the operator's actions obviously do not belong to the operation actions based on the human skeleton key point information is as follows: if the human skeleton key points of the operator's hands in a certain image in the extracted image set are not within the area at one end of the work tool, or the angle between the corresponding skeleton key points of the operator's two arms is greater than the set angle threshold, then the image is determined to be an image in which the operator's actions obviously do not belong to the operation actions.

[0010] Furthermore, when the operation is to measure the boss gap, the method of filtering out images in which the operator's movements are obviously not work movements based on the human skeleton key point information is as follows: if the operator in an image in the extracted image set is not in a sitting posture or any hand key point is not in the corresponding work area, then the image is determined to be an image in which the operator's movements are obviously not work movements; wherein, if the angle between the skeleton key points of the waist and thigh is less than the set sitting angle threshold and the center position of the hip skeleton key point is in the circumscribed rectangular area of ​​the stool on which the operator is sitting in the image, then the operator is determined to be in a sitting posture.

[0011] Furthermore, the method of using each image in the secondary image set to detect the action standardization of the workshop wheelset operation is: extracting the human skeleton key point information of the operator in each image in the secondary image set, inputting the extracted human skeleton key point information into the action standard classification model, and outputting the result of whether the operator's working action is standard; wherein the action standard classification model is obtained by training a neural network model using a data set, and the data set contains the human skeleton key point information of the operator when performing standardized operations and the corresponding classification labels of whether they are standard.

[0012] Furthermore, the neural network model is a dual-stream adaptive graph convolutional network model, an AGCN model or an ST-GCN model.

[0013] Furthermore, the target segmentation model is a SegNet image segmentation model.

[0014] In order to solve the above technical problems, the present invention also provides a workshop wheelset operation action standardization detection system based on image segmentation, including a processor, which executes a computer program to implement the steps of the workshop wheelset operation action standardization detection method based on image segmentation as mentioned above.

[0015] The beneficial effects are as follows: the present invention is an improved invention creation. Unlike the prior art, the present invention no longer directly uses the captured images to perform behavioral normative detection. Instead, after obtaining the images of the scene, it first performs preprocessing to filter out interference information of other personnel and operators' actions that are not operating actions; specifically, an image segmentation method is used to process each image in the initial image set of the acquired on-site operators during the operation, and the operators, operating areas and operation-related tools in each image are segmented, and a screening is performed according to the segmentation results to retain images in which operators and operation-related tools exist in the operating areas of the initial image set, thereby obtaining a primary image set; the human skeleton key point information of the operators in each image in the primary image set is extracted, and images in which the operators' actions are obviously not operating actions are filtered out according to the human skeleton key point information, thereby obtaining a secondary image set; and each image in the secondary image set is used to perform action normative detection of the workshop wheelset operation. The present invention performs two screenings after acquiring the initial image set, thereby avoiding the influence of the actions of multiple personnel in non-operating areas and the non-operating actions of operating personnel on the accuracy of the final detection results, improving the accuracy and efficiency of detecting the normativeness of operating personnel's behavior, and issuing timely warnings when irregular behaviors are detected to avoid the occurrence of operating accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a method for detecting the normalization of wheelset operation actions in a workshop based on image segmentation according to an embodiment of the present invention; Figure 2~Figure 6 It is a diagram of on-site operations of operators at different time points in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention obtains an initial image set of on-site workers during operation, processes each image in the initial image set using an image segmentation method, segments the workers and work-related tools in each image, performs a screening based on the segmentation results, and retains images in the work area of ​​the initial image set where both workers and work-related tools exist, thereby obtaining a primary image set; extracts the human skeleton key point information of the workers in each image in the primary image set, and screens out images whose movements of the workers are obviously not work movements based on the human skeleton key point information, thereby obtaining a secondary image set; and uses each image in the secondary image set to perform action normative detection of workshop wheelset operations. The problem of low detection accuracy of existing work normative detection methods is solved.

[0018] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0019] Method Example: In this embodiment, a method for detecting the standardization of wheelset operation in a workshop based on image segmentation is used. Taking the action of releasing the wheel stopper in the wheelset operation in the workshop as an example, the standardization of the operation of the operator is detected through the action standardization detection method. The principles of the method and steps for detecting other operation actions in the wheelset operation in the workshop are similar, such as manual torque review, measuring boss clearance and other operation actions. Figure 1 As shown, the detection method specifically comprises the following steps: Step 1: Obtain an initial set of images during the field operator's work.

[0020] This embodiment obtains the image of the on-site worker by intercepting a segment of the on-site monitoring video and obtaining the image of the on-site worker from the intercepted segment. As other implementations, the acquisition method can be selected according to actual needs, such as taking the image by a camera.

[0021] Step 2: Screen the images obtained in step 1 to obtain an image set, and use an image segmentation method to segment the images of the workers, work areas, and work-related tools in each image of the initial image set. For example, use image segmentation methods such as SegNet and Segformer to perform image segmentation; perform a screening based on the segmentation results to retain the images in the work area of ​​the initial image set where both workers and work-related tools exist, thereby obtaining an image set; the specific screening methods include: Method 1: First use the trained work area target segmentation model to identify the work area in each image in the initial image set, then input the initial image set into the trained operator target segmentation model for initial screening of the initial image set, and input the images with operators in the screened work area into the trained work tool target segmentation model for further screening, thus obtaining an image set. Specifically: ① Identify the working area in each image in the initial image set through the trained working area target segmentation model; ② Based on the initial image set after the work area is identified in step ①, filter out the interference images where the workers have not entered the work area; after preprocessing the acquired images, filter out the interference images that are not in the work area (such as Figure 2 The image of the operator is identified in the frame top corner area in the image, and no further analysis is performed; wherein the data set used to train the operator target segmentation model is an image set containing the operator segmentation results; ③ According to the atlas after the screening in step ②, the images in which the operator is in the working area but the work-related tools are not in the working area are screened out through the trained work tool target segmentation model, and the next step of analysis is not performed to obtain a primary image set. Among them, the data set used to train the work tool target segmentation model is an image set containing the work tool segmentation results.

[0022] In addition, as another implementation method, the screening order of steps ② and ③ can be set according to actual needs, that is, images in which there is no working tool in the working area can be screened out first.

[0023] Method 2: Use the trained target segmentation model to screen the initial image set to retain the images in the initial image set where both workers and work-related tools exist in the work area, thereby obtaining an image set; wherein the data set for training the target segmentation model is an image set containing the segmentation results of workers, work areas, and work-related tools.

[0024] Among them, the work area target segmentation model, the operator target segmentation model, the work tool target segmentation model and the target segmentation model are all deep learning models. The dataset used to train the work area target segmentation model is an image set containing the work area segmentation results; the dataset used to train the operator target segmentation model is an image set containing the operator segmentation results; the dataset used to train the work tool target segmentation model is an image set containing the work tool segmentation results; the dataset used to train the target segmentation model is an image set containing the operator, work area and work-related tool segmentation results.

[0025] like Figure 3 , Figure 4 As shown, it is one of the images of the operator placing the wheel stopper. In this image, the operator is in the working area but the wheel stopper of the working tool is still placed in the tool placement area, so this image is filtered out and not analyzed in the next step. As another embodiment, when the operator performs the working action of measuring the boss gap, if the working tool vernier caliper is not identified in the working area, this image is filtered out and not analyzed in the next step.

[0026] Different samples are used to train the deep learning model according to different work actions. For example, if the work action is manual torque review, the trained deep learning model is used to screen to retain images with both workers and work tool wrenches in the work area; the data set used to train the model is an image set containing the segmentation results of the work area, workers, and work tool wrenches; if the work action is measuring the boss gap, the data set for training the model is an image set containing the segmentation results of the work area, workers, and work tool gap measuring ruler.

[0027] Step 3: Perform secondary screening on the primary image set obtained in step 2 to obtain a secondary image set, and screen non-operational actions in combination with the key point information of the human skeleton, including: The information of the relevant human skeleton key points when the operator is working in each image is obtained, the angle information between the relevant skeleton key points is calculated, and compared with the angle information between the relevant skeleton key points when the operator is working. If the degree of difference is greater than the set degree threshold, it is determined that the image is an image in which the operator's action is obviously not a work action.

[0028] When detecting whether the operator's action is the action of releasing the wheel stopper, the difference between the angles of the operator's waist bone key points and the leg bone key points in the image and the angles of the operator's waist bone key points and the leg bone key points during standard operation is used to determine whether the image is an image of the operator performing a working action; if the angles of the operator's waist bone key points and the leg bone key points in the image are 30% greater or less than the angles of the operator's waist bone key points and the leg bone key points during standard operation (i.e., the set degree threshold), then it is determined that the image is not an image of the operator performing a working action, which means that the image is an image in which the operator's action is obviously not a working action. Figure 5 As shown, this is one of the images of the operator placing the wheel stopper. It can be clearly seen from the figure that the operator's action is not a bending posture, so this image is filtered out.

[0029] Step 4: Based on the secondary image set obtained in step 3, the standardization and type of actions in the work area are detected to determine whether the operator's work actions are standardized.

[0030] like Figure 6As shown in the figure, the action of the operator in the secondary image set is an image of the operation action. The sequence skeleton point information of the operator's bending action in each image is extracted to distinguish the behavior category, so as to further clarify whether the operator has bent down to release the wheel stopper. In this embodiment, the extracted sequence skeleton point information is detected by using a two-stream adaptive graph convolutional network (Two-Stream Adaptive Graph Convolutional Networks, 2s-AGCN) to distinguish different behavior categories. As other implementation methods, other neural network models can be selected according to actual needs, such as AGCN model (Adaptive Graph Convolutional Network), ST-GCN model (Spatial Temporal Graph Convolutional Network), etc. The human skeleton key point information when the operator performs standardized operations and the corresponding classification labels of whether they are standardized are used as training data sets, and the two-stream adaptive graph convolution network (i.e., the action standard classification model) is trained to extract the human skeleton key point information of each image in the secondary image set and input it into the trained two-stream adaptive graph convolutional network, and the result of whether the operator's operation action is standardized is obtained by output. Among them, the action standard classification model is obtained by training the neural network model using a data set, which contains the key point information of the human skeleton when the operator performs standardized operations and the corresponding classification labels of whether it is standardized.

[0031] In addition, the method for detecting the standardization of work movements of the present invention can not only detect the standardization of each single work movement, but also distinguish different work categories based on the key point information of the human skeleton and related work tools when completing related work movements of different work categories; in addition, by processing the acquired images in chronological order, the order in which the workers complete a group of jobs can be identified, and it can be detected whether the workers perform the work in accordance with standard work procedures and standard work movements.

[0032] For example, if the working action is manual torque review, the trained deep learning model is used to screen the original image set to retain images in which both workers and working tool wrenches are present in the working area; the data set used to train the model is an image set containing the segmentation results of the working area, workers, and working tool wrenches; then the images in the image set that are not working actions are screened out, and when the key points of the worker's hands are within a section of the wrench and the angle between the corresponding key points of each arm is greater than 20° (that is, the set angle threshold), it is determined that the image is not an image in which the worker performs a working action; then the skeletal key point information of each image after screening is extracted and input into the trained action specification classification model to determine whether the working action is standard; the training data of the action specification classification model is the human skeletal key point information of the standard operation of the worker performing the manual torque review working action.

[0033] If the working action is to measure the boss gap, the trained deep learning model is used to filter the original image set to retain the images in which both the operator and the working tool gap measuring ruler are present in the working area; the data set used to train the model is an image set containing the segmentation results of the working area, the operator and the working tool gap measuring ruler; then the images in the image set that are not working actions are filtered out, and some non-working postures are filtered out when the operator is sitting, any hand is raised and the hand key point is in the axle box area; then the skeletal key point information of each image after filtering is extracted and input into the trained action standard classification model to determine whether the action is standard; the training data of the action standard classification model is the human skeletal key point information of the standard working action of the operator measuring the boss gap. Among them, whether the included angle of the bone key points between the waist and thigh is less than 120° can be used to judge whether the operator is in a non-standing state, and the position relationship between the person and the stool can be judged according to whether the center position of the hip key points on both sides is within the circumscribed rectangular area of ​​the stool, so as to realize the judgment of whether the operator is in a sitting posture, that is, when the included angle of the bone key points between the waist and thigh is less than 120° and the center position of the hip bone key points is within the circumscribed rectangular area of ​​the stool, the operator is judged to be in a sitting posture.

[0034] The invention discloses a method for detecting the standardization of wheelset operation actions in a workshop based on image segmentation, image recognition and detection of key points of the human skeleton of operators during operation, so as to complete the detection of the standardization of the operators' actions, conduct real-time monitoring of missed operations and irregular behaviors in the operation process, and issue timely warnings for missed operations and irregular behaviors in the operation process after the operation is completed, thereby improving the standardization of operation actions and avoiding operation accidents caused by missed operations or irregular operations.

[0035] System Example: The present invention provides a workshop wheelset operation action standardization detection system based on image segmentation, including a memory, a processor and an internal bus, and the processor and the memory communicate and exchange data with each other through the internal bus. The memory includes at least one software function module stored in the memory, and the processor executes various functional applications and data processing by running the computer programs and modules stored in the memory, thereby realizing a workshop wheelset operation action standardization detection method based on image segmentation introduced in the method embodiment of the present invention. The principle, implementation process, and achievable effects of the method have been fully introduced in the method embodiment and will not be repeated here.

Claims

1. A method for detecting the standardization of wheelset operation in a workshop based on image segmentation, characterized in that: The method includes: An initial image set of on-site workers during work is obtained, and each image in the initial image set is processed using an image segmentation method to segment the workers, work areas, and work-related tools in each image, and a screening is performed based on the segmentation results to retain images in the work area of ​​the initial image set where both workers and work-related tools exist, thereby obtaining a primary image set; human skeleton key point information of the workers in each image in the primary image set is extracted, and images in which the workers' actions are obviously not work actions are screened out based on the human skeleton key point information, thereby obtaining a secondary image set; Each image in the secondary image set is used to perform a motion standardization inspection on the wheelset operation in the workshop.

2. The method for detecting the normalization of wheelset operation in a workshop based on image segmentation according to claim 1 is characterized in that: The method of using the image segmentation method to process each image in the initial image set is: input each image in the initial image set into a target segmentation model, and output the segmentation results of the workers, work areas and work-related tools; wherein the target segmentation model is obtained by training a deep learning model using a data set, and the data set is an image set containing the segmentation results of the workers, work areas and work-related tools.

3. The method for detecting the normalization of wheelset operation in a workshop based on image segmentation according to claim 1 is characterized in that: The method of using the image segmentation method to process each image in the initial image set is as follows: each image in the initial image set is input into a work area target segmentation model to identify the work area in each image in the initial image set, and then the initial image set is input into a worker target segmentation model to identify the worker in each image in the initial image set, and images in the initial image set where the worker is not in the work area are filtered out; each image in the filtered initial image set is then input into a work tool target segmentation model to identify the work tool in each image in the initial image set, and images in the initial image set where the work tool is not in the work area are filtered out, thereby obtaining an image set; Among them, the work area target segmentation model is obtained by training the deep learning model using the work area image dataset, and the work area image dataset is an image set containing the work area segmentation results; the operator target segmentation model is obtained by training the deep learning model using the operator image dataset, and the operator image dataset is an image set containing the operator segmentation results; the work tool target segmentation model is obtained by training the deep learning model using the work tool image dataset, and the work tool image dataset is an image set containing the work tool segmentation results.

4. The method for detecting the normalization of wheelset operation in a workshop based on image segmentation according to claim 1 is characterized in that: When the operation is to place the wheel stopper, the method of filtering out images in which the operator's actions are obviously not work actions based on the human skeleton key point information is as follows: if the angle between the operator's waist bone key points and the leg bone key points in a certain image in the extracted image set is different from the angle between the operator's waist bone key points and the leg bone key points during the operator's standard operation by more than a set degree threshold, then the image is judged as an image in which the operator's actions are obviously not work actions.

5. The method for detecting the normalization of wheelset operation in a workshop based on image segmentation according to claim 1 is characterized in that: When the operation is manual torque review, the method of filtering out images in which the operator's movements are obviously not work movements based on the human skeleton key point information is as follows: if the human skeleton key points of the operator's hands in a certain image extracted from the primary image set are not within the area at one end of the work tool, or the angle between the corresponding skeleton key points of the operator's two arms is greater than the set angle threshold, then the image is determined to be an image in which the operator's movements are obviously not work movements.

6. The method for detecting the normalization of wheelset operation in a workshop based on image segmentation according to claim 1 is characterized in that: When the operation is to measure the boss gap, the method of filtering out images in which the operator's movements are obviously not work movements based on the human skeleton key point information is as follows: if the operator in an image in the extracted image set is not in a sitting posture or any hand key point is not in the corresponding work area, then the image is determined to be an image in which the operator's movements are obviously not work movements; wherein, if the angle between the skeleton key points of the waist and thigh is less than the set sitting angle threshold and the center position of the hip skeleton key point is in the circumscribed rectangular area of ​​the stool on which the operator is sitting in the image, then the operator is determined to be in a sitting posture.

7. The method for detecting the normalization of wheelset operation in a workshop based on image segmentation according to claim 1 is characterized in that: The method of using each image in the secondary image set to detect the action standardization of the workshop wheelset operation is: extracting the human skeleton key point information of the operator in each image in the secondary image set, inputting the extracted human skeleton key point information into the action standard classification model, and outputting the result of whether the operator's working action is standard; wherein the action standard classification model is obtained by training a neural network model using a data set, and the data set contains the human skeleton key point information of the operator when performing standardized operations and the corresponding classification labels of whether they are standard.

8. The method for detecting the normalization of wheelset operation in a workshop based on image segmentation according to claim 7 is characterized in that: The neural network model is a dual-stream adaptive graph convolutional network model, an AGCN model or an ST-GCN model.

9. The method for detecting the normalization of wheelset operation in a workshop based on image segmentation according to claim 2 is characterized in that: The target segmentation model is a SegNet image segmentation model.

10. A workshop wheelset operation action standardization detection system based on image segmentation, comprising a processor, characterized in that: The processor executes a computer program to implement the steps of the method for detecting the standardization of workshop wheelset operation actions based on image segmentation as described in any one of claims 1 to 9.

Images