Monitoring method, device, equipment, storage medium and vehicle

Abstract

The invention discloses a monitoring method, device and equipment, a storage medium and a vehicle. The method comprises the steps that under the condition that a body-in-white is located at a preset position, multiple frames of first images are acquired, the first images comprise a target user and the body-in-white, the target user is a user performing quality detection on the body-in-white, and the body-in-white comprises multiple preset detection areas; performing human body key point detection on the target user in the multiple frames of first images to obtain key point information corresponding to the target user; performing association analysis on the multiple pieces of key point information to obtain multiple actions and behaviors of the target user; and when the plurality of action behaviors do not meet a preset condition, outputting early warning information, the preset condition being that the plurality of action behaviors comprise a preset detection behavior, and in each preset detection area, the occurrence frequency of the preset detection behavior is not less than a preset threshold. According to the monitoring method provided by the embodiment of the invention, the standardization of the target user on the quality detection of the body in white can be improved.

Description

Technical Field

[0001] The present application belongs to the field of monitoring technology, and in particular, relates to a monitoring method, device, equipment, storage medium and vehicle. Background Art

[0002] Usually, after the body-in-white is welded, the staff needs to conduct a quality inspection on the body-in-white. Only when the quality of the body-in-white meets the standard will the next operation process be carried out. However, if the body-in-white is released when it does not meet the standard, it will lead to subsequent repairs and rework, increasing the production cost of the vehicle. Therefore, the quality inspection of the body-in-white is very important.

[0003] In actual situations, due to the high cycle time of the welding workshop, that is, the frequency of testing multiple BIWs is high, it is easy for the staff to make inaccurate test results due to omissions and irregular testing actions, which in turn increases the production cost of the vehicle. Therefore, it is very important to monitor the staff's actions during the quality inspection of the BIW.

[0004] Currently, there is no monitoring method for monitoring the actions and behaviors of workers, resulting in low accuracy in quality inspection of the body-in-white, which in turn leads to higher production costs for the vehicle. Summary of the invention

[0005] The embodiments of the present application provide a monitoring method, device, equipment, storage medium and vehicle, which can improve the standardization of the quality inspection of the body-in-white by the target user, thereby improving the accuracy of the quality inspection of the body-in-white and reducing the production cost of the vehicle.

[0006] In a first aspect, an embodiment of the present application provides a monitoring method, the method comprising:

[0007] When the BIW is at a preset position, a plurality of first image frames are acquired, wherein the first image includes a target user and the BIW, the target user is a user who performs quality inspection on the BIW, and the BIW includes a plurality of preset inspection areas;

[0008] Performing body key point detection on the target user in the multiple frames of the first image respectively to obtain key point information corresponding to the target user;

[0009] Performing correlation analysis on the multiple key point information to obtain multiple action behaviors of the target user;

[0010] When the multiple action behaviors do not meet the preset conditions, a warning message is output, wherein the preset conditions are that the multiple action behaviors include preset detection behaviors, and in each of the preset detection areas, the number of occurrences of the preset detection behaviors is not less than a preset threshold.

[0011] In a possible implementation, the key point information includes hand key point information, and after obtaining the multiple action behaviors of the target user, the method further includes:

[0012] For each of the action behaviors, when the action behavior is a preset detection behavior, determine the hand key point information corresponding to the action behavior to obtain at least one of the hand key point information;

[0013] Mapping at least one of the hand key point information to a first canvas including the multiple preset detection areas, wherein the multiple preset detection areas in the first canvas are the same as the multiple preset detection areas in the first image, and the size of the first canvas is the same as the size of the first image;

[0014] When the plurality of action behaviors do not meet the preset conditions, outputting warning information includes:

[0015] In the first canvas, for each of the preset detection areas, when the amount of the hand key point information in the preset detection area is less than the preset threshold, a warning message is output.

[0016] In a possible implementation manner, before mapping the at least one hand key point information to the first canvas including the plurality of preset detection areas, the method further includes:

[0017] The plurality of preset detection areas in the first image are mapped into the first canvas.

[0018] In a possible implementation, the performing correlation analysis on the plurality of key point information to obtain the plurality of action behaviors of the target user includes:

[0019] For each frame of the first image, mapping the key point information to a second canvas having the same size as the first image, to obtain multiple frames of second images;

[0020] Acquire n consecutive frames of images from the multiple frames of second images, where n is a positive integer;

[0021] Comparing the key point information in the n frames of image to obtain changes of the n key point information over time;

[0022] The key point information in the target image is classified according to the changes of the n key point information over time to obtain the action behavior of the target user in the target image, and the target image is the first frame image in the n frames of images.

[0023] In a possible implementation manner, before acquiring n consecutive frames of images from the multiple frames of second images, the method further includes:

[0024] For each frame of the second image, obtaining a target area including the key point information;

[0025] Cropping the second image according to the target area to obtain multiple frames of third images;

[0026] Scaling the multiple frames of third images to the same size by bilinear interpolation;

[0027] The step of acquiring n consecutive frames of images from the plurality of frames of second images comprises:

[0028] Continuous n frames of images are acquired from the plurality of third image frames having the same size.

[0029] In a possible implementation manner, acquiring n consecutive frames of images from the plurality of third frames of images having the same size includes:

[0030] For the multiple frames of third images with the same size, the sliding window is moved according to a preset step length to divide the multiple frames of third images into a plurality of image groups, each of which includes n frames of images.

[0031] In a possible implementation manner, the performing human body key point detection on the target user in the multiple frames of the first image respectively includes:

[0032] For each frame of the first image, when the first image includes multiple users, human body key point detection is performed on the multiple users respectively, and the multiple users include the target user.

[0033] In a possible implementation manner, after performing body key point detection on the multiple users respectively, the method further includes:

[0034] Obtaining the distance between each of the users and the BIW to obtain a plurality of distances;

[0035] determining a minimum distance among the plurality of distances;

[0036] A user corresponding to the minimum distance is determined as the target user.

[0037] In a possible implementation, acquiring multiple frames of first images includes:

[0038] Get surveillance video stream;

[0039] Frame extraction is performed on the monitoring video stream to obtain multiple frames of first images.

[0040] In a second aspect, an embodiment of the present application provides a monitoring device, the device comprising:

[0041] An acquisition module, used for acquiring a plurality of frames of first images when the BIW is at a preset position, wherein the first images include a target user and the BIW, the target user is a user who performs quality inspection on the BIW, and the BIW includes a plurality of preset inspection areas;

[0042] A detection module, used to perform human body key point detection on the target user in the multiple frames of the first image to obtain key point information corresponding to the target user;

[0043] An analysis module, used to perform correlation analysis on the plurality of key point information to obtain a plurality of action behaviors of the target user;

[0044] An output module is used to output warning information when the multiple action behaviors do not meet the preset conditions. The preset conditions are that the multiple action behaviors include preset detection behaviors, and in each of the preset detection areas, the number of occurrences of the preset detection behaviors is not less than a preset threshold.

[0045] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;

[0046] When the processor executes the computer program instructions, the processor implements any possible implementation method of the first aspect described above.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements a method in any possible implementation method of the first aspect described above.

[0048] In a fifth aspect, an embodiment of the present application provides a vehicle, the vehicle comprising at least one of the following:

[0049] A monitoring device as in any one of the embodiments of the second aspect;

[0050] An electronic device as in any one of the embodiments of the third aspect;

[0051] A computer-readable storage medium as in any embodiment of the fourth aspect.

[0052] The monitoring method, device, equipment, storage medium and vehicle of the embodiment of the present application can accurately obtain the target user's action behavior in the process of the target user performing quality inspection on the body-in-white by acquiring multiple frames of first images including the target user and the body-in-white when the body-in-white is in a preset position, performing human body key point detection on the target user in the multiple frames of first images respectively, obtaining key point information corresponding to the target user, and performing correlation analysis on multiple key point information to obtain multiple action behaviors of the target user. By outputting warning information when multiple action behaviors do not meet preset conditions, that is, by monitoring multiple action behaviors of the target user, when multiple action behaviors do not include preset detection behaviors, or multiple action behaviors include preset detection action behaviors, but for each preset detection area, the number of occurrences of the preset detection behavior in the preset detection area is less than the preset threshold, outputting warning information, it is possible to improve the standardization of the target user's quality inspection on the body-in-white, thereby improving the accuracy of the quality inspection on the body-in-white and reducing the production cost of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 It is a flow chart of a monitoring method provided in an embodiment of the present application;

[0055] Figure 2 is a schematic diagram of a body-in-white provided in an embodiment of the present application;

[0056] Figure 3 is a schematic diagram of a second image provided in an embodiment of the present application;

[0057] Figure 4 is a schematic diagram of a third image provided in an embodiment of the present application;

[0058] Figure 5 is a schematic diagram of performing action recognition through a sliding window provided in an embodiment of the present application;

[0059] Figure 6 It is a flow chart of another monitoring method provided in an embodiment of the present application;

[0060] Figure 7 It is a flow chart of another monitoring method provided in an embodiment of the present application;

[0061] Figure 8 is a structural schematic diagram of a monitoring device provided in an embodiment of the present application;

[0062] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0064] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0065] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0066] In actual situations, a welding workshop may include multiple workstations. When the BIW is in a designated position, the left and right sides of the BIW may correspond to two different workstations. In each workstation, a worker may perform quality inspection on the BIW. Specifically, the process of quality inspection on the BIW may be a process in which the worker touches the BIW with his hands and determines whether there are defects in the BIW based on the touch results.

[0067] Based on this, in order to solve the problems of the prior art, the embodiments of the present application provide a monitoring method, device, equipment, storage medium and vehicle. Among them, the monitoring method can be applied to the Manufacturing Execution System (MES). The MES system is a production information management system for the execution layer of the workshop of a manufacturing enterprise.

[0068] The following first introduces the monitoring method provided in the embodiments of the present application.

[0069] Figure 1FIG. 1 is a flow chart of a monitoring method provided in an embodiment of the present application. Figure 1 As shown, the monitoring method provided in the embodiment of the present application includes the following steps:

[0070] S110, when the BIW is at a preset position, acquiring a plurality of first image frames, the first image including a target user and the BIW, the target user being a user who performs quality inspection on the BIW, and the BIW including a plurality of preset inspection areas;

[0071] S120, performing body key point detection on the target user in the multiple frames of the first image to obtain key point information corresponding to the target user;

[0072] S130, performing correlation analysis on multiple key point information to obtain multiple action behaviors of the target user;

[0073] S140. Outputting warning information when multiple action behaviors do not meet preset conditions. The preset conditions are that the multiple action behaviors include preset detection behaviors, and in each preset detection area, the number of occurrences of the preset detection behaviors is not less than a preset threshold.

[0074] The monitoring method of the embodiment of the present application obtains multiple frames of first images including the target user and the body-in-white when the body-in-white is in a preset position, performs human body key point detection on the target user in the multiple frames of the first images respectively, obtains key point information corresponding to the target user, and performs correlation analysis on multiple key point information to obtain multiple action behaviors of the target user, so that the action behaviors of the target user can be accurately obtained during the process of the target user performing quality inspection on the body-in-white. By outputting warning information when multiple action behaviors do not meet preset conditions, that is, by monitoring multiple action behaviors of the target user, when multiple action behaviors do not include preset detection behaviors, or multiple action behaviors include preset detection action behaviors, but for each preset detection area, the number of occurrences of the preset detection behavior in the preset detection area is less than the preset threshold, outputting warning information, it is possible to improve the standardization of the quality inspection of the body-in-white by the target user, thereby improving the accuracy of the quality inspection of the body-in-white and reducing the production cost of the vehicle.

[0075] The specific implementation methods of the above steps are introduced below.

[0076] In some embodiments, in S110, the body-in-white refers to a body that has been welded but not yet painted, according to the body terminology standard and the definition in the textbook. The preset position may be a preset position where the body-in-white undergoes quality inspection. The quality inspection of the body-in-white may be performed on a plurality of preset inspection areas of the body-in-white. Figure 2As shown, the plurality of preset detection areas may include, for example, a first preset detection area 21, a second preset detection area 22, and a third preset detection area 23. In addition, the first image may be a monitoring image corresponding to a target workstation, and the target workstation may be any one of the plurality of workstations. The target user may be a user who performs quality inspection on the body-in-white in the target workstation.

[0077] As an example, the MES system can detect the position of the body-in-white. When it is detected that the body-in-white reaches the preset position, the "start" signal can be triggered, that is, the MES system can start to monitor the action of the target user. The action of the target user can be used to perform quality inspection on the body-in-white. In addition, the first step to monitor the action of the target user can be to call the image acquisition device corresponding to the target workstation and obtain the monitoring video stream through the image acquisition device. The image acquisition device can be a color camera (RGB camera). After obtaining the monitoring video stream, the picture sequence frames in the monitoring video stream can be determined as multiple frames of first images.

[0078] Based on this, in order to improve the efficiency of action recognition, in some embodiments, the above S110 may specifically include:

[0079] Get surveillance video stream;

[0080] Frame extraction is performed on the monitoring video stream to obtain multiple frames of first images.

[0081] Here, frame extraction can be extracting a frame of image in the surveillance video stream every preset time. Since there is almost no difference between the continuous multiple frames of surveillance images in the surveillance video stream, the difference between the multiple frames of first images can be improved by extracting frames from the surveillance video stream. Furthermore, by performing action recognition based on the multiple frames of first images obtained after frame extraction, the efficiency of action recognition can be improved.

[0082] In some embodiments, in S120, the key point information may include coordinate information corresponding to a plurality of human key points. The plurality of human key points may include 19 preset key points of the head, neck, shoulder, elbow, hand, etc. Based on this, the key point information may include coordinate information corresponding to the left palm key point and coordinate information corresponding to the right palm key point of the target user.

[0083] As an example, the MES system may include a target detection model. The target detection model may be used to detect key points of a target user in the first image. Specifically, after inputting multiple frames of the first image into the target detection model, the target detection model may detect the multiple frames of the first image frame by frame, and extract key point information corresponding to each of the multiple frames of the first image.

[0084] As an example, before performing human key point detection, personnel identification can be performed through a target detection model to determine the number of users in the first image. The first image may not include any user, may include one user, or may include multiple users. If multiple frames of the first image do not include any user, the monitoring video stream can be acquired again, and the above steps can be repeated to obtain multiple frames of the first image. In the case where the first image includes one user, it can be determined that the user is the target user, and human key point detection can be performed on the target user. In the case where the first image includes multiple users, the target user can be first determined from the multiple users, and then the human key points of the target user can be detected. Since the first image is multiple frames, if the target user is repeatedly determined for each frame of the first image, the detection efficiency of the human key points will be reduced.

[0085] Based on this, in order to improve the detection efficiency of key points of the human body, in some embodiments, the above S120 may specifically include:

[0086] For each frame of the first image, when the first image includes multiple users, human body key point detection is performed on the multiple users respectively, and the multiple users include the target user.

[0087] Based on this, in some embodiments, after performing the human body key point detection on multiple users respectively, the following steps may also be included:

[0088] Obtain the distance between each user and the BIW to obtain multiple distances;

[0089] determining a minimum distance among a plurality of distances;

[0090] The user corresponding to the minimum distance is determined as the target user.

[0091] Here, in the case where multiple users are included in the first image, the multiple users may include a target worker and at least one non-target worker. According to the prior rule, since the target worker can perform quality inspection on the body-in-white, the target worker may be closest to the body-in-white. Therefore, among the multiple users, the user closest to the body-in-white may be the target user. After determining the target user, the Kalman filter may be used to track the target user. That is, in the embodiment of the present application, only the key point information of the target user may be focused on later.

[0092] In some embodiments, in S130, the action behavior may include "left hand working", "right hand working" and "not in working state", etc. Since the multiple frames of first images are extracted from the monitoring video stream, the multiple frames of first images may be arranged in chronological order.

[0093] As an example, since the key point information includes coordinate information corresponding to multiple key points of the human body, multiple action behaviors of the target user can be determined by analyzing the changes in the key point information over time. For example, if the vertical coordinate corresponding to the left palm key point and the vertical coordinate corresponding to the left elbow key point in the key point information both show an upward trend over time, it can be determined that the target user is raising his left hand, and then the action behavior can be determined as "left hand working".

[0094] Based on this, in some embodiments, the above S130 may specifically include:

[0095] For each frame of the first image, mapping the key point information to a second canvas having the same size as the first image, to obtain multiple frames of the second image;

[0096] Acquire n consecutive frames of images from the plurality of frames of the second image, where n is a positive integer;

[0097] Compare the key point information in n frames of images to obtain the changes of n key point information over time;

[0098] The key point information in the target image is classified according to the change of the n key point information over time, and the action behavior of the target user in the target image is obtained. The target image is the first frame image in the n frames of images.

[0099] Here, if Figure 3 As shown, the second image may include key point information but not other information. The key point information may include coordinate information corresponding to a plurality of human key points. When drawing the human key points into the second canvas, in order to make the human key points more recognizable, the human key points may be drawn in the form of Gaussian distribution, that is, each human key point is subject to Gaussian distribution.

[0100] In addition, an action behavior can be determined by n key information. Among them, n can be 3, 4, 5, etc., for example, and is not limited here. In addition, the change of n key point information over time can include that the vertical coordinates corresponding to the n left palm key points all show an upward trend over time, the horizontal coordinates corresponding to the n left palm key points all show an upward trend over time, the vertical coordinates corresponding to the n left elbow key points all show an upward trend over time, the vertical coordinates corresponding to the n right palm key points do not change over time, and the horizontal coordinates corresponding to the n right palm key points do not change over time. In addition, the classification result obtained by classifying the key point information can be an action behavior.

[0101] As an example, if the n ordinates corresponding to the left palm key points all show an upward trend over time, the n abscissas corresponding to the left palm key points all show an upward trend over time, and the n ordinates corresponding to the left elbow key points all show an upward trend over time, it can be determined that the user is raising his left hand. Of course, the above is just an example. If any two of the above three conditions are met, it can also be determined that the user is raising his left hand, or it can be determined that the user is raising his left hand only after adding other conditions, which is not limited here. That is, the specific classification rules can be predetermined by the user. In this way, after obtaining the change of n key point information over time, it can be determined based on the predetermined classification rules according to the change of n key point information over time whether the target user is raising the left hand, lowering the left hand, raising the right hand, lowering the right hand, and the left and right hands are both not moving. If the target user is raising the left hand, lowering the left hand, etc., the action behavior of the target user can be classified as "left hand work". If the target user is raising the right hand, lowering the right hand, etc., the action behavior of the target user can be classified as "right hand work". If the target user's left and right hands are both motionless, the target user's action behavior may be classified as "not in working state".

[0102] As an example, the MES system may include an action recognition network. The action recognition network may perform action recognition based on key point information. Specifically, after multiple frames of second images are input into the action recognition network, the action recognition network may move the sliding window according to a preset step length m for the multiple frames of second images, starting from the first frame of the second image, and divide the multiple frames of second images into multiple image groups. Each image group may include n frames of images, and m may be 1, for example. That is, the action recognition network may assign the action behavior recognized by each sliding window to the first frame of the sliding window, and move the sliding window backward along the time series of the video stream with a step length of 1 until multiple frames of second images are assigned classification results (i.e., action behaviors). The classification result of the last sliding window may be assigned to all second images in the sliding window.

[0103] like Figure 3 As shown, the proportion of key point information in the second image is relatively small. If the action recognition network directly performs action recognition based on the second image, it will lead to a waste of computing resources and reduce the efficiency and accuracy of action recognition. Therefore, in order to improve the efficiency and accuracy of action recognition, in some embodiments, before obtaining continuous n frames of images in the multiple frames of the second image, it may also include:

[0104] For each frame of the second image, obtaining a target area including key point information;

[0105] Cropping the second image according to the target area to obtain multiple frames of third images;

[0106] The multiple frames of the third image are scaled to the same size by using bilinear interpolation;

[0107] Based on this, acquiring n consecutive frames of images from multiple frames of second images may specifically include:

[0108] Continuous n frames of images are acquired from a plurality of third image frames having the same size.

[0109] Here, the target area can be the area between the maximum x-axis coordinates and the minimum x-axis coordinates of all human key points. Specifically, for each frame of the second image, after obtaining the maximum x-axis coordinates and the minimum x-axis coordinates of all human key points, the second image can be cropped according to the size of the minimum rectangular frame surrounding all human key points to obtain a key point heat map with a channel number of 1 (i.e., the third image). The schematic diagram of the third image can be as follows Figure 4 shown.

[0110] In addition, when using the action recognition network to recognize multiple frames of third images, in order to ensure the success of action recognition, the multiple frames of third images can be first scaled to the same size, and then the multiple frames of third images with the same size are input into the action recognition network.

[0111] In this way, since the key point information accounts for a large proportion in the third image, obtaining n consecutive frames of images from multiple frames of third images of the same size, and then performing action recognition based on the n frames of images can improve the efficiency and accuracy of action recognition.

[0112] Based on this, in order to ensure the accuracy of action recognition while taking into account the real-time performance of action recognition and meet the rhythm and requirements in the factory environment, in some embodiments, the above-mentioned acquisition of n consecutive frames of images from multiple frames of third images of the same size may specifically include:

[0113] For multiple frames of third images with the same size, the sliding window is moved according to a preset step length to divide the multiple frames of third images into multiple image groups, each image group includes n frames of images.

[0114] Here, after inputting multiple frames of third images of the same size into the action recognition network, the action recognition network can move the sliding window according to a preset step length m for the multiple frames of third images, starting from the first frame of the third image, and divide the multiple frames of third images into multiple image groups. Each image group may include n frames of images, and m may be 1, for example. That is, the action recognition network can assign the action behavior recognized by each sliding window to the first frame of the sliding window, and move the sliding window backward along the time series of the video stream with a step length of 1 until multiple frames of third images are assigned classification results (i.e., action behaviors). The classification result of the last sliding window can be assigned to all third images in the sliding window.

[0115] As an example, a schematic diagram of action recognition using a sliding window can be as follows: Figure 5 As shown. Figure 5 In , the size of the sliding window can be 3, and the moving step can be 1. In addition, A can represent "left hand working", B can represent "right hand working", and C can represent "not in working state". That is, by performing action recognition on the first three frames of images, it can be determined that the action behavior corresponding to the first frame of image is "left hand working". By performing action recognition on the second to fourth frames of images, it can be determined that the action behavior corresponding to the second frame of image is "left hand working". By performing action recognition on the third to fifth frames of images, it can be determined that the action behavior corresponding to the third frame of image is "right hand working". By performing action recognition on the last three frames of images, it can be determined that the action behavior corresponding to the last three frames of images is "not in working state".

[0116] In this way, through the sliding window combined with the technical route of the action recognition network based on human key points, behavioral actions can be classified for unedited videos of any length, while ensuring the accuracy of action recognition and taking into account the real-time nature of action recognition to meet the rhythm and needs within the factory environment.

[0117] In some embodiments, in S140, the preset detection behavior may include "left hand working" and "right hand working". In addition, since the action behavior can correspond to the first image one by one, and the multiple frames of the first image are picture sequence frames obtained by extracting frames from the monitoring video stream, the number of times the preset detection behavior appears in the preset detection area can correspond to the target user's quality detection time for the preset detection area. In addition, the preset thresholds corresponding to different preset detection areas can be the same or different, which is not limited here.

[0118] As an example, if the multiple action behaviors do not include the preset detection behavior, it can be determined that the target user has not performed quality inspection on the body-in-white. If the multiple action behaviors include the preset detection behavior, and the preset detection behavior appears in the preset detection area, it can be determined that the target user has inspected the preset detection area of ​​the body-in-white. If the number of times the preset detection behavior appears in the preset detection area is not less than the preset threshold, it can be determined that the target user's inspection of the preset detection area of ​​the body-in-white is in compliance with the specification. If the number of times the preset detection behavior appears in the preset detection area is less than the preset threshold, it can be determined that the target user's inspection of the preset detection area of ​​the body-in-white is not in compliance with the specification.

[0119] As an example, the time that the BIW stays at a preset position may be fixed, so before the MES system detects that the BIW is about to leave the preset position (for example, 1 minute before the BIW leaves the preset position), it can determine whether multiple actions meet the preset conditions. If multiple actions do not meet the preset conditions, that is, if the target user does not detect the BIW, or the detection of a preset detection area of ​​the BIW does not meet the specification, an early warning message can be output in the MES system so that the management personnel can prevent the vehicle from being released in time.

[0120] In addition, regardless of whether multiple action behaviors meet preset conditions, multiple action behaviors can be summarized, saved in the MES system, and displayed in the MES system.

[0121] See also Figure 6 The monitoring method provided in the embodiment of the present application may further include the following steps after S130 shown in the above embodiment:

[0122] S610: for each action behavior, if the action behavior is a preset detection behavior, determine the hand key point information corresponding to the action behavior to obtain at least one hand key point information;

[0123] S620, mapping at least one hand key point information to a first canvas including a plurality of preset detection areas, wherein the plurality of preset detection areas in the first canvas are the same as the plurality of preset detection areas in the first image, and a size of the first canvas is the same as a size of the first image;

[0124] Based on this, the above S140 may specifically include:

[0125] S141. In the first canvas, for each preset detection area, when the amount of hand key point information in the preset detection area is less than a preset threshold, output warning information.

[0126] Therefore, by monitoring the target user's motion behavior based on fine-grained hand key point information in a complex factory environment, the accuracy and robustness of monitoring can be improved.

[0127] The specific implementation methods of the above steps are introduced below.

[0128] In some embodiments, in S610, if the action behavior is "left hand working", the coordinate information corresponding to the key points of the left palm can be extracted. If the action behavior is "right hand working", the coordinate information corresponding to the key points of the right palm can be extracted. If the action behavior is "not in working state", the key point information may not be extracted.

[0129] As an example, before extracting the hand key point information, a result queue may be constructed. The length of the result queue may be the same as the number of the first image. After extracting the hand key point information, the hand key point information may be placed in the result queue. If the key point information is not to be extracted, "None" may be placed in the result queue.

[0130] In some embodiments, in S620 , the first canvas may include a plurality of preset detection areas but not other information.

[0131] As an example, at least one hand key point information may be mapped to the first canvas in sequence according to the arrangement order in the result queue to determine the ownership of all hand key points (left palm key points or right palm key points) in the result queue. During the mapping process, if a hand key point falls within a preset detection area, the hand key point may belong to this preset detection area. If the hand key point does not fall into any preset detection area, the hand key point may not belong to any preset detection area.

[0132] Based on this, in some embodiments, before the above S620, the following may also be included:

[0133] A plurality of preset detection areas in the first image are mapped into the first canvas.

[0134] Here, before mapping multiple preset detection areas to the first canvas, the preset detection areas in the body-in-white and the category identifiers corresponding to the preset detection areas can be exported as a json file, and then each preset detection area in the json file is drawn in the first canvas. Among them, the category identifier can be used to distinguish different preset detection areas. The preset detection areas can include multiple categories such as the front hood and the rear hood.

[0135] In some embodiments, in S141, if no hand key points fall within a certain detection area, or the number of hand key points falling within is less than a preset threshold, it can be determined that the target user does not meet the specifications during the detection of the preset monitoring area, and a warning message can be output.

[0136] In addition, the other steps of the method in the embodiment of the present application can be seen above Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0137] In order to better describe the entire solution, some specific examples are given based on the above embodiments.

[0138] For example, Figure 7 As shown, a monitoring method provided in an embodiment of the present application may include the following steps:

[0139] S71, acquiring a plurality of frames of first images when it is detected that the BIW has reached a preset position;

[0140] S72, detecting the multiple frames of the first image frame by frame by using the target detection model to obtain at least one user in the first image and key point information corresponding to each user;

[0141] S73. In the case of multiple users, determine the target user using a priori rules, and track the target user using Kalman filtering;

[0142] S74, constructing a sliding window;

[0143] S75, performing correlation analysis on multiple key point information of the target user based on the sliding window to obtain multiple action behaviors;

[0144] S76, for each action behavior, if the action behavior is a preset detection behavior, determine the hand key point information corresponding to the action behavior, and put the hand key point information into a result queue;

[0145] S77, mapping a plurality of preset detection areas in the first image to the first canvas;

[0146] S78, traverse the result queue and map the hand key point information to the first canvas;

[0147] S79, judging whether the target user has passed the quality inspection of the preset detection area according to the number of hand key point information falling in each preset detection area;

[0148] S710. Display the monitoring results.

[0149] Therefore, by low-coupling the key points and action behaviors of the human body with the preset detection areas of the body in white, the portability and reusability of the algorithm can be improved. In addition, through the embodiments of the present application, the core algorithm problem of automatically monitoring the operation of workers is solved at low cost and low computational effort. By applying the embodiments of the present application to the relevant production lines, the degree of automation of the production lines can be improved, achieving the effect of reducing costs and increasing efficiency. Among them, the relevant production lines may include any workstations that require workers to touch their hands for inspection.

[0150] Based on the monitoring method provided in the above embodiment, the present application also provides a specific implementation of the monitoring device. Please refer to the following embodiment.

[0151] like Figure 8 As shown, the monitoring device 800 provided in the embodiment of the present application includes the following modules:

[0152] An acquisition module 810 is used to acquire a plurality of frames of first images when the BIW is at a preset position, wherein the first images include a target user and the BIW, the target user is a user who performs quality inspection on the BIW, and the BIW includes a plurality of preset inspection areas;

[0153] A detection module 820 is used to perform human body key point detection on the target user in the multiple frames of the first image to obtain key point information corresponding to the target user;

[0154] An analysis module 830 is used to perform correlation analysis on multiple key point information to obtain multiple action behaviors of the target user;

[0155] The output module 840 is used to output warning information when multiple action behaviors do not meet preset conditions. The preset conditions are that the multiple action behaviors include preset detection behaviors, and in each preset detection area, the number of occurrences of the preset detection behaviors is not less than a preset threshold.

[0156] The monitoring device 800 is described in detail below, as shown below:

[0157] In some embodiments, the key point information includes hand key point information. Based on this, the monitoring device 800 may further include:

[0158] A determination module, configured to, after obtaining a plurality of action behaviors of the target user, determine, for each action behavior, hand key point information corresponding to the action behavior if the action behavior is a preset detection behavior, and obtain at least one piece of hand key point information;

[0159] A first mapping module, configured to map at least one hand key point information to a first canvas including a plurality of preset detection areas, wherein the plurality of preset detection areas in the first canvas are the same as the plurality of preset detection areas in the first image, and the size of the first canvas is the same as the size of the first image;

[0160] Based on this, the output module 840 may specifically include:

[0161] The output submodule is used to output warning information in the first canvas for each preset detection area when the amount of hand key point information in the preset detection area is less than a preset threshold.

[0162] In some embodiments, the monitoring device 800 may further include:

[0163] The second mapping module is used to map multiple preset detection areas in the first image into the first canvas before mapping at least one hand key point information into the first canvas including the multiple preset detection areas.

[0164] In some embodiments, the analysis module 830 may specifically include:

[0165] A mapping submodule, for mapping key point information to a second canvas having the same size as the first image for each frame of the first image, to obtain multiple frames of the second image;

[0166] A first acquisition submodule is used to acquire n consecutive frames of images from multiple frames of second images, where n is a positive integer;

[0167] A comparison submodule is used to compare the key point information in n frames of images to obtain the changes of the n key point information over time;

[0168] The classification submodule is used to classify the key point information in the target image according to the changes of the n key point information over time, and obtain the action behavior of the target user in the target image, where the target image is the first frame image in the n frames of images.

[0169] In some embodiments, the analysis module 830 may further include:

[0170] A second acquisition submodule is used to acquire a target area including key point information for each frame of the second image before acquiring n consecutive frames of images in the plurality of frames of the second image;

[0171] A cropping submodule, used for cropping the second image according to the target area to obtain multiple frames of third images;

[0172] A scaling submodule, used for scaling multiple frames of third images to the same size by bilinear interpolation;

[0173] Based on this, the first acquisition submodule may specifically include:

[0174] The acquisition unit is used to acquire n consecutive frames of images from a plurality of third frames of images with the same size.

[0175] In some embodiments, the acquisition unit may specifically include:

[0176] The moving subunit is used to move the sliding window according to a preset step length for multiple frames of third images of the same size, and divide the multiple frames of third images into multiple image groups, each image group includes n frames of images.

[0177] In some embodiments, the detection module 820 may specifically include:

[0178] The detection submodule is used to perform human body key point detection on multiple users respectively for each frame of the first image when the first image includes multiple users, and the multiple users include a target user.

[0179] In some embodiments, the detection module 820 may further include:

[0180] The third acquisition submodule is used to acquire the distance between each user and the body-in-white after performing body key point detection on multiple users respectively, so as to obtain multiple distances;

[0181] A first determination submodule, used to determine a minimum distance among multiple distances;

[0182] The second determination submodule is used to determine the user corresponding to the minimum distance as the target user.

[0183] In some embodiments, the acquisition module 810 may specifically include:

[0184] The fourth acquisition submodule is used to acquire the monitoring video stream;

[0185] The frame extraction submodule is used to extract frames from the monitoring video stream to obtain multiple frames of first images.

[0186] The monitoring device of the embodiment of the present application obtains multiple frames of first images including the target user and the body-in-white when the body-in-white is in a preset position, performs human body key point detection on the target user in the multiple frames of the first images respectively, obtains key point information corresponding to the target user, and performs correlation analysis on multiple key point information to obtain multiple action behaviors of the target user, so that the action behaviors of the target user can be accurately obtained during the process of the target user performing quality inspection on the body-in-white. By outputting warning information when multiple action behaviors do not meet preset conditions, that is, by monitoring multiple action behaviors of the target user, when multiple action behaviors do not include preset detection behaviors, or multiple action behaviors include preset detection action behaviors, but for each preset detection area, the number of occurrences of the preset detection behavior in the preset detection area is less than the preset threshold, outputting warning information, it is possible to improve the standardization of the quality inspection of the body-in-white by the target user, thereby improving the accuracy of the quality inspection of the body-in-white and reducing the production cost of the vehicle.

[0187] Based on the monitoring method provided in the above embodiment, the embodiment of the present application also provides a specific implementation of the electronic device. Fig. 9 A schematic diagram of an electronic device 900 provided in an embodiment of the present application is shown.

[0188] The electronic device 900 may include a processor 910 and a memory 920 storing computer program instructions.

[0189] Specifically, the processor 910 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0190] The memory 920 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 920 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 920 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 920 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 920 is a non-volatile solid-state memory.

[0191] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present application.

[0192] The processor 910 implements any one of the monitoring methods in the above embodiments by reading and executing computer program instructions stored in the memory 920 .

[0193] In one example, the electronic device 900 may further include a communication interface 930 and a bus 940. Fig. 9 As shown, the processor 910, the memory 920, and the communication interface 930 are connected via a bus 940 and communicate with each other.

[0194] The communication interface 930 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0195] Bus 940 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In suitable cases, bus 940 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0196] Exemplarily, the electronic device 900 may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA).

[0197] The electronic device can execute the monitoring method in the embodiment of the present application, thereby realizing the combination Figure 1 , Figures 6 to 8 Described monitoring method and device.

[0198] In addition, in combination with the monitoring method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the monitoring methods in the above embodiment is implemented.

[0199] In addition, the embodiment of the present application further provides a vehicle, which may include at least one of the following:

[0200] A monitoring device as in any one of the embodiments of the second aspect;

[0201] An electronic device as in any one of the embodiments of the third aspect;

[0202] The computer-readable storage medium in any one of the embodiments of the fourth aspect will not be described in detail here.

[0203] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0204] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0205] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0206] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0207] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A monitoring method, characterized in that: include: When the BIW is at a preset position, a plurality of first image frames are acquired, wherein the first image includes a target user and the BIW, the target user is a user who performs quality inspection on the BIW, and the BIW includes a plurality of preset inspection areas; Performing body key point detection on the target user in the multiple frames of the first image to obtain key point information of the target user; Performing correlation analysis on the multiple key point information to obtain multiple action behaviors of the target user; When the multiple action behaviors do not meet the preset conditions, a warning message is output, wherein the preset conditions are that the multiple action behaviors include preset detection behaviors, and in each of the preset detection areas, the number of occurrences of the preset detection behaviors is not less than a preset threshold.

2. The method according to claim 1, characterized in that The key point information includes hand key point information. After obtaining the multiple action behaviors of the target user, the method further includes: For each of the action behaviors, when the action behavior is a preset detection behavior, determine the hand key point information corresponding to the action behavior to obtain at least one of the hand key point information; Mapping at least one of the hand key point information to a first canvas including the multiple preset detection areas, wherein the multiple preset detection areas in the first canvas are the same as the multiple preset detection areas in the first image, and the size of the first canvas is the same as the size of the first image; When the plurality of action behaviors do not meet the preset conditions, outputting warning information includes: In the first canvas, for each of the preset detection areas, when the amount of the hand key point information in the preset detection area is less than the preset threshold, a warning message is output.

3. The method according to claim 2, characterized in that Before mapping at least one of the hand key point information to the first canvas including the plurality of preset detection areas, the method further includes: The plurality of preset detection areas in the first image are mapped into the first canvas.

4. The method according to claim 1, characterized in that: The performing correlation analysis on the plurality of key point information to obtain a plurality of action behaviors of the target user includes: For each frame of the first image, mapping the key point information to a second canvas having the same size as the first image, to obtain multiple frames of second images; Acquire n consecutive frames of images from the multiple frames of second images, where n is a positive integer; Comparing the key point information in the n frames of image to obtain the change of the n key point information over time; The key point information in the target image is classified according to how the n key point information changes over time, so as to obtain the action behavior of the target user in the target image, where the target image is the first frame image in the n frames of images.

5. The method according to claim 4, characterized in that Before acquiring n consecutive frames of images from the multiple frames of second images, the method further includes: For each frame of the second image, obtaining a target area including the key point information; Cropping the second image according to the target area to obtain multiple frames of third images; Scaling the multiple frames of third images to the same size by bilinear interpolation; The step of acquiring n consecutive frames of images from the plurality of frames of second images comprises: Continuous n frames of images are acquired from the plurality of third image frames having the same size.

6. The method according to claim 5, characterized in that The step of acquiring n consecutive frames of images from the plurality of third frames of images having the same size comprises: For the multiple frames of third images with the same size, the sliding window is moved according to a preset step length to divide the multiple frames of third images into a plurality of image groups, each of which includes n frames of images.

7. The method according to claim 1, characterized in that The detecting of human body key points of the target user in the multiple frames of the first image respectively includes: For each frame of the first image, when the first image includes multiple users, human body key point detection is performed on the multiple users respectively, and the multiple users include the target user.

8. The method according to claim 7, characterized in that After performing body key point detection on the multiple users respectively, the method further includes: Obtaining the distance between each of the users and the BIW to obtain a plurality of distances; determining a minimum distance among the plurality of distances; A user corresponding to the minimum distance is determined as the target user.

9. The method according to claim 1, characterized in that: The acquiring of multiple frames of first images comprises: Get surveillance video stream; Frame extraction is performed on the monitoring video stream to obtain multiple frames of first images.

10. A monitoring device, characterized in that: The device comprises: An acquisition module, used for acquiring a plurality of frames of first images when the BIW is at a preset position, wherein the first images include a target user and the BIW, the target user is a user who performs quality inspection on the BIW, and the BIW includes a plurality of preset inspection areas; A detection module, used to perform human body key point detection on the target user in the multiple frames of the first image to obtain key point information corresponding to the target user; An analysis module, used to perform correlation analysis on the plurality of key point information to obtain a plurality of action behaviors of the target user; An output module is used to output warning information when the multiple action behaviors do not meet the preset conditions. The preset conditions are that the multiple action behaviors include preset detection behaviors, and in each of the preset detection areas, the number of occurrences of the preset detection behaviors is not less than a preset threshold.

11. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the monitoring method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the monitoring method according to any one of claims 1 to 9 is implemented.

13. A vehicle, characterized in that: Include at least one of the following: The monitoring device according to claim 10; The electronic device as claimed in claim 11; The computer readable storage medium of claim 12.

Images