A method and device for identifying target working status

Abstract

The present application provides a method and device for identifying the working state of a target, which uses a target detection model to mark multiple target detection objects in a current frame video image; for each target detection object, the position state data of the target detection object is used to match multiple frames of historical frame video images; the predicted motion state of the target detection object is predicted using a Kalman filter model; the predicted motion state of the target detection object is updated by determining the Kalman gain coefficient of the target detection object in the current frame video image, and the actual motion state of the target detection object is determined; based on the actual motion state of the target detection object in each frame of the video image, it is determined whether the target detection object has an abnormal working state within a specific time range. In this way, by determining the actual motion state of each target detection object, the working state of each target detection object can be accurately judged and tracked, and the working efficiency of the target detection object can be improved.

Description

Technical Field

[0001] The present application relates to the field of target tracking technology, and in particular to a method and device for identifying a target working state. Background Art

[0002] Target detection technology refers to the detection of the behavior of the target detection object within a certain time interval by using a set camera device. The detection frame is marked on the basis of the captured image collected by the camera device, the coordinate information of each detection frame in the captured image is recorded, and the working status of the target detection object is determined by judging the movement status of the detection frame within a specific time.

[0003] At present, a single-person single-camera method is usually adopted to detect and track the target detection object; if multiple people appear in the captured image, the target detection object cannot be accurately distinguished from the captured image, and further, the working status of the target detection object cannot be accurately judged, making it easy for the target detection object to fall into abnormal working conditions such as sleeping on the job, resulting in a decrease in the work efficiency of the target detection object. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a method and device for identifying the working status of a target, which can use the collected video stream data to accurately mark multiple target detection objects that need to be tracked, and according to the actual motion state of each target detection object in each frame of video image, accurately judge the working status of each target detection object and track the target detection object, thereby indirectly improving the working efficiency of the target detection object.

[0005] The present invention provides a method for identifying a target working state, and the method includes:

[0006] Perform frame processing on the collected video stream data to obtain multiple frames of video images;

[0007] Annotate the multiple frames of video images one by one, and for a current frame of video image in the annotation stage, use a target detection model to perform reasoning annotation on the current frame of video image, and annotate multiple target detection objects in the current frame of video image;

[0008] For each target detection object, using the position state data represented by the target detection object in the current frame video image, matching multiple frames of historical frame video images in which the target detection object is detected;

[0009] By analyzing the historical motion state of the target detection object in the plurality of historical frame video images, the predicted motion state of the target detection object in the current frame video image is predicted using a Kalman filter model;

[0010] By determining the Kalman gain coefficient of the target detection object in the current frame video image, updating the predicted motion state of the target detection object, and tracking and determining the actual motion state of the target detection object in the current frame video image;

[0011] Based on the actual motion state of the target detection object in each frame of video image, it is determined whether the target detection object has an abnormal working state within a specific time range.

[0012] In a possible implementation, for the current frame video image in the annotation stage, using the target detection model to perform reasoning annotation on the current frame video image, and annotating multiple target detection objects in the current frame video image, includes:

[0013] For the current frame video image, using the target detection model, marking multiple candidate detection frames corresponding to each target detection object in the current frame video image;

[0014] For each target detection object, a non-maximum suppression algorithm is used to filter out redundant detection frames in multiple candidate detection frames corresponding to the target detection object, and determine a target detection frame for marking the target detection object;

[0015] Until the redundant detection frames of each target detection object are filtered out, multiple target detection objects are marked in the current frame video image through the target detection frame of each target detection object.

[0016] In a possible implementation, for the current frame video image, using the target detection model to mark multiple candidate detection boxes corresponding to each target detection object in the current frame video image includes:

[0017] For the current frame video image, using the target detection model, annotating the detection object in the current frame video image, and annotating a plurality of candidate detection frames in the current frame video image;

[0018] According to the target detection object marked by each candidate detection frame, the multiple candidate detection frames are classified and processed to determine a plurality of separate candidate detection frames corresponding to each target detection object.

[0019] In a possible implementation, for each target detection object, using the position state data represented by the target detection object in the current frame video image, matching multiple frames of historical frame video images in which the target detection object is detected includes:

[0020] For each target detection object, the position state data represented by the target detection object in the current frame video image is extracted through the pre-trained feature learning model;

[0021] By determining the error covariance matrix between the position state data of the target detection object and the position state data of the corresponding target detection object in the historical frame video image, multiple frames of historical frame video images in which the target detection object exists are matched.

[0022] In a possible implementation, the using of the Kalman filter model to predict the predicted motion state of the target detection object in the current frame video image by analyzing the historical motion state of the target detection object in the multiple frames of historical frame video images includes:

[0023] By means of the object identifier assigned to the target detection object, the target detection object is associated with the multiple frames of historical frame video images to determine the historical movement speed and historical movement trajectory of the target detection object in each frame of historical frame video image;

[0024] The Kalman filter model is used to predict the predicted motion state of the target detection object in the current frame of the video image based on the historical motion speed and historical motion trajectory of the target detection object in each frame of the historical frame video image.

[0025] In a possible implementation manner, after tracking and determining the actual motion state of the target detection object in the current frame video image, the recognition method further includes:

[0026] In a current frame video image, a frame motion state of a target detection frame used to mark the target detection object is tracked and determined, and the target detection frame is associated with an object identifier assigned to the target detection object; wherein the frame motion state includes a frame position and a frame trajectory.

[0027] In a possible implementation manner, the Kalman gain coefficient of the target detection object in the current frame video image is determined by the following steps:

[0028] Determine a measurement matrix, a predicted state covariance, and a measurement noise covariance matrix of a target detection frame for marking the target detection object in the current frame video image;

[0029] Based on the measurement matrix, the predicted state covariance and the measurement noise covariance matrix, a Kalman gain coefficient generated by the target detection box in the current frame video image is determined.

[0030] In a possible implementation, the identification method further includes:

[0031] The predicted state covariance is updated based on the Kalman gain coefficient and the measurement matrix.

[0032] In a possible implementation, determining whether the target detection object is in an abnormal working state within a specific time range based on the actual motion state of the target detection object in each frame of the video image includes:

[0033] Based on the actual motion state of the target detection object in each frame of the video image, determining the position offset of the target detection object within a specific time range;

[0034] If the duration that the position offset is less than the offset threshold is greater than the specific time range, it is determined that the target detection object is in an abnormal working state within the specific time range;

[0035] Otherwise, it is determined that the target detection object is in a normal working state within a specific time range.

[0036] In a possible implementation, the identification method further includes:

[0037] The target detection frame is used to mark each frame of the video image of the target detection object to obtain the marked video data.

[0038] The embodiment of the present application also provides a device for identifying a target working state, the device comprising:

[0039] A framing module is used to perform framing processing on the collected video stream data to obtain multiple frames of video images;

[0040] An object annotation module is used to perform inference annotation on the current frame video image using a target detection model, and annotate multiple target detection objects in the current frame video image;

[0041] An image matching module is used to match, for each target detection object, multiple frames of historical frame video images in which the target detection object is detected using the position state data represented by the target detection object in the current frame video image;

[0042] An operation state estimation module is used to predict the predicted motion state of the target detection object in the current frame video image by analyzing the historical motion state of the target detection object in the multiple frames of historical frame video images using a Kalman filter model;

[0043] An operation state adjustment module is used to update the predicted motion state of the target detection object by determining the Kalman gain coefficient of the target detection object in the current frame video image, and track and determine the actual motion state of the target detection object in the current frame video image;

[0044] The working state diagnosis module is used to determine whether the target detection object has an abnormal working state within a specific time range based on the actual motion state of the target detection object in each frame of video image.

[0045] In a possible implementation, the object annotation module is used to annotate the multiple frames of video images one by one, and for a current frame of video image in the annotation stage, uses a target detection model to perform reasoning annotation on the current frame of video image. When multiple target detection objects are annotated in the current frame of video image, the object annotation module is used to:

[0046] For the current frame video image, using the target detection model, marking multiple candidate detection frames corresponding to each target detection object in the current frame video image;

[0047] For each target detection object, a non-maximum suppression algorithm is used to filter out redundant detection frames in multiple candidate detection frames corresponding to the target detection object, and determine a target detection frame for marking the target detection object;

[0048] Until the redundant detection frames of each target detection object are filtered out, multiple target detection objects are marked in the current frame video image through the target detection frame of each target detection object.

[0049] In a possible implementation, when the object annotation module is used to annotate multiple candidate detection boxes corresponding to each target detection object in the current frame video image using the target detection model, the object annotation module is used to:

[0050] For the current frame video image, using the target detection model, annotating the detection object in the current frame video image, and annotating a plurality of candidate detection frames in the current frame video image;

[0051] According to the target detection object marked by each candidate detection frame, the multiple candidate detection frames are classified and processed to determine a plurality of separate candidate detection frames corresponding to each target detection object.

[0052] In a possible implementation manner, when the image matching module is used to match multiple frames of historical frame video images in which the target detection object is detected using the position state data represented by the target detection object in the current frame video image for each target detection object, the image matching module is used to:

[0053] For each target detection object, the position state data represented by the target detection object in the current frame video image is extracted through the pre-trained feature learning model;

[0054] By determining the error covariance matrix between the position state data of the target detection object and the position state data of the corresponding target detection object in the historical frame video image, multiple frames of historical frame video images in which the target detection object exists are matched.

[0055] In a possible implementation, when the running state estimation module is used to use a Kalman filter model to predict the predicted motion state of the target detection object in the current frame video image by analyzing the historical motion state of the target detection object in the multiple frames of historical frame video images, the running state estimation module is used to:

[0056] By means of the object identifier assigned to the target detection object, the target detection object is associated with the multiple frames of historical frame video images to determine the historical movement speed and historical movement trajectory of the target detection object in each frame of historical frame video image;

[0057] The Kalman filter model is used to predict the predicted motion state of the target detection object in the current frame of the video image based on the historical motion speed and historical motion trajectory of the target detection object in each frame of the historical frame video image.

[0058] In a possible implementation, after tracking and determining the actual motion state of the target detection object in the current frame video image, the recognition device further includes an association module, and the association module is used to:

[0059] In a current frame video image, a frame motion state of a target detection frame used to mark the target detection object is tracked and determined, and the target detection frame is associated with an object identifier assigned to the target detection object; wherein the frame motion state includes a frame position and a frame trajectory.

[0060] In a possible implementation manner, the recognition device further includes a coefficient calculation module, and the coefficient calculation module is used to determine the Kalman gain coefficient of the target detection object in the current frame video image through the following steps:

[0061] Determine a measurement matrix, a predicted state covariance, and a measurement noise covariance matrix of a target detection frame for marking the target detection object in the current frame video image;

[0062] Based on the measurement matrix, the predicted state covariance and the measurement noise covariance matrix, a Kalman gain coefficient generated by the target detection box in the current frame video image is determined.

[0063] In a possible implementation manner, the recognition device further includes a variance adjustment module, and the variance adjustment module is used to:

[0064] The predicted state covariance is updated based on the Kalman gain coefficient and the measurement matrix.

[0065] In a possible implementation, when the working state diagnosis module is used to determine whether the target detection object has an abnormal working state within a specific time range based on the actual motion state of the target detection object in each frame of video image, the working state diagnosis module is used to:

[0066] Based on the actual motion state of the target detection object in each frame of the video image, determining the position offset of the target detection object within a specific time range;

[0067] If the duration that the position offset is less than the offset threshold is greater than the specific time range, it is determined that the target detection object is in an abnormal working state within the specific time range;

[0068] Otherwise, it is determined that the target detection object is in a normal working state within a specific time range.

[0069] In a possible implementation, the recognition device further includes an image stitching module, and the image stitching module is used to:

[0070] The target detection frame is used to mark each frame of the video image of the target detection object to obtain the marked video data.

[0071] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the target working state identification method as described above are performed.

[0072] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the target working state identification method as described above are executed.

[0073] The target working state recognition method and device provided in the embodiment of the present application perform frame processing on the collected video stream data to obtain multiple frames of video images; annotate the multiple frames of video images one by one, and for the current frame of video image in the annotation stage, use the target detection model to perform inference annotation on the current frame of video image, and annotate multiple target detection objects in the current frame of video image; for each target detection object, use the position state data represented by the target detection object in the current frame of video image to match the multiple frames of historical frame video images in which the target detection object is detected; use the Kalman filter model to predict the predicted motion state of the target detection object in the current frame of video image by analyzing the historical motion state of the target detection object in multiple frames of historical frame video images; update the predicted motion state of the target detection object by determining the Kalman gain coefficient of the target detection object in the current frame of video image, and determine the actual motion state of the target detection object in the current frame of video image; based on the actual motion state of the target detection object in each frame of video image, determine whether the target detection object has an abnormal working state within a specific time range. In this way, the collected video stream data can be used to accurately mark multiple target detection objects that need to be tracked, and based on the actual motion state of each target detection object in each frame of video image, the working state of each target detection object can be accurately judged and tracked, thereby indirectly improving the working efficiency of the target detection object.

[0074] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1 A flow chart of a method for identifying a target working state provided in an embodiment of the present application;

[0077] Figure 2 A schematic diagram of a detection frame filtering process provided in an embodiment of the present application;

[0078] Figure 3 A schematic diagram of a detection frame labeling process provided in an embodiment of the present application;

[0079] Figure 4A schematic diagram of the structure of a target working state identification device provided in an embodiment of the present application;

[0080] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0082] Research has found that, at present, the target detection object is usually detected and tracked by a single person and a single camera device; if multiple people appear in the captured image, the target detection object cannot be accurately distinguished from the captured image, and then, the working status of the target detection object cannot be accurately judged, making it easy for the target detection object to have abnormal working conditions such as sleeping on the job, resulting in a decrease in the work efficiency of the target detection object.

[0083] Based on this, an embodiment of the present application provides a method for identifying the working status of a target, which can use the collected video stream data to accurately mark multiple target detection objects that need to be tracked, and according to the actual motion state of each target detection object in each frame of video image, it can achieve accurate judgment of the working status of each target detection object and tracking of the target detection object, thereby indirectly improving the working efficiency of the target detection object.

[0084] See also Figure 1 , Figure 1 A flow chart of a method for identifying a target working state provided in an embodiment of the present application. Figure 1 As shown in , the target working state identification method provided by the embodiment of the present application includes:

[0085] S101, performing frame processing on the collected video stream data to obtain multiple frames of video images.

[0086] S102, annotating the multiple frames of video images one by one, and for a current frame of video image in the annotation stage, using a target detection model to perform inference annotation on the current frame of video image, and annotating multiple target detection objects in the current frame of video image.

[0087] S103 . For each target detection object, using the position state data represented by the target detection object in the current frame video image, match multiple frames of historical frame video images in which the target detection object is detected.

[0088] S104, using a Kalman filter model, by analyzing the historical motion states of the target detection object in the multiple frames of historical frame video images, predicting the predicted motion state of the target detection object in the current frame video image.

[0089] S105 , updating the predicted motion state of the target detection object by determining the Kalman gain coefficient of the target detection object in the current frame video image, and tracking and determining the actual motion state of the target detection object in the current frame video image.

[0090] S106: Based on the actual motion state of the target detection object in each frame of the video image, determine whether the target detection object has an abnormal working state within a specific time range.

[0091] The present application provides a method for identifying the working state of a target, which performs frame processing on the collected video stream data to obtain multiple frames of video images; annotates multiple frames of video images one by one, and for the current frame of video images in the annotation stage, uses the target detection model to annotate multiple target detection objects in the current frame of video images; and for each target detection object, uses the position state data of the target detection object to match multiple frames of historical frame video images in which the target detection object exists; with the help of the matched multiple frames of historical frame video images, the annotated position of the target detection object in the current frame of video images is predicted and corrected to more accurately annotate the target detection object in the current frame of video images; uses the actual motion state of the target detection object in each video image to analyze and determine whether the target detection object has an abnormal working state within a specific time range. In this way, the collected video stream data can be used to accurately annotate multiple target detection objects that need to be tracked, and according to the actual motion state of each target detection object in each frame of video images, the accurate judgment of the working state of each target detection object is realized, thereby indirectly improving the working efficiency of the target detection object.

[0092] Here, in the subsequent detection process, it is necessary to track the working status of the target detection object within a specific time range. Therefore, it is necessary to collect video stream data of the target detection object; and, considering that a frame-by-frame analysis method is required in the subsequent process, the position of the target detection object in each frame of the video image is marked and tracked. Therefore, it is also necessary to perform frame processing on the collected video stream data.

[0093] In step S101, the collected video stream data is processed by frame segmentation, and the video stream data is segmented frame by frame, thereby obtaining multiple frames of video images.

[0094] The video stream data is collected by a camera device installed in the office area; the camera device is used to collect the work status video stream data of the staff who are in the office area and in a working state.

[0095] After multiple frames of video images are segmented, the target detection model is used to identify multiple target detection objects that need to be detected in each frame of the video image one by one, and the target detection objects are labeled.

[0096] In step S102, the multiple frames of video images obtained by segmentation are labeled one by one. For the current frame of video image that is in the labeling stage, a pre-trained target detection model is used to infer and label the potential target detection objects in the current frame of video image, and the labeling of each target detection object is completed one by one, thereby labeling multiple target detection objects in the current frame of video image.

[0097] Here, when the target detection model marks the target detection object, it will mark a large number of candidate detection frames on the same target detection object. In order to be able to mark the target detection object in the current frame video image more finely, it is also necessary to screen the candidate detection frames initially marked by the target detection model to filter out redundant detection frames with inaccurate marking positions.

[0098] In one implementation, step S102 includes:

[0099] S1021. For the current frame video image, use the target detection model to mark multiple candidate detection boxes corresponding to each target detection object in the current frame video image.

[0100] In this step, the current frame video image is input into the pre-trained target detection model so that the potential target detection objects appearing in the current frame video image are preliminarily labeled through the target detection model, so as to determine, for each target detection object, multiple candidate detection boxes for labeling the target detection object.

[0101] In one implementation, step S1021 includes:

[0102] S10211. For the current frame video image, use the target detection model to mark the detection object in the current frame video image, and mark multiple candidate detection boxes in the current frame video image.

[0103] S10212. Classify the multiple candidate detection frames according to the target detection object marked by each candidate detection frame, and determine multiple separate candidate detection frames corresponding to each target detection object.

[0104] In this step, for the multiple marked candidate detection frames, the multiple marked candidate detection frames in the current frame video image are clustered according to the target detection object marked by each candidate detection frame, so as to cluster the candidate detection frames used to mark the same target detection object into one category. In this way, in the current frame video image, for each target detection object, multiple separate candidate detection frames for marking the target detection object can be determined.

[0105] Here, in order to ensure the accuracy of the annotation, when the target detection object is preliminarily annotated, as many candidate detection frames containing the target detection object as possible are annotated. However, for a target detection object, among the multiple candidate detection frames annotated for it, there is only one detection frame that accurately annotates it. Other detection frames can be regarded as redundant detection frames. Redundant detection frames will interfere with the tracking process of the target detection object, and thus, the target detection object cannot be accurately tracked. Therefore, it is necessary to further filter out the redundant detection frames in the multiple recognition detection results.

[0106] S1022. For each target detection object, a non-maximum suppression algorithm is used to filter out redundant detection frames in multiple candidate detection frames corresponding to the target detection object, and determine a target detection frame for marking the target detection object.

[0107] In this step, for each target detection object, a non-maximum suppression algorithm is used to filter out redundant detection frames in the multiple candidate detection frames corresponding to the target detection object by comparing the confidence of the multiple candidate detection frames corresponding to the target detection object with the confidence threshold and intersection-over-union threshold set for the target detection object, so as to determine a target detection frame for accurately marking the target detection object.

[0108] S1023, until the redundant detection frames of each target detection object are filtered out, a plurality of target detection objects are marked in the current frame video image through the target detection frame of each target detection object.

[0109] In this step, the redundant detection frames of each target detection object are filtered out one by one until the redundant detection frames of all target detection objects are filtered out. The target detection frame of each target detection object is used to mark each target detection object in the current frame video image, thereby achieving accurate marking of multiple target detection objects in the current frame video image.

[0110] See also Figure 2 , Figure 2 A schematic diagram of a detection frame filtering process provided by an embodiment of the present application. Figure 2 As shown, image preprocessing is performed on the current frame video image; feature extraction is performed on the preprocessed current frame video image; multiple candidate detection frames are marked in the current frame video image using the extracted features; multiple candidate detection frames are classified according to the target detection objects marked by the candidate detection frames; the non-maximum suppression method is used to filter out redundant detection frames in the multiple candidate detection frames; and the current frame video image marked with the target detection frame is output.

[0111] Here, in order to track the target detection object, it is necessary to use the historical frame video images in which the target detection object exists, so as to accurately determine the actual motion state of the target detection object in the current frame video image by analyzing the historical frame video images of the target detection object.

[0112] In step S103, for each target detection object marked in the current frame video image, the position state data of the target detection object is extracted, and the position state data represented by the target detection object in the current frame video image is used to match multiple frames of historical frame video images in which the target detection object is detected from a large number of historical frame video images.

[0113] Here, the position state data of the target detection object in the current frame video image can be extracted with the help of a pre-trained feature learning model (for example, a deep learning model such as a convolutional neural network).

[0114] In one implementation, step S103 includes:

[0115] S1031. For each target detection object, extract the position state data represented by the target detection object in the current frame video image through a pre-trained feature learning model.

[0116] In this step, the current frame video image is input into a pre-trained feature learning model, and for each target detection object appearing in the current frame video image, high-dimensional feature embedding is performed in the current frame video image through the pre-trained feature learning model to extract the appearance information represented by the target detection object in the current frame video image and obtain the position status data of the target detection object.

[0117] S1032. Match multiple frames of historical frame video images in which the target detection object exists by determining an error covariance matrix between the position state data of the target detection object and the position state data of the corresponding target detection object in the historical frame video image.

[0118] In this step, the error covariance matrix between the position state data of the target detection object and the position state data of each target detection object in the historical frame video image can be determined by calculating the cosine similarity or Euclidean distance between the feature data, that is, by calculating the cosine similarity or Euclidean distance between the position state data of the target detection object and the position state data of the corresponding target detection object in the historical frame video image, and then, a plurality of historical frame video images in which the target detection object is also detected can be matched from a large number of historical frame video images.

[0119] Here, in order to avoid the problem of being unable to accurately determine the motion state of the target detection object due to the loss of video frame images, the Kalman filter method is adopted to accurately estimate the motion state of the target detection object through two steps of prediction and correction.

[0120] In step S104, the Kalman filter model is used to predict the motion state of the target detection object in the current frame video image by analyzing the historical motion state of the target detection object in multiple frames of historical frame video images, so as to predict the predicted motion state of the target detection object in the current frame video image.

[0121] The motion state refers to the motion speed and motion trajectory of the target detection object.

[0122] In one implementation, step S104 includes:

[0123] S1041. Associating the target detection object with the plurality of historical frame video images by means of the object identifier assigned to the target detection object, and determining the historical motion speed and historical motion trajectory of the target detection object in each frame of the historical frame video image.

[0124] In this step, between multiple frames of historical frame video images, in order to accurately locate the position of the target detection object in each frame of historical frame video image, the object identification (ID information) assigned to the target detection object is used to associate the target detection object in each frame of historical frame video image, and the historical movement speed and historical movement trajectory of the target detection object in each frame of historical frame video image are determined.

[0125] S1042: Using the Kalman filter model, based on the historical motion speed and historical motion trajectory of the target detection object in each frame of the historical frame video image, predict the predicted motion state of the target detection object in the current frame video image.

[0126] In this step, the Kalman filter model is used to predict the motion state (i.e., motion trajectory and motion speed) that the target detection object should have in the current frame video image by utilizing the historical motion speed and historical motion trajectory of the target detection object in each frame of the historical frame video image, so as to obtain the predicted motion state (i.e., predicted motion trajectory and predicted motion speed) of the target detection object in the current frame video image.

[0127] In order to reduce the estimation error of the motion state of the target detection state, on the basis of the predicted motion state of the target detection object, combined with the Kalman gain coefficient generated by the target detection object in the current frame video image, the predicted operating state is adjusted to achieve accurate estimation of the actual motion state of the target detection object.

[0128] In step S105, the Kalman gain coefficient of the target detection object in the current frame video image is determined, and the predicted motion state of the target detection object is updated using the determined Kalman gain coefficient to determine the actual motion state of the target detection object in the current frame video image.

[0129] Here, the actual motion state obtained by updating the predicted motion state is more in line with the motion trend of the target detection object in space than the predicted motion state of the target detection object and the current motion state in the current frame video image; even when the current frame video image is discontinuous with the historical frame video image, the actual motion state of the target detection object can still be accurately determined.

[0130] In one implementation, after tracking and determining the actual motion state of the target detection object in the current frame video image, the recognition method further includes:

[0131] In the current frame video image, a frame motion state of a target detection frame used to mark the target detection object is tracked and determined, and the target detection frame is associated with an object identifier assigned to the target detection object.

[0132] In this step, after the target detection object is determined from the current frame video image, the frame motion state of the target detection frame used to mark the target detection object is tracked and determined, and the target detection frame is associated with the object identifier assigned to the target detection object, so that the object identifier can be used in subsequent processes to track the target detection object.

[0133] The frame motion state includes the frame position and the frame trajectory.

[0134] In one implementation, the Kalman gain coefficient of the target detection object in the current frame video image is determined by the following steps:

[0135] Step 1: Determine a measurement matrix H, a predicted state covariance P', and a measurement noise covariance matrix R for marking a target detection frame of the target detection object in the current frame video image.

[0136] Step 2: Based on the measurement matrix H, the predicted state covariance P' and the measurement noise covariance matrix R, determine the Kalman gain coefficient generated by the target detection frame in the current frame video image.

[0137] In this step, the Kalman gain coefficient K generated by the target detection frame in the current frame video image is calculated by the following formula:

[0138] K=P′*H T *(H*P′*H T +R) -1 ;

[0139] Where K is the Kalman gain coefficient.

[0140] Here, in order to assist the model modification in estimating the motion state of the target detection object and improve the accuracy of the motion state prediction result, after determining the Kalman gain coefficient of the target detection object, the predicted state covariance P' can be reversely adjusted by the Kalman gain coefficient.

[0141] In one embodiment, the identification method further includes: updating the predicted state covariance based on the Kalman gain coefficient and the measurement matrix.

[0142] In this step, the update of the predicted state covariance is achieved through the following formula:

[0143] P = (IK*H)*P′;

[0144] Where P is the updated predicted state covariance.

[0145] Here, if the target detection object temporarily "disappears" in the video image, if it reappears in the subsequent video image, it can also be associated with the historical video image with the help of the object identifier assigned to the target detection object.

[0146] If the target detection object appears in the video image for the first time, an initial motion state is assigned to the target detection object.

[0147] Here, the Kalman filter model can be obtained by training sample videos that are pre-labeled with data labels; specifically, the sample videos are labeled, and first, data labels are assigned to the sample videos based on the actual situation of the sample videos; wherein the data labels include abnormal labels (i.e., sleeping labels) and normal labels (i.e., non-sleeping labels); secondly, the labeled sample videos are converted into files in the YOLO format; finally, the labeled sample videos are divided into a training set, a test set, and a validation set; and the training set, the test set, and the validation set are used to complete the training of the Kalman filter model.

[0148] In step S106, the actual motion state of the target detection object in each frame of the video image is analyzed, and by determining the offset of the target detection object within a specific time range, it is determined whether the target detection object has an abnormal working state (for example, a sleeping state) within the specific time range.

[0149] Here, the abnormal working state means that the target detection object is always in a state without deviation within a specific time. At this time, it can be considered that the target detection object has entered a "sleep state". Therefore, it can be considered that the target detection object is in an abnormal working state.

[0150] In one implementation, step S106 includes:

[0151] S1061. Determine a position offset of the target detection object within a specific time range based on an actual motion state of the target detection object in each frame of the video image.

[0152] In this step, based on the actual motion state of the target detection object in each frame of the video image (ie, the actual motion trajectory and the actual running speed), the position offset of the target detection object at the position within a specific time range is determined.

[0153] S1062: If the duration that the position offset is less than the offset threshold is greater than the specific time range, determine that the target detection object is in an abnormal working state within the specific time range.

[0154] In this step, when there is a position offset of the target detection object, the position offset is compared with a preset offset threshold. If the position offset of the target detection object in multiple video images is less than the offset threshold for a duration greater than a specific time range, it means that the target detection object has no position change within the specific time range corresponding to the video image (i.e., the acquisition time range of the multiple video images). At this time, it can be considered that the target detection object has entered a "sleep state", and it is determined that the target detection object is in an abnormal working state within the specific time range.

[0155] S1063. Otherwise, determine that the target detection object is in a normal working state within a specific time range.

[0156] In one embodiment, the identification method also includes: if it is determined that the target detection object is in an abnormal working state, changing the display color of the target detection box used to mark the target detection object in the video image, so as to prompt the monitoring personnel that the target detection object is in an abnormal working state through the display color; and sending abnormal prompt information to the monitoring personnel.

[0157] In one embodiment, the recognition method further includes: splicing each frame of the video image in which the target detection object is marked with the target detection frame to obtain marked video data.

[0158] See also Figure 3 , Figure 3 A schematic diagram of a detection frame labeling process provided by an embodiment of the present application. Figure 3 As shown, step 301, obtain video stream data; step 302, extract multiple frames of video images from the video stream data; step 303, mark the target detection object in the current frame video image, and extract the position status data of the target detection object; step 304, associate the historical frame video images in which the target detection object appears; step 305, use the Kalman filter model to determine the actual motion state of the target detection object in the current frame video image; step 306, if the target detection object "disappears" in the video image, the target detection object is disappeared or the initial running state is assigned; step 307, combine each frame of the video image marked with the target detection frame mark; step 308, output the video data marked with the target detection frame mark.

[0159] The target working state recognition method provided by the embodiment of the present application performs frame processing on the collected video stream data to obtain multiple frames of video images; annotates the multiple frames of video images one by one, and for the current frame of video image in the annotation stage, uses the target detection model to perform inference annotation on the current frame of video image, and annotates multiple target detection objects in the current frame of video image; for each target detection object, uses the position state data represented by the target detection object in the current frame of video image to match the multiple frames of historical frame video images in which the target detection object is detected; uses the Kalman filter model to predict the predicted motion state of the target detection object in the current frame of video image by analyzing the historical motion state of the target detection object in multiple frames of historical frame video images; updates the predicted motion state of the target detection object by determining the Kalman gain coefficient of the target detection object in the current frame of video image, and determines the actual motion state of the target detection object in the current frame of video image; based on the actual motion state of the target detection object in each frame of video image, determines whether the target detection object has an abnormal working state within a specific time range. In this way, the collected video stream data can be used to accurately mark multiple target detection objects that need to be tracked, and based on the actual motion state of each target detection object in each frame of video image, the working state of each target detection object can be accurately judged and tracked, thereby indirectly improving the working efficiency of the target detection object.

[0160] See also Figure 4 , Figure 4 This is a schematic diagram of a target working state recognition device provided in an embodiment of the present application. Figure 4 As shown in , the identification device 400 includes:

[0161] The frame segmentation module 410 is used to perform frame segmentation processing on the collected video stream data to obtain multiple frames of video images;

[0162] The object annotation module 420 is used to annotate the multiple frames of video images one by one, and for a current frame of video image in the annotation stage, use the target detection model to perform reasoning annotation on the current frame of video image, and annotate multiple target detection objects in the current frame of video image;

[0163] The image matching module 430 is used to match, for each target detection object, multiple frames of historical frame video images in which the target detection object is detected using the position state data represented by the target detection object in the current frame video image;

[0164] The running state estimation module 440 is used to predict the predicted motion state of the target detection object in the current frame video image by analyzing the historical motion state of the target detection object in the multiple frames of historical frame video images using a Kalman filter model;

[0165] The running state adjustment module 450 is used to update the predicted motion state of the target detection object by determining the Kalman gain coefficient of the target detection object in the current frame video image, and track and determine the actual motion state of the target detection object in the current frame video image;

[0166] The working state diagnosis module 460 is used to determine whether the target detection object has an abnormal working state within a specific time range based on the actual motion state of the target detection object in each frame of video image.

[0167] Furthermore, the object annotation module 420 is used to perform reasoning annotation on the current frame video image in the annotation stage using the target detection model. When multiple target detection objects are annotated in the current frame video image, the object annotation module 420 is used to:

[0168] For the current frame video image, using the target detection model, marking multiple candidate detection frames corresponding to each target detection object in the current frame video image;

[0169] For each target detection object, a non-maximum suppression algorithm is used to filter out redundant detection frames in multiple candidate detection frames corresponding to the target detection object, and determine a target detection frame for marking the target detection object;

[0170] Until the redundant detection frames of each target detection object are filtered out, multiple target detection objects are marked in the current frame video image through the target detection frame of each target detection object.

[0171] Furthermore, when the object labeling module 420 is used to label multiple candidate detection boxes corresponding to each target detection object in the current frame video image using the target detection model, the object labeling module 420 is used to:

[0172] For the current frame video image, using the target detection model, annotating the detection object in the current frame video image, and annotating a plurality of candidate detection frames in the current frame video image;

[0173] According to the target detection object marked by each candidate detection frame, the multiple candidate detection frames are classified and processed to determine a plurality of separate candidate detection frames corresponding to each target detection object.

[0174] Further, when the image matching module 430 is used to match multiple frames of historical frame video images in which the target detection object is detected for each target detection object using the position state data represented by the target detection object in the current frame video image, the image matching module 430 is used to:

[0175] For each target detection object, the position state data represented by the target detection object in the current frame video image is extracted through the pre-trained feature learning model;

[0176] By determining the error covariance matrix between the position state data of the target detection object and the position state data of the corresponding target detection object in the historical frame video image, multiple frames of historical frame video images in which the target detection object exists are matched.

[0177] Further, when the running state estimation module 440 is used to predict the predicted motion state of the target detection object in the current frame video image by analyzing the historical motion state of the target detection object in the multiple frames of historical frame video images using the Kalman filter model, the running state estimation module 440 is used to:

[0178] By means of the object identifier assigned to the target detection object, the target detection object is associated with the multiple frames of historical frame video images to determine the historical movement speed and historical movement trajectory of the target detection object in each frame of historical frame video image;

[0179] The Kalman filter model is used to predict the predicted motion state of the target detection object in the current frame of the video image based on the historical motion speed and historical motion trajectory of the target detection object in each frame of the historical frame video image.

[0180] Furthermore, the recognition device 400 further includes a coefficient calculation module (not shown in the figure), which is used to determine the Kalman gain coefficient of the target detection object in the current frame video image through the following steps:

[0181] Determine a measurement matrix, a predicted state covariance, and a measurement noise covariance matrix of a target detection frame for marking the target detection object in the current frame video image;

[0182] Based on the measurement matrix, the predicted state covariance and the measurement noise covariance matrix, a Kalman gain coefficient generated by the target detection box in the current frame video image is determined.

[0183] Furthermore, after tracking and determining the actual motion state of the target detection object in the current frame video image, the recognition device 400 further includes an association module (not shown in the figure), which is used to:

[0184] In a current frame video image, a frame motion state of a target detection frame used to mark the target detection object is tracked and determined, and the target detection frame is associated with an object identifier assigned to the target detection object; wherein the frame motion state includes a frame position and a frame trajectory.

[0185] Furthermore, the identification device 400 further includes a variance adjustment module (not shown in the figure), and the variance adjustment module is used to:

[0186] The predicted state covariance is updated based on the Kalman gain coefficient and the measurement matrix.

[0187] In a possible implementation, when the working state diagnosis module is used to determine whether the target detection object has an abnormal working state within a specific time range based on the actual motion state of the target detection object in each frame of video image, the working state diagnosis module is used to:

[0188] Based on the actual motion state of the target detection object in each frame of the video image, determining the position offset of the target detection object within a specific time range;

[0189] If the duration that the position offset is less than the offset threshold is greater than the specific time range, it is determined that the target detection object is in an abnormal working state within the specific time range;

[0190] Otherwise, it is determined that the target detection object is in a normal working state within a specific time range.

[0191] Furthermore, the recognition device 400 further includes an image stitching module (not shown in the figure), and the image stitching module is used to:

[0192] The target detection frame is used to mark each frame of the video image of the target detection object to obtain the marked video data.

[0193] The target working state recognition device provided in the embodiment of the present application performs frame processing on the collected video stream data to obtain multiple frames of video images; annotates the multiple frames of video images one by one, and for the current frame of video image in the annotation stage, uses the target detection model to perform inference annotation on the current frame of video image, and annotates multiple target detection objects in the current frame of video image; for each target detection object, uses the position state data represented by the target detection object in the current frame of video image to match the multiple frames of historical frame video images in which the target detection object is detected; uses the Kalman filter model to predict the predicted motion state of the target detection object in the current frame of video image by analyzing the historical motion state of the target detection object in the multiple frames of historical frame video images; updates the predicted motion state of the target detection object by determining the Kalman gain coefficient of the target detection object in the current frame of video image, and determines the actual motion state of the target detection object in the current frame of video image; based on the actual motion state of the target detection object in each frame of video image, determines whether the target detection object has an abnormal working state within a specific time range. In this way, the collected video stream data can be used to accurately mark multiple target detection objects that need to be tracked, and based on the actual motion state of each target detection object in each frame of video image, the working state of each target detection object can be accurately judged and tracked, thereby indirectly improving the working efficiency of the target detection object.

[0194] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in , the electronic device 500 includes a processor 510 , a memory 520 and a bus 530 .

[0195] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the above-mentioned Figure 1 The steps of the method for identifying the target working state in the method embodiment shown, and the specific implementation manner can be found in the method embodiment, which will not be repeated here.

[0196] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the method for identifying the target working state in the method embodiment shown, and the specific implementation manner can be found in the method embodiment, which will not be repeated here.

[0197] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0198] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0199] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0201] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be represented in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0202] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for identifying a target working state, characterized in that: The identification method comprises: Perform frame processing on the collected video stream data to obtain multiple frames of video images; Annotate the multiple frames of video images one by one, and for a current frame of video image in the annotation stage, use a target detection model to perform reasoning annotation on the current frame of video image, and annotate multiple target detection objects in the current frame of video image; For each target detection object, using the position state data represented by the target detection object in the current frame video image, matching multiple frames of historical frame video images in which the target detection object is detected; By analyzing the historical motion state of the target detection object in the plurality of historical frame video images, the predicted motion state of the target detection object in the current frame video image is predicted using a Kalman filter model; By determining the Kalman gain coefficient of the target detection object in the current frame video image, updating the predicted motion state of the target detection object, and tracking and determining the actual motion state of the target detection object in the current frame video image; Based on the actual motion state of the target detection object in each frame of video image, it is determined whether the target detection object has an abnormal working state within a specific time range.

2. The identification method according to claim 1, characterized in that: The method of using a target detection model to perform inference annotation on a current frame video image in the annotation stage, and annotating a plurality of target detection objects in the current frame video image, includes: For the current frame video image, using the target detection model, marking multiple candidate detection frames corresponding to each target detection object in the current frame video image; For each target detection object, a non-maximum suppression algorithm is used to filter out redundant detection frames in multiple candidate detection frames corresponding to the target detection object, and determine a target detection frame for marking the target detection object; Until the redundant detection frames of each target detection object are filtered out, multiple target detection objects are marked in the current frame video image through the target detection frame of each target detection object.

3. The identification method according to claim 2, characterized in that: The method of marking multiple candidate detection frames corresponding to each target detection object in the current frame video image by using the target detection model includes: For the current frame video image, using the target detection model, annotating the detection object in the current frame video image, and annotating a plurality of candidate detection frames in the current frame video image; According to the target detection object marked by each candidate detection frame, the multiple candidate detection frames are classified and processed to determine a plurality of separate candidate detection frames corresponding to each target detection object.

4. The identification method according to claim 1, characterized in that: For each target detection object, using the position state data represented by the target detection object in the current frame video image, matching multiple frames of historical frame video images in which the target detection object is detected, includes: For each target detection object, the position state data represented by the target detection object in the current frame video image is extracted through the pre-trained feature learning model; By determining the error covariance matrix between the position state data of the target detection object and the position state data of the corresponding target detection object in the historical frame video image, multiple frames of historical frame video images in which the target detection object exists are matched.

5. The identification method according to claim 1, characterized in that: The method of using the Kalman filter model to predict the predicted motion state of the target detection object in the current frame video image by analyzing the historical motion state of the target detection object in the multiple frames of historical frame video images includes: By means of the object identifier assigned to the target detection object, the target detection object is associated with the multiple frames of historical frame video images to determine the historical movement speed and historical movement trajectory of the target detection object in each frame of historical frame video image; The Kalman filter model is used to predict the predicted motion state of the target detection object in the current frame of the video image based on the historical motion speed and historical motion trajectory of the target detection object in each frame of the historical frame video image.

6. The identification method according to claim 1, characterized in that: After tracking and determining the actual motion state of the target detection object in the current frame video image, the recognition method further includes: In a current frame video image, a frame motion state of a target detection frame used to mark the target detection object is tracked and determined, and the target detection frame is associated with an object identifier assigned to the target detection object; wherein the frame motion state includes a frame position and a frame trajectory.

7. The identification method according to claim 1, characterized in that: The Kalman gain coefficient of the target detection object in the current frame video image is determined by the following steps: Determine a measurement matrix, a predicted state covariance, and a measurement noise covariance matrix of a target detection frame for marking the target detection object in the current frame video image; Based on the measurement matrix, the predicted state covariance and the measurement noise covariance matrix, a Kalman gain coefficient generated by the target detection box in the current frame video image is determined.

8. The identification method according to claim 1, characterized in that: The determining whether the target detection object has an abnormal working state within a specific time range based on the actual motion state of the target detection object in each frame of the video image includes: Based on the actual motion state of the target detection object in each frame of the video image, determining the position offset of the target detection object within a specific time range; If the duration that the position offset is less than the offset threshold is greater than the specific time range, it is determined that the target detection object is in an abnormal working state within the specific time range; Otherwise, it is determined that the target detection object is in a normal working state within a specific time range.

9. The identification method according to claim 1, characterized in that: The identification method further comprises: The target detection frame is used to mark each frame of the video image of the target detection object to obtain the marked video data.

10. A device for identifying a target working state, characterized in that: The identification device comprises: A framing module is used to perform framing processing on the collected video stream data to obtain multiple frames of video images; An object annotation module is used to annotate the multiple frames of video images one by one, and for a current frame of video image in the annotation stage, use a target detection model to perform reasoning annotation on the current frame of video image, and annotate multiple target detection objects in the current frame of video image; An image matching module is used to match, for each target detection object, multiple frames of historical frame video images in which the target detection object is detected using the position state data represented by the target detection object in the current frame video image; An operation state estimation module is used to predict the predicted motion state of the target detection object in the current frame video image by analyzing the historical motion state of the target detection object in the multiple frames of historical frame video images using a Kalman filter model; An operation state adjustment module is used to update the predicted motion state of the target detection object by determining the Kalman gain coefficient of the target detection object in the current frame video image, and track and determine the actual motion state of the target detection object in the current frame video image; The working state diagnosis module is used to determine whether the target detection object has an abnormal working state within a specific time range based on the actual motion state of the target detection object in each frame of video image.

Images