A method and apparatus for detecting behavior of a terminal device

Abstract

The application discloses a kind of detection method and device using the behavior of terminal equipment, comprising: obtaining image frame in video stream, if it is determined that image frame has at least one target object using terminal equipment, then determine the first image corresponding to any target object in image frame;Wherein, the first image includes the detection target of target object;First image is input into behavior detection model, according to the detection result of detection target output by behavior detection model, determine that target object is in the behavior of using terminal equipment;Wherein, behavior detection model is obtained based on self-supervised pre-training model according to image data with label training, label represents the behavior of using terminal equipment, equivalent to behavior detection model is supervised training with labeled image data and is obtained, so behavior detection model is supervised model, to realize the behavior of using terminal equipment based on supervised model detection, improve the accuracy of detecting the behavior of using terminal equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a method and device for detecting behavior of using a terminal device. BACKGROUND

[0002] Using a terminal device (such as playing with a mobile phone, playing with an iPad, etc.) is a common lifestyle. However, long-time playing with a mobile phone can easily cause fatigue and eye fatigue, etc. problems, especially playing with a mobile phone during working hours, which can easily reduce work efficiency, and even cause damage to physical health.

[0003] At present, the detection method based on image recognition can detect a target object in a behavior of using a terminal device, such as detection based on terminal device recognition and detection of human body posture of a target object. However, when the angle, distance, etc. of the target object holding the terminal device changes, it will affect the accuracy of detection; when the hand or head of the target object is blocked by other factors, it will also affect the accuracy of detection, resulting in low accuracy of detecting the behavior of the target object using the terminal device.

[0004] Therefore, there is an urgent need for a detection method for the behavior of using a terminal device to improve the accuracy of detecting the behavior of the target object using the terminal device. SUMMARY

[0005] The embodiments of the present application provide a detection method and device for the behavior of using a terminal device, which are used to detect the behavior of using a terminal device based on a supervised model, and improve the accuracy of detecting the behavior of using a terminal device.

[0006] In a first aspect, the embodiments of the present application provide a detection method for the behavior of using a terminal device, comprising:

[0007] obtaining an image frame in a video stream;

[0008] if it is determined that there is at least one target object using a terminal device in the image frame, determining a first image corresponding to any target object in the image frame; the first image includes a detection target of the target object;

[0009] inputting the first image into a behavior detection model, and determining that the target object is in a behavior of using a terminal device according to a detection result of the detection target output by the behavior detection model; the behavior detection model is obtained by training a self-supervised pre-training model according to image data with labels; the labels represent the behavior of using a terminal device.

[0010] In the technical solution, the video stream can be real-time monitoring video data of a monitoring camera, the image frame is obtained by taking a screenshot or image conversion on the video stream according to a preset frequency, and the target object is a person object. The behavior detection model is obtained by supervised training based on image data representing labels of behaviors of the terminal device, that is, the behavior detection model is a supervised model, so as to ensure the detection accuracy of the behavior detection model. Therefore, whether the target object is in the behavior of using the terminal device is determined according to the detection result of the detection target of the target object output by the detection model, and then the behavior of using the terminal device is detected based on the supervised model, and the accuracy of detecting the behavior of using the terminal device is improved.

[0011] Optionally, the behavior detection model is obtained by training a self-supervised pre-training model based on image data with labels, and the behavior detection model comprises:

[0012] extracting image data from a historical video stream;

[0013] preprocessing the image data to obtain a training set; the training set comprises a first data set without labels and a second data set with labels, the second data set comprises positive samples and negative samples, the positive samples are images of normal behaviors of using the terminal device, and the negative samples are images of abnormal behaviors of using the terminal device;

[0014] performing self-supervised training according to the first data set to determine a self-supervised pre-training model;

[0015] performing supervised training based on the self-supervised pre-training model according to the positive samples and the negative samples to determine a behavior detection model.

[0016] In the technical solution, the labels of the positive samples and the negative samples can be labels annotated by a terminal device based on an artificial annotation method, representing behaviors of using the terminal device and whether the behaviors are normal behaviors, such as labels representing that a behavior of a target object holding a terminal device is normal and a behavior of the target object watching the terminal device is abnormal. The self-supervised pre-training model can be obtained by performing self-supervised training in an MAE manner, and is used to learn terminal device features in an image, so as to ensure the detection accuracy of the behavior detection model trained based on the self-supervised pre-training model and the positive samples and the negative samples.

[0017] Optionally, preprocessing the image data comprises:

[0018] determining images with person objects from the image data;

[0019] determining images with a similarity less than a preset threshold from the images with person objects;

[0020] determine an image meeting a preset image parameter from the images with the similarity less than the preset threshold; the preset image parameter includes brightness, encoding parameter and / or resolution of the image.

[0021] In the technical solution, the image with the person object is determined from the image data, so that the empty shot image without the person object can be avoided to participate in the training of the behavior detection model, and the accuracy of the training of the behavior detection model is ensured. The image with the similarity less than the preset threshold is determined from the images with the person object, so that the redundant image data with high similarity is reduced, and the efficiency of the training of the behavior detection model is improved. The image meeting the preset image parameter is determined from the images with the similarity less than the preset threshold, so that the image quality of the training set is ensured, and the accuracy and reliability of the training of the behavior detection model are ensured.

[0022] Optionally, the determining the image frame with at least one target object using the terminal device includes:

[0023] inputting the image frame into the behavior detection model to obtain a detection result of the image frame; the detection result of the image frame represents a possibility that the image frame has the target object and the terminal device;

[0024] if the detection result of the image frame is greater than a first threshold, it is determined that the image frame has at least one target object using the terminal device.

[0025] In the technical solution, the detection result of the behavior detection model is a confidence value in a numerical form, and the higher the confidence value of the image frame is, the greater the possibility that the image frame has the target object and the terminal device is. The first threshold can be a value preset according to experience. The greater the first threshold is, the greater the accuracy that the image frame has the target object and the terminal device is. The smaller the first threshold is, the smaller the missing detection rate of the image frame with the target object and the terminal device is, and thus the accuracy of the determination of the image frame with at least one target object using the terminal device is improved.

[0026] Optionally, the determining the first image corresponding to any target object in the image frame includes:

[0027] For any target object, the first image corresponding to the target object is cropped from the image frame according to the position coordinates of the target object in the image frame; the position coordinates of the target object in the image frame are obtained according to the behavior detection model.

[0028] In the technical solution, the first image corresponding to the target object is determined in the image frame, so that the detection of the target object is realized in a targeted manner, and the accuracy of the detection of the target object by the behavior detection model is improved.

[0029] Optionally, the detection target includes the terminal device in the first image and the limb features of the target object; the detection result of the detection target characterizes the probability that the target object is using the terminal device.

[0030] Based on the detection results of the target object output by the behavior detection model, it is determined that the target object is performing a behavior of using a terminal device, including:

[0031] If the detection result of the terminal device in the first image and the detection result of the limb features of the target object are both greater than the second threshold, then it is determined that the target object is in the state of using the terminal device.

[0032] If, within a preset time period after the current moment, the target object is in a state of using a terminal device for a period of time longer than a time threshold, then it is determined that the target object is in the behavior of using a terminal device.

[0033] In the above technical solution, the detection result of the target represents the probability that the target object is using a terminal device. Similarly, the detection result of the target represents the probability that the target object is in a state of using a terminal device. Therefore, the higher the detection result of the target, the greater the probability that the target object is using a terminal device. Thus, it can be determined whether the target object is using a terminal device based on the detection result of the target. The second threshold can be a preset value based on experience. The larger the second threshold, the greater the accuracy in determining whether the target object is using a terminal device; the smaller the second threshold, the lower the false negative rate in determining whether the target object is using a terminal device, thereby improving the accuracy in determining whether the target object is using a terminal device.

[0034] Optionally, after determining that the target object is using a terminal device, the method further includes:

[0035] An alarm message is triggered, which includes the location information and / or identity information of the target object.

[0036] In the above technical solution, the alarm information includes the location and / or identity information of the target object, which can improve the efficiency of locating the target object that needs to be reminded, so as to remind the target object.

[0037] Secondly, embodiments of the present invention provide a device for detecting the behavior of a terminal device, comprising:

[0038] The acquisition module is used to acquire image frames from the video stream;

[0039] a processing module, configured to: if it is determined that the image frame has at least one target object using a terminal device, determine a first image corresponding to any target object in the image frame; the first image includes a detection target of the target object;

[0040] input the first image into a behavior detection model, and determine, according to a detection result of the detection target output by the behavior detection model, that the target object is in a behavior of using a terminal device; the behavior detection model is obtained based on a self-supervised pre-training model and according to image data with labels; the labels represent the behavior of using the terminal device.

[0041] Optionally, the processing module is specifically configured to:

[0042] extract image data from a historical video stream;

[0043] preprocess the image data to obtain a training set; the training set includes a first data set without labels and a second data set with labels; the second data set includes positive samples and negative samples; the positive samples are images of normal behaviors of using a terminal device; and the negative samples are images of abnormal behaviors of using a terminal device.

[0044] perform self-supervised training according to the first data set to determine a self-supervised pre-training model;

[0045] perform supervised training based on the self-supervised pre-training model and according to the positive samples and the negative samples to determine a behavior detection model.

[0046] Optionally, the processing module is specifically configured to:

[0047] determine, from the image data, an image with a person object;

[0048] determine, from the image with the person object, an image with a similarity less than a preset threshold;

[0049] determine, from the image with the similarity less than the preset threshold, an image meeting a preset image parameter; the preset image parameter includes brightness, encoding parameter and / or resolution of the image.

[0050] Optionally, the processing module is specifically configured to:

[0051] input the image frame into the behavior detection model to obtain a detection result of the image frame; the detection result of the image frame represents a possibility that the image frame has a target object and a terminal device;

[0052] if the detection result of the image frame is greater than a first threshold, it is determined that the image frame has at least one target object using a terminal device.

[0053] Optionally, the processing module is specifically used for:

[0054] For any target object, a first image corresponding to the target object is cropped from the image frame according to a position coordinate of the target object in the image frame; the position coordinate of the target object in the image frame is obtained according to the behavior detection model.

[0055] Optionally, the detection target includes a terminal device in the first image and a limb feature of the target object; and a detection result of the detection target represents a possibility that the target object is in the behavior of using the terminal device.

[0056] The processing module is specifically used for:

[0057] If the detection result of the terminal device in the first image and the detection result of the limb feature of the target object are both greater than a second threshold value, it is determined that the target object is in a state of using the terminal device.

[0058] If, within a preset time period after a current time, a time in which the target object is in the state of using the terminal device is greater than a time threshold value, it is determined that the target object is in the behavior of using the terminal device.

[0059] Optionally, the processing module is further used for:

[0060] After it is determined that the target object is in the behavior of using the terminal device, an alarm information is triggered, and the alarm information includes position information and / or identity information of the target object.

[0061] In a third aspect, an embodiment of the present application further provides a computer device, which comprises:

[0062] a memory, configured to store program instructions;

[0063] a processor, configured to invoke the program instructions stored in the memory, and execute the above-mentioned detection method of the behavior of using the terminal device according to the obtained program.

[0064] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are used to make a computer execute the above-mentioned detection method of the behavior of using the terminal device. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to make the technical solutions in the embodiments of the present application clearer, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0066] Figure 1 A system architecture schematic diagram provided for the embodiments of the present application;

[0067] Figure 2 A flowchart of a detection method of behavior of a terminal device provided for the embodiments of the present application;

[0068] Figure 3 A schematic diagram of an image frame provided for the embodiments of the present application;

[0069] Figure 4 A flowchart of determining a behavior detection model based on a Transformer model provided for the embodiments of the present application;

[0070] Figure 5 A structural schematic diagram of a detection device of behavior of a terminal device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0071] In order to make the technical solutions in the embodiments of the present application clearer, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0072] Using a terminal device is a common behavior in daily life at present, such as playing a mobile phone, playing an iPad, playing a game console, etc. Taking playing a mobile phone as an example, during leisure time, staff members use a mobile phone to read novels, watch movies, listen to music, play games, etc. to relax. However, long-time playing of a mobile phone can easily lead to eye fatigue, inattentiveness, etc. For example, playing a mobile phone during working time can easily lead to a decline in work efficiency and quality, etc. Therefore, it is necessary to detect the behavior of using a terminal device, so as to help managers to manage and monitor the safety of employees, and to remind the target object of using a terminal device to avoid the target object from having problems such as eye fatigue, inattentiveness, a decline in work efficiency, a decline in work quality, etc. due to using a terminal device.

[0073] Currently, the detection of the behavior of using the terminal device generally includes manual detection and image recognition-based detection. The manual detection refers to detecting whether the target object is in the behavior of using the terminal device by a designated person. The image recognition-based detection refers to detecting whether the target object is in the behavior of using the terminal device by the limb postures such as the hand and head of the terminal device and the target object. For example, the image recognition-based detection determines that the target object is in the behavior of using the terminal device when the target object holds the terminal device. For another example, the image recognition-based detection determines that the target object is in the behavior of using the terminal device when the front of the head of the target object is opposite to the screen of the terminal device and the distance between the head of the target object and the terminal device satisfies a preset range.

[0074] However, the manual detection is low in efficiency and high in cost, which is not conducive to large-scale behavior detection. When the angle, distance and other information of the terminal device held by the target object changes, when other objects are identified as the terminal device, when the hand or head of the target object is blocked or moves too fast, when the posture of using the terminal device of the target object has new changes, or when the terminal device is small in size or blocked, the accuracy of the detection of the behavior of using the terminal device is affected, resulting in low accuracy of the detection of the behavior of using the terminal device.

[0075] Therefore, there is an urgent need for a method for detecting the behavior of using the terminal device, which can improve the accuracy of detecting the behavior of using the terminal device while ensuring the detection efficiency.

[0076] Figure 1 An exemplary system architecture to which embodiments of the present application can be applied is shown, which includes a server 100. The server 100 can include a processor 110, a communication interface 120 and a memory 130.

[0077] The communication interface 120 is configured to acquire a video stream, for example, video data captured by a monitoring camera in real time.

[0078] The processor 110 is the control center of the server 100, which connects all parts of the server 100 through various interfaces and routes, executes various functions of the server 100 and processes data by running or executing software programs / modules stored in the memory 130 and calling data stored in the memory 130. Optionally, the processor 110 can include one or more processing units.

[0079] The memory 130 can be used to store software programs and modules, and the processor 110 executes various function applications and data processing by running the software programs and modules stored in the memory 130. The memory 130 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function, etc.; and the data storage area can store data created according to business processing, etc. In addition, the memory 130 can include a high-speed random access memory, and can also include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0080] It should be noted that the above Figure 1 The structure shown is only an example, and the embodiments of the present application are not limited thereto.

[0081] Based on the above description, Figure 2 An exemplary flowchart of a detection method of a behavior of using a terminal device provided by an embodiment of the present application is shown in FIG. 2, which includes the following steps: Figure 2 The flowchart includes the following steps:

[0082] In step 210, an image frame in a video stream is acquired.

[0083] In the embodiment of the present application, the video stream is a real-time video stream acquired from a camera according to a video stream protocol, so as to ensure the real-time performance of detecting the behavior of using the terminal device. The video stream protocol includes but is not limited to RTSP (RealTime Streaming Protocol), HTTP (Hyper text Transfer Protocol), WebRTC (Web Real-Time Communications), and the like, and the present application is not limited thereto.

[0084] In one possible implementation, the image frame is acquired from the video stream according to a preset frequency, such as converting the video stream into a picture format at a frequency of 2 frames per second, and then obtaining the image frame. It can be understood that the image frame is acquired periodically, and the preset frequency can be a value preset according to experience, which is not limited herein. In another possible implementation, the image frame is acquired from the video stream according to a preset image parameter (such as resolution, brightness, etc.), such as acquiring an image frame with a resolution greater than 1600x1280 from the video stream. It can be understood that the two ways can be used in parallel to acquire the image frame from the video stream, such as acquiring the image frame according to the preset frequency, and then determining the image frame to be detected from the acquired image frame according to the preset image parameter.

[0085] In step 220, if it is determined that the image frame has at least one target object using the terminal device, a first image corresponding to any target object in the image frame is determined.

[0086] In the embodiment of the present application, the target object is a human object using the terminal device in the image frame, and a human object not using the terminal device is not the target object, i.e., it is determined that the human object not using the terminal device is in a state of not using the terminal device. The first image includes a detection target of the target object, and the detection target includes the terminal device and the limb features of the target object in the first image. For example, the terminal device in the first image is a mobile phone, an iPad, a notebook computer, etc., and the limb features of the target object are hand features, head features, shoulder features, etc. The detection result of the detection target represents the possibility of the target object being in the behavior of using the terminal device, i.e., the greater the value of the detection result of the detection target, the higher the possibility of the target object being in the behavior of using the terminal device.

[0087] In a possible implementation, whether the image frame has the target object using the terminal device is determined according to a pre-trained image recognition model. For example, the image frame is input into the pre-trained image recognition model to obtain a recognition result output by the pre-trained image recognition model, and whether the image frame has the target object using the terminal device is determined according to the recognition result. It can be understood that the pre-trained image recognition model can be pre-trained according to the features of the terminal device, which is not limited here.

[0088] In another possible implementation, whether the image frame has the target object using the terminal device is determined according to a pre-trained behavior detection model. For example, the image frame is input into the behavior detection model to obtain a detection result of the image frame. The detection result of the image frame is a confidence value in the form of a numerical value, which represents the possibility of the image frame having the target object and the terminal device. When the detection result of the image frame is greater than a first threshold value, it is determined that the image frame has at least one target object using the terminal device. For example, the detection result of the image frame is 0.9, and it is determined that the image frame is greater than 0.85 (the first threshold value), so it is determined that the image frame has at least one target object using the terminal device. It can be understood that the first threshold value can be a value pre-set according to experience, such as 0.7, 0.6, etc., which is used to ensure the accuracy of the image frame determined to have at least one target object using the terminal device, which is not limited here. When the detection result of the image frame is less than or equal to the first threshold value, it is determined that the image frame does not have the target object using the terminal device, and then the image frame is discarded, and the next image frame is detected.

[0089] In the embodiments of the present application, based on the input image frame, the output result of the behavior detection model not only includes the detection result of the image frame, but also includes the position coordinates of the at least one target object using the terminal device in the image frame when the image frame has at least one target object using the terminal device. The position coordinates can also be understood as the person bounding box range of the target object in the image frame, or the area where the target object is located in the image frame, which will not be described here.

[0090] Based on the above description, Figure 3 An exemplary schematic diagram of an image frame is provided in the embodiments of the present application, which includes three target objects, object a, object b and object c. Among them, object a and object b are person objects using terminal devices, and object c is a person object not using a terminal device, so object a and object b are target objects. Taking object a as an example, the dashed box corresponding to object a is the person bounding box range of object a in the image frame formed based on the position coordinates of object a, and the person bounding box range of object a includes the terminal device used by object a. Based on the position coordinates of object b in the image frame, the dashed box corresponding to object b is the person bounding box range of object b in the image frame formed based on the position coordinates of object b, and the person bounding box range of object b includes the terminal device used by object b. Based on the position coordinates of object c in the image frame, the dashed box corresponding to object c is the person bounding box range of object c in the image frame formed based on the position coordinates of object c, and the person bounding box range of object c does not include the terminal device used by object c. Figure 3 It can be seen that for any target object, a first image corresponding to the target object is cropped from the image frame according to the position coordinates of the target object in the image frame, and the first image includes the terminal device used by the target object.

[0091] In some embodiments, after the first image corresponding to the target object is determined, the first image is subjected to quality processing, such as magnification, clarity improvement, brightness, etc., so as to improve the accuracy of the behavior detection model in detecting the first image.

[0092] Step 230, inputting the first image into the behavior detection model, and determining that the target object is in the behavior of using the terminal device according to the detection result of the detection target output by the behavior detection model.

[0093] In a possible implementation, the detection result of the detection target is a confidence value in numerical form, which represents the possibility of the target object being in the behavior of using the terminal device. Therefore, the behavior of using the terminal device is determined based on the state of using the terminal device, and therefore the detection result of the detection target also represents the possibility of the target object being in the state of using the terminal device.

[0094] For example, if the detection result of the terminal device in the first image and the detection result of the limb feature of the target object are both greater than the second threshold, it is determined that the target object is in the state of using the terminal device. For example, the confidence of the terminal device in the first image corresponding to the target object is 0.93, the confidence of the hand feature of the target object is 0.87, and the confidence of the head feature of the target object is 0.91. The confidences 0.93, 0.87, and 0.91 are all greater than 0.8 (the second threshold), and it is determined that the target object is in the state of using the terminal device. It can be understood that the second threshold can be a value preset according to experience, such as 0.75 or 0.7, which is used to ensure the accuracy of determining that the target object is in the state of using the terminal device, and is not limited herein. If the detection result of any detection target is less than or equal to the second threshold, it is determined that the target object is not in the state of using the terminal device, and then the first image corresponding to the target object is discarded, and the subsequent operation of determining that the target object is in the behavior of using the terminal device is not performed.

[0095] In another possible implementation, based on the input first image, the output result of the behavior detection model includes not only the detection result of the detection target, but also the position coordinates of the detection target in the first image. The position coordinates can also be understood as the bounding box range of the detection target in the image frame, or the region where the detection target is located in the first image, and details are not repeated herein.

[0096] For the detection target, a second image corresponding to the detection target is cropped from the first image according to the position coordinates of the detection target in the first image. Then, whether the target object is in the state of using the terminal device is determined according to the second image corresponding to the detection target of the target object and / or the detection result of the detection target of the target object. For example, the second image corresponding to the detection target of the target object is the image corresponding to the hand feature, and the position, angle, motion state, shape, and the like of the hand in the second image are weighted and summed with the detection result of the hand feature of the target object to obtain a summation result. Then, whether the target object is in the state of using the terminal device is determined according to the summation result.

[0097] After it is determined that the target object is in the state of using the terminal device, the target object is marked, and then the behavior state of the target object in a preset time period after the current time is recorded. If the target object is in the state of using the terminal device for more than a time threshold in the preset time period, it is determined that the target object is in the behavior of using the terminal device. The time threshold is a value preset according to experience based on the preset time period, such as 80% of the preset time period. For example, if the preset time period is 2 minutes, the time threshold is 96 seconds. That is, if the target object is determined to be in the behavior of using the terminal device for more than 96 seconds in 2 minutes, it is determined that the target object is in the behavior of using the terminal device.

[0098] For example, the image frames are determined from a video stream captured in real time by a monitoring camera, assuming that the image frames are obtained from the video stream at a frequency of 2 frames per second, i.e. 240 frames of images are obtained in 2 minutes. Among them, 200 frames of images determine that the object a is in a state of using the terminal device, and the other 40 frames of images determine that the object a is in a state of not using the terminal device because the image does not have the object a, or the object a does not use the terminal device, or the confidence of the detection target of the object a is less than the second threshold value, and then the time when the object a is in the state of using the terminal device is 100 seconds, and thus it is determined that the object a is in the behavior of using the terminal device.

[0099] Based on the above description, the behavior of the target object using the terminal device can also be determined according to the number of image frames in which the target object is determined to be in the state of using the terminal device. For example, if the number of image frames in which the target object is in the state of using the terminal device within a preset time period after the current time is greater than a number threshold, it is determined that the target object is in the behavior of using the terminal device. For another example, if the target object is determined to be in the state of using the terminal device for a plurality of times (such as 10 times) in succession, it is determined that the target object is in the behavior of using the terminal device.

[0100] After determining the behavior of the target object using the terminal device, an alarm information is triggered. The alarm information includes the location information and / or the identity information of the target object. It can be understood that the alarm information can be displayed by text or broadcast by voice, and the embodiments of the present application are not limited specifically. For example, after determining the behavior of the target object using the terminal device, a pop-up window is displayed in a monitoring display of a monitoring room, and the content of the pop-up window includes the location information of the office, workstation, seat, etc. where the staff in the behavior of using the terminal device is located, and the identity information of the staff such as name, gender, work number, student number, etc.

[0101] In steps 220 and 230, the behavior detection model is obtained based on a self-supervised pre-training model after training based on image data with labels, and the self-supervised pre-training model is obtained based on a deep learning model and a data set without labels, and the labels represent the behavior of using the terminal device. The embodiments of the present application take the deep learning model as an example of Transformer, but the deep learning model is not limited, and the training set can also be called training sample.

[0102] Exemplarily, image data is extracted from historical video streams, and the image data is preprocessed to obtain a training set. The historical video streams can be captured by cameras in different scenes, different positions, and different shooting angles to ensure the richness and reliability of the training samples. The image data is extracted from the historical video streams according to frame extraction processing. The frame extraction processing refers to extracting image data from a data stream at a preset frequency, for example, extracting 2 frames of images per second from the data stream, thereby reducing redundant images and reducing the repetition of training samples.

[0103] The preprocessing includes empty lens processing, image deduplication processing, and general processing.

[0104] Further, the empty lens processing refers to determining images with a person object from the image data. Exemplarily, an empty lens image without a person object is filtered out by a person object recognition model, for example, image data without a staff member at a workstation after the staff member leaves the workstation. Since the empty lens image does not have a person object, the accuracy and reliability of the training samples are improved by removing the empty lens image.

[0105] The image deduplication processing refers to determining images with a similarity less than a preset threshold from the images with the person object. For example, the similarity between images with a person object is calculated by an image hash method or the like, and image deduplication is performed according to the similarity, thereby reducing images with too high similarity, avoiding repetition of training samples, and improving the accuracy of training samples.

[0106] The general processing includes dark processing, encoding exception processing, and clarity processing, which refers to determining images satisfying preset image parameters from images with a similarity less than a preset threshold. The preset image parameters include brightness, encoding parameters, and / or resolution of the image. For example, the dark processing refers to determining images greater than a brightness threshold from images with a similarity less than a preset threshold; the encoding exception processing refers to determining images without decoding exception from images with a similarity less than a preset threshold. For example, a JPEG image can have compression distortion and other exceptions, resulting in image decoding exception; the clarity processing refers to determining images greater than a resolution or pixel threshold from images with a similarity less than a preset threshold, thereby ensuring the quality of the training samples and improving the accuracy and reliability of the model training.

[0107] After the image data is preprocessed, a part of the preprocessed image data is taken as a first data set (also referred to as a first training sample, a label-free training sample) without a label, and another part of the preprocessed image data is taken as a second data set (also referred to as a second training sample, a labeled training sample) with a label based on a manually labeled label, so as to obtain a training set. Wherein, the label of the image of the second training sample includes a positive label representing a positive sample and a negative label representing a negative sample, the positive label includes angle categories of a hand of a person object, a head, a front, a side and a back of a terminal device, and the negative label includes categories such as similar objects of the terminal device and a position direction of the terminal device. Further, the annotation format of the label can be COCO format, and the image annotated by COCO format is stored in JSON format. The label in this format can include information such as information of the image, position of the target, category and attribute. It can be understood that the annotation format of the label is only an example, and the embodiment of the application does not limit the annotation format of the label.

[0108] The positive sample is an image of a normal behavior of using the terminal device, such as a label of the positive sample representing that the action of the person object holding the terminal device is normal, the distance between the head of the person object and the terminal device is normal, the terminal device is in a vertical or horizontal direction (i.e., the position direction of the terminal device is normal), and the screen of the terminal device faces the face of the person object. The negative sample is an image of an abnormal behavior of using the terminal device, such as a label of the negative sample representing that the back cover of the terminal device faces the face of the person object, the action of the person object holding the terminal device is abnormal, and the distance between the head of the person object and the terminal device is abnormal.

[0109] In some embodiments, the negative sample can be generated based on the original image data. For example, a terminal device image is randomly selected to fill a position in the original image data, and the terminal device image is filled into the original image data through rotation, translation, scaling and other operations, thereby generating a negative sample, so as to increase the diversity and quantity of the negative sample and ensure the accuracy and reliability of the training of the behavior detection model.

[0110] After obtaining the first data set and the positive sample and the negative sample, self-supervised training is performed according to the first data set to determine a self-supervised pre-training model, and then supervised training is performed according to the positive sample and the negative sample based on the self-supervised pre-training model to determine the behavior detection model.

[0111] In the embodiment of the application, the Transformer model is taken as an example. Referring to Figure 4 , Figure 4 A flowchart of determining a behavior detection model based on a Transformer model is exemplarily provided in the embodiment of the application. The flowchart includes the following steps:

[0112] Step 410, acquire historical video stream shot by the monitoring camera.

[0113] Step 420, extract image data from the historical video stream; the step can determine the image data from the historical video stream according to the above-mentioned frame extraction processing mode.

[0114] Step 430, determine the first data set; the first data set of the step is a part of the pre-processed image data, which is a training sample without label.

[0115] Step 440, determine the second data set; the second data set of the step is another part of the pre-processed image data, which is a training sample with label.

[0116] Step 450, determine the self-supervised training model; the self-supervised training model of the step is obtained by performing Transformer self-supervised training according to the training sample without label. Further, the training process of the self-supervised pre-training model is as follows: the image in the first data set without label is cut into different small blocks, that is, each small block contains a part of image information, so as to realize that the self-supervised pre-training model predicts through other parts of information in the case of missing a part of information, improve the robustness and generalization ability of the self-supervised pre-training model. Then, the multiple small blocks are randomly arranged, so as to realize that the self-supervised pre-training model does not depend on the order of the small blocks, avoid overfitting and other problems, and improve the accuracy of the self-supervised pre-training model. Then, a part of the small blocks are randomly selected, and the part of the small blocks are covered with black according to the mask index, the mask index is proportional to the number of small blocks to be covered, so as to realize that the self-supervised pre-training model predicts through other parts of information in the case of missing a part of information. Then, the part of the small blocks after covering is input into the encoder for encoding to obtain the vector identifier output by the encoder. For any small block, the vector output by the encoder is added to the position vector representing the position of the small block in the image, and then the position coding of each small block is obtained, the position coding is spliced with the covered small block to form a complete sequence, such as the size of the original image is m*n, and the sequence length corresponding to the original image is m*n. Finally, the sequence corresponding to the original image is input into the self-supervised pre-training model for training, and then the trained self-supervised pre-training model is obtained.

[0117] Step 460, determine positive samples and negative samples; the positive samples and the negative samples of the step are obtained by distinguishing the second data set based on the label category, and the negative samples can be generated based on the original image data of the positive samples.

[0118] Step 470, determine the behavior detection model; the behavior detection model of this step is based on the self-supervised pre-training model, and is determined according to the positive sample and the negative sample for the supervised training of the Transformer. Further, the training process of the behavior detection model is: the positive sample and the negative sample are randomly shuffled, such as according to the shuffle parameter in the data loader (such as PyTorch DataLoader) in the deep learning framework, so as to increase the generalization ability of the behavior detection model, avoid the behavior detection model depending on the order of the positive sample and the negative sample, and improve the robustness of the behavior detection model. The shuffled positive sample and negative sample are divided into training sub-samples and verification samples according to a preset ratio (such as 8:2), that is, 80% of the positive sample and the negative sample are used to train the behavior detection model, and 20% of the positive sample and the negative sample are used to verify the behavior detection model. The training sub-sample is input into the initial behavior detection model; wherein the initial weight of the initial behavior detection model is the weight of the self-supervised pre-training model. The model is iterated with SGD as the optimizer, and in the iteration process, the weight of the behavior detection model is updated by methods such as back propagation and gradient descent of the loss function, until the model converges. Finally, the converged behavior detection model is verified according to the verification sample, and the trained behavior detection model is obtained after the behavior detection model is verified to pass according to the value of map.

[0119] In the embodiment of the application, the training sample is determined by pre-processing the image data, so as to ensure the image quality of the training sample and improve the accuracy of the training of the self-supervised pre-training model and the behavior detection model. The self-supervised pre-training model is determined based on the Transformer self-supervised training, so as to extract the behavior features of similar scenes, reduce the number of labeled training samples, and save the cost of labeling labels. The behavior detection model is determined based on the Transformer supervised training, so as to ensure the accuracy of the detection of the behavior detection model. The behavior detection model is used for detecting the image frame, determining the first image corresponding to the target object, and detecting the detection target of the target object in the first image, so as to realize the secondary detection of the behavior of the terminal device based on the supervised model, and the detection target is part of the features of the target object, that is, part of the features of the target object is detected, the precision of the behavior detection of the target object is improved, and the accuracy of the detection of the behavior of the terminal device is improved.

[0120] Based on the same technical concept, Figure 5 An exemplary structure diagram of a behavior detection device of a terminal device provided by the embodiment of the application is shown, which can execute the flow of the behavior detection method of the terminal device.

[0121] As Figure 5 shown, the device specifically includes:

[0122] The acquisition module 510 is configured to acquire an image frame in a video stream.

[0123] The processing module 520 is configured to, if it is determined that the image frame has at least one target object using a terminal device, determine a first image corresponding to any target object in the image frame; the first image includes a detection target of the target object.

[0124] The first image is input into a behavior detection model, and a detection result of the detection target output by the behavior detection model is used to determine that the target object is in a behavior of using a terminal device; the behavior detection model is obtained by training a self-supervised pre-training model based on image data with labels; the labels represent the behavior of using a terminal device.

[0125] Optionally, the processing module 520 is specifically configured to:

[0126] extract image data from historical video streams;

[0127] preprocess the image data to obtain a training set; the training set includes a first data set without labels and a second data set with labels; the second data set includes positive samples and negative samples; the positive samples are images of normal behaviors of using a terminal device, and the negative samples are images of abnormal behaviors of using a terminal device;

[0128] perform self-supervised training according to the first data set to determine a self-supervised pre-training model;

[0129] perform supervised training based on the self-supervised pre-training model according to the positive samples and the negative samples to determine a behavior detection model.

[0130] Optionally, the processing module 520 is specifically configured to:

[0131] determine images with human objects from the image data;

[0132] determine images with a similarity less than a preset threshold from the images with human objects;

[0133] determine images meeting preset image parameters from the images with a similarity less than a preset threshold; the preset image parameters include brightness, encoding parameters and / or resolution of the images.

[0134] Optionally, the processing module 520 is specifically configured to:

[0135] input the image frame into the behavior detection model to obtain a detection result of the image frame; the detection result of the image frame represents a possibility that the image frame has a target object and a terminal device;

[0136] If the detection result of the image frame is greater than a first threshold value, it is determined that the target object having at least one terminal device in the image frame.

[0137] Optionally, the processing module 520 is specifically configured to:

[0138] For any target object, a first image corresponding to the target object is cropped from the image frame according to the position coordinates of the target object in the image frame; the position coordinates of the target object in the image frame are obtained according to the behavior detection model.

[0139] Optionally, the detection target includes a terminal device and a limb feature of the target object in the first image; the detection result of the detection target represents a possibility that the target object is in a behavior of using the terminal device.

[0140] The processing module 520 is specifically configured to:

[0141] If the detection result of the terminal device in the first image and the detection result of the limb feature of the target object are both greater than a second threshold value, it is determined that the target object is in a state of using the terminal device.

[0142] If, within a preset time period after the current time, the time that the target object is in the state of using the terminal device is greater than a time threshold value, it is determined that the target object is in the behavior of using the terminal device.

[0143] Optionally, the processing module 520 is further configured to:

[0144] After it is determined that the target object is in the behavior of using the terminal device, an alarm information is triggered, and the alarm information includes position information and / or identity information of the target object.

[0145] Based on the same technical concept, the embodiment of the present application further provides a computer device, comprising:

[0146] a memory configured to store program instructions;

[0147] a processor configured to invoke the program instructions stored in the memory, and execute the above-mentioned detection method of the behavior of using the terminal device according to the obtained program.

[0148] Based on the same technical concept, the embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are used to make a computer execute the above-mentioned detection method of the behavior of using the terminal device.

[0149] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In one

[0150] The present application is described in reference to the flow diagrams and / or block diagrams of the methods, apparatus (systems) and computer program products according to this application. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the present application can include functional or operational blocks that may Figure 1 The flow diagrams and / or block diagrams in the present application can include functional or operational blocks that may

[0151] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the present application can include functional or operational blocks that may Figure 1 The flow diagrams and / or block diagrams in the present application can include functional or operational blocks that may

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the present application can include functional or operational blocks that may Figure 1 The flow diagrams and / or block diagrams in the present application can include functional or operational blocks that may

[0153] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A detection method using behavior of a terminal device, characterized by, The method comprises: acquiring an image frame in a video stream; if it is determined that the image frame has at least one target object using a terminal device, determining a first image corresponding to any target object in the image frame; the first image comprises a detection target of the target object; inputting the first image into a behavior detection model, and determining that the target object is in a behavior of using a terminal device according to a detection result of the detection target output by the behavior detection model; the behavior detection model is obtained based on a self-supervised pre-training model after training according to image data with labels; the label represents a behavior of using a terminal device; the behavior detection model is obtained based on a self-supervised pre-training model after training according to image data with labels, comprising: extracting image data from historical video streams; preprocessing the image data to obtain a training set; the training set comprises a first data set without labels and a second data set with labels; the second data set comprises positive samples and negative samples; the positive samples are images of normal behaviors of using a terminal device, and the negative samples are images of abnormal behaviors of using a terminal device; the images of normal behaviors of using a terminal device comprise one or more of the following: images of normal actions of a human object holding a terminal device, images of normal distances between a head of the human object and the terminal device, images of the terminal device in a vertical or horizontal direction, and images of a screen of the terminal device facing a face of the human object; the images of abnormal behaviors of using a terminal device comprise one or more of the following: images of a back cover of the terminal device facing the face of the human object, images of abnormal actions of the human object holding the terminal device, and images of abnormal distances between the head of the human object and the terminal device; performing self-supervised training according to the first data set to determine a self-supervised pre-training model; performing supervised training according to the positive samples and the negative samples based on the self-supervised pre-training model to determine a behavior detection model.

2. The method of claim 1, wherein, The preprocessing of the image data comprises: determining images with human objects from the image data; determining images with a similarity less than a preset threshold from the images with human objects; determining images meeting preset image parameters from the images with a similarity less than a preset threshold; the preset image parameters comprise brightness, encoding parameters and / or resolution of the images.

3. The method of claim 1, wherein, The determination of the image frame having at least one target object using a terminal device comprises: inputting the image frame into the behavior detection model to obtain a detection result of the image frame; the detection result of the image frame represents a possibility that the image frame has a target object and a terminal device; if the detection result of the image frame is greater than a first threshold, it is determined that the image frame has at least one target object using a terminal device.

4. The method of claim 1, wherein, The determination of the first image corresponding to any target object in the image frame comprises: for any target object, the first image corresponding to the target object is cropped from the image frame according to a position coordinate of the target object in the image frame; the position coordinate of the target object in the image frame is obtained according to the behavior detection model.

5. The method of claim 1, wherein, The detection target includes a terminal device in the first image and a limb feature of the target object; and a detection result of the detection target represents a possibility that the target object is in a behavior of using the terminal device. The method further includes: if the detection result of the terminal device in the first image and the detection result of the limb feature of the target object are both greater than a second threshold, determining that the target object is in a state of using the terminal device; if a time that the target object is in the state of using the terminal device is greater than a time threshold within a preset time period after a current time, determining that the target object is in the behavior of using the terminal device.

6. The method according to any one of claims 1 to 3, wherein The method further includes: triggering alarm information, the alarm information including location information and / or identity information of the target object.

7. A detection apparatus using behavior of a terminal device, characterized by, The method further includes: acquiring an image frame in a video stream; if it is determined that there is at least one target object using a terminal device in the image frame, determining a first image corresponding to any target object in the image frame; the first image including a detection target of the target object; inputting the first image into a behavior detection model, and determining, according to a detection result of the detection target output by the behavior detection model, that the target object is in a behavior of using the terminal device. The behavior detection model is obtained based on a self-supervised pre-training model according to image data with labels; and the labels represent a behavior of using a terminal device. The method further includes: extracting image data from a historical video stream; preprocessing the image data to obtain a training set; the training set including a first data set without labels and a second data set with labels; the second data set including positive samples and negative samples; the positive samples being images of a normal behavior of using a terminal device, and the negative samples being images of an abnormal behavior of using a terminal device; the images of the normal behavior of using the terminal device including one or more of the following: images of a normal action of a character object holding a terminal device, images of a normal distance between a head of the character object and the terminal device, images of the terminal device being in a vertical or horizontal direction, and images of a screen of the terminal device facing a face of the character object; and the images of the abnormal behavior of using the terminal device including one or more of the following: images of a back cover of the terminal device facing the face of the character object, images of an abnormal action of the character object holding the terminal device, and images of an abnormal distance between the head of the character object and the terminal device; performing self-supervised training according to the first data set to determine a self-supervised pre-training model; performing supervised training according to the positive samples and the negative samples based on the self-supervised pre-training model to determine a behavior detection model.

8. A computer device, comprising: The method further includes: a memory for storing program instructions; a processor for invoking the program instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to perform the method of any one of claims 1 to 6.

Images