Calling detection method and device based on attitude information reasoning

Abstract

The invention discloses a call making detection method and device based on attitude information reasoning, relates to the technical field of image processing, and solves the problems of distraction, reduced working efficiency and potential safety hazards caused by using a mobile phone to call in a specific working area. A set of intelligent detection system is established, illegal calling behaviors are automatically recorded and analyzed, data support is provided for managers, more effective communication behavior specifications are conveniently formulated and executed, and a more efficient and safer working environment is created. In addition, the problems of high time cost, limited monitoring effect and the like of a traditional manual monitoring mode are solved, and the recognition efficiency and accuracy are improved at the same time. It is ensured that improper behaviors can be effectively restrained, interference to daily work of staff can be reduced, and the actual application value and the user satisfaction degree of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for detecting a phone call based on posture information reasoning. Background Art

[0002] Nowadays, mobile phone communication has become an indispensable part of people's daily life. However, in certain specific areas, such as the cab, production line and factory workshop, making phone calls may distract attention, affect work efficiency, and even cause safety hazards. Therefore, developing a technology that can automatically detect phone calls in these areas is of great significance for improving work efficiency, ensuring production safety and standardizing communication behavior. With the rapid development of artificial intelligence technology, significant progress has been made in the fields of human posture analysis, behavior analysis and target recognition. Through computer vision and deep learning and other methods, automatic recognition and analysis of phone calls by people in specific areas can be achieved, which solves the problems of high time cost and limited monitoring effect of traditional manual monitoring methods, and greatly improves the efficiency and accuracy of recognition.

[0003] "Smoking and phone call detection method and system combining action recognition and target detection CN116580452A", this technology provides an improved yolov5 target detection algorithm and 2D key point and 3D human posture estimation, but this technology does not perform specific algorithmic analysis and judgment on the key points of the human body, and there is a high false detection rate problem in complex background conditions.

[0004] "A method and system for detecting smoking and talking on the phone at a gas station based on deep learning CN114387557A", this technology provides a detection method based on the yolov3 deep learning model, and also sets up a region of interest module, a head target detection module, an image classification module and a finite state machine module. However, the detection model used in this method is relatively old, and it does not use posture information detection and analysis methods and multi-model fusion judgment methods, and cannot solve the problem of low recognition rate under complex backgrounds.

[0005] "A video-based phone call detection method CN109684990B" provides a phone call detection method based on the SSD deep model, and analyzes and judges by setting threshold screening, gesture and mouth position relationship, and finally adds a cumulative frame number judgment method. However, the deep learning model used in this method has a large number of parameters and models, which will lead to a slow detection speed.

[0006] It can be seen that most of the existing methods use public data sets or annotate the original data of the collected video, and do not perform targeted data enhancement on the collected image data sets. At the same time, the common phone call detection lacks the algorithm reasoning process combined with the posture distance and angle information. In addition, the common phone call detection has no continuous frame judgment or adopts a simple continuous frame judgment process. Summary of the invention

[0007] In view of the above problems, the present invention provides a method and device for detecting a phone call based on posture information reasoning, which are used to overcome the above problems or at least partially solve the above problems.

[0008] The present invention provides the following scheme:

[0009] A method for detecting a phone call based on posture information reasoning, comprising:

[0010] Obtaining a video stream of the monitoring area, and converting the video stream into image frames to obtain a picture to be tested;

[0011] Inputting the image to be tested into a phone call detection neural network algorithm based on posture information reasoning, so that the phone call detection neural network algorithm outputs a phone call detection result;

[0012] The calling detection neural network algorithm includes a posture information detection model, a calling detection model and a human posture analysis algorithm; the calling detection model includes a calling detection sub-model and a mobile phone auxiliary detection sub-model;

[0013] The posture information detection model is used to process the image to be tested to obtain a first image with position coordinates of key points of human posture;

[0014] The calling detection sub-model and the mobile phone auxiliary detection sub-model are used to receive the first picture and detect the mobile phone in the first picture, so as to determine that the first picture is a target picture after detecting that the first picture contains a mobile phone;

[0015] The human posture analysis algorithm is used to perform human posture analysis on the relative position distances and angles of various key points of the human body in the target image in combination with the position coordinates of the key points of the human posture, so as to output the phone call detection result according to the analysis result.

[0016] Preferably: the mutual position distance of each key point is calculated by using the Euclidean distance formula of the image two-dimensional space with the upper left corner of the target image as the coordinate origin; the angle is calculated by using the two-dimensional image vector method;

[0017] The human posture analysis algorithm includes distance determination and angle determination, and if either the distance determination or the angle determination is met, it is determined that there is a phone call in the target image.

[0018] Preferably, the key points of the human body posture include nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip and right hip.

[0019] Preferably, the distance determination includes:

[0020] Calculate the distance from the center point of the phone to the left wrist and the distance to the right wrist, compare the two distances, and determine that the smaller distance value indicates that the phone is in the corresponding hand position;

[0021] After determining that the Y-axis coordinate value of the wrist key point corresponding to the hand position is less than the Y-axis coordinate value of the elbow, it is determined that the hand holding the mobile phone has been lifted;

[0022] Calculate the distance from the center point of the mobile phone to the wrist at the corresponding hand position and the distance from the center point of the mobile phone to the corresponding shoulder at the corresponding hand position. When the distance to the wrist is less than the distance to the corresponding shoulder, determine that the first distance determination is met.

[0023] Calculate the distance from the center point of the mobile phone to the nose and the distance from the center point of the mobile phone to the corresponding elbow of the corresponding hand position, and when the distance to the nose is less than the distance to the corresponding elbow, determine that the second distance determination is satisfied;

[0024] Calculate the distance from the corresponding shoulder of the corresponding hand position to the corresponding wrist of the corresponding hand position, and the distance from the corresponding shoulder of the corresponding hand position to the corresponding elbow of the corresponding hand position. When the distance to the corresponding wrist is less than the distance to the corresponding elbow, it is determined that the third distance judgment is met.

[0025] Preferably: the angle determination includes:

[0026] Calculate the first angle of the wrist position, the elbow position, and the shoulder position corresponding to the hand position according to the vector method, and determine that the first angle is less than 90°, then determine that the first angle determination is satisfied;

[0027] Calculate the second angle between the center point position of the mobile phone, the shoulder position corresponding to the hand position, and the hip position corresponding to the hand position, and determine that the second angle is greater than 90° and less than 180°, then determine that the second angle judgment is met.

[0028] Preferably: converting the video stream into image frames to obtain a number of pictures to be tested, inputting the pictures to be tested into a posture information detection model, and storing the detection results into a storage queue;

[0029] A continuous frame analysis algorithm is used to perform initial judgment on continuous frames. The initial judgment includes making a judgment when the storage queue has 10 frames of images stored. If more than 5 frames of images meet the judgment of the posture information detection model, the first picture with the position coordinates of the key points of the human posture is obtained and the judgment conditions are met and the next judgment can be entered; otherwise, the queue clears all images.

[0030] Preferably: a plurality of the first images are respectively input into a phone call detection sub-model and a mobile phone auxiliary detection sub-model, so that the phone call detection sub-model and the mobile phone auxiliary detection sub-model output a plurality of the phone call detection results;

[0031] The storage queue receives a number of the call detection results;

[0032] The continuous frame analysis algorithm is used to perform continuous frame judgment, and the continuous frame judgment includes making a judgment when the storage queue is full of the phone call detection results corresponding to 30 frames of images. If more than 20 frames of images meet the human posture analysis judgment, the result of successfully detecting the phone call action is output; otherwise, the storage queue clears all data and waits to store the data incoming in the next second.

[0033] Preferably: the training data set for the phone call detection neural network algorithm includes a training human posture data set and a phone call detection data set after data enhancement processing; the data enhancement includes at least image flipping, image rotation, image random cropping, image illumination transformation and image color transformation.

[0034] Preferably, the training human posture dataset is obtained by:

[0035] Acquire a public human posture dataset as a first posture dataset;

[0036] In the target area, video images containing pedestrians are acquired through surveillance videos, and human posture information of pedestrians in the video is annotated to form a second posture data set;

[0037] Based on the first posture data set and the second posture data set, images of pedestrians are individually cropped and human posture information is annotated to form a third posture data set;

[0038] Combining the first posture data set, the second posture data set and the third posture data set to obtain the training human posture data set;

[0039] The phone call detection dataset is obtained in the following way:

[0040] Obtain images of people making phone calls from a public dataset, use image annotation software to annotate the collected images with rectangular frames, annotate the mobile phone and the hand position overlapping the mobile phone, and label them as "making phone calls" to obtain the first phone call dataset;

[0041] In the target area, a video containing a person making a phone call is obtained through surveillance video, the video containing the person making a phone call is intercepted into a video frame image, and the video frame image is annotated with a rectangular frame using image annotation software, the annotated content is a mobile phone and a hand position overlapping with the mobile phone, and the label name is "making a phone call" to obtain a second phone call data set;

[0042] Obtain mobile phone pictures in public data sets and on the Internet; use image annotation software to annotate the mobile phone pictures with rectangular frames, with the annotation content being the location of the mobile phone and the label name being mobile phone to obtain a third phone call data set;

[0043] The first phone call data set and the second phone call data set are used to train the phone call detection sub-model, and the third phone call data set is used to train the auxiliary mobile phone detection sub-model.

[0044] A phone call detection device based on posture information reasoning is used to execute the above-mentioned phone call detection method based on posture information reasoning, and the device comprises:

[0045] A video stream acquisition unit, used to acquire a video stream of a monitoring area, and convert the video stream into an image frame to obtain a picture to be tested;

[0046] A phone call detection result output unit, used for inputting the image to be tested into a phone call detection neural network algorithm based on posture information reasoning, so that the phone call detection neural network algorithm outputs a phone call detection result;

[0047] The calling detection neural network algorithm includes a posture information detection model, a calling detection model, a human posture analysis algorithm and a continuous frame analysis algorithm; the calling detection model includes a calling detection sub-model and a mobile phone auxiliary detection sub-model;

[0048] The posture information detection model is used to process the image to be tested to obtain a first image with position coordinates of key points of human posture;

[0049] The phone call detection sub-model and the mobile phone auxiliary detection sub-model are used to receive the first picture and detect the mobile phone in the first picture, so as to determine that the first picture is a target picture after detecting that the first picture contains a mobile phone;

[0050] The human posture analysis algorithm is used to perform human posture analysis on the relative position distances and angles of various key points of the human body in the target image in combination with the position coordinates of the key points of the human posture, so as to output the phone call detection result according to the analysis result.

[0051] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0052] The embodiment of the present application provides a method and device for detecting phone calls based on posture information reasoning. The method solves the problems of distraction, decreased work efficiency and potential safety hazards caused by using mobile phones to make calls in a specific work area. An intelligent detection system has been established to automatically record and analyze illegal phone calls, provide data support for managers, and facilitate the formulation and implementation of more effective communication behavior norms to create a more efficient and safe working environment. It also solves the problems of high time cost and limited monitoring effect in traditional manual monitoring methods, while improving recognition efficiency and accuracy. It ensures that inappropriate behavior can be effectively curbed, while reducing interference with employees' daily work, and improving the practical application value and user satisfaction of the system.

[0053] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 is a flow chart of a method for detecting a phone call based on posture information reasoning provided by an embodiment of the present invention;

[0056] Figure 2 is a data processing flow chart of a call detection neural network model provided by an embodiment of the present invention;

[0057] Figure 3 It is a diagram of building a neural network model for phone call detection provided by an embodiment of the present invention;

[0058] Figure 4 is an overall flow chart of a human posture analysis algorithm provided by an embodiment of the present invention;

[0059] Figure 5 is an overall flow chart of a continuous frame analysis algorithm provided by an embodiment of the present invention;

[0060] Figure 6It is a structural schematic diagram of a phone call detection device based on posture information reasoning provided by an embodiment of the present invention;

[0061] Figure 7 It is a structural schematic diagram of a phone call detection device based on posture information reasoning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0063] See also Figure 1 , a method for detecting a phone call based on posture information reasoning is provided in an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0064] S101: Acquire a video stream of a monitoring area, and convert the video stream into image frames to obtain a picture to be tested;

[0065] S102: inputting the image to be tested into a phone call detection neural network algorithm based on posture information reasoning, so that the phone call detection neural network algorithm outputs a phone call detection result;

[0066] The calling detection neural network algorithm includes a posture information detection model, a calling detection model, a human posture analysis algorithm and a continuous frame analysis algorithm; the calling detection model includes a calling detection sub-model and a mobile phone auxiliary detection sub-model;

[0067] The posture information detection model is used to process the image to be tested to obtain a first image with position coordinates of key points of human posture;

[0068] The calling detection sub-model and the mobile phone auxiliary detection sub-model are used to receive the first picture and detect the mobile phone in the first picture, so as to determine that the first picture is a target picture after detecting that the first picture contains a mobile phone;

[0069] The human posture analysis algorithm is used to perform human posture analysis on the relative position distances and angles of various key points of the human body in the target image in combination with the position coordinates of the key points of the human posture, so as to output the phone call detection result according to the analysis result.

[0070] In specific implementation, the embodiment of the present application can provide that the mutual position distance of each key point is calculated by using the upper left corner of the target image as the coordinate origin and adopting the Euclidean distance formula of the image two-dimensional space; the angle is calculated by using the two-dimensional image vector method;

[0071] The human body posture analysis includes distance determination and angle determination, and if either the distance determination or the angle determination is met, it is determined that there is a phone call in the target image.

[0072] Furthermore, in order to reduce the amount of parameters and calculations, the embodiment of the present application may provide that the key points of the human body posture include nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip and right hip.

[0073] In specific implementation, the embodiment of the present application may provide that the distance determination includes:

[0074] Calculate the distance from the center point of the phone to the left wrist and the distance to the right wrist, compare the two distances, and determine that the smaller distance value indicates that the phone is in the corresponding hand position;

[0075] After determining that the Y-axis coordinate value of the wrist key point corresponding to the hand position is less than the Y-axis coordinate value of the elbow, it is determined that the hand holding the mobile phone has been lifted;

[0076] Calculate the distance from the center point of the mobile phone to the wrist at the corresponding hand position and the distance from the center point of the mobile phone to the corresponding shoulder at the corresponding hand position. When the distance to the wrist is less than the distance to the corresponding shoulder, determine that the first distance determination is met.

[0077] Calculate the distance from the center point of the mobile phone to the nose and the distance from the center point of the mobile phone to the corresponding elbow of the corresponding hand position, and when the distance to the nose is less than the distance to the corresponding elbow, determine that the second distance determination is satisfied;

[0078] Calculate the distance from the corresponding shoulder of the corresponding hand position to the corresponding wrist of the corresponding hand position, and the distance from the corresponding shoulder of the corresponding hand position to the corresponding elbow of the corresponding hand position. When the distance to the corresponding wrist is less than the distance to the corresponding elbow, it is determined that the third distance judgment is met.

[0079] The angle determination includes:

[0080] Calculate the first angle of the wrist position, the elbow position, and the shoulder position corresponding to the hand position according to the vector method, and determine that the first angle is less than 90°, then determine that the first angle determination is satisfied;

[0081] Calculate the second angle between the center point position of the mobile phone, the shoulder position corresponding to the hand position, and the hip position corresponding to the hand position, and determine that the second angle is greater than 90° and less than 180°, then determine that the second angle judgment is met.

[0082] It should be noted that the above-mentioned distance position judgment and angle size judgment only need to meet one of the requirements.

[0083] In order to further improve the accuracy of the judgment, the embodiment of the present application can also adopt a continuous frame analysis algorithm. When it is specifically implemented, the embodiment of the present application will create a storage queue, and the storage queue receives a number of the phone call detection results. First, an initial judgment of the continuous frame will be performed. When the queue stores 10 frames of images, a judgment will be made. If more than 5 frames of images meet the human posture analysis judgment, the judgment is satisfied and the next judgment can be entered; otherwise, the queue clears all images. Then a continuous frame judgment is performed. When the storage queue is full of 30 frames of images corresponding to the phone call detection results, a judgment is made. If more than 20 frames of images meet the human posture analysis judgment standard, the result of successfully detecting the phone call action is output; otherwise, the storage queue clears all data and waits for the data incoming in the next second to be stored.

[0084] In order to further improve the model training effect, the embodiment of the present application can also provide the data set for training the phone call detection neural network, including a training human posture data set and a phone call detection data set after data enhancement processing; the data enhancement includes at least image flipping, image rotation, image random cropping, image illumination transformation and image color transformation.

[0085] The training human body posture data set is obtained in the following manner:

[0086] Acquire a public human posture dataset as a first posture dataset;

[0087] In the target area, video images containing pedestrians are acquired through surveillance videos, and human posture information of pedestrians in the video is annotated to form a second posture data set;

[0088] Based on the first posture data set and the second posture data set, images of pedestrians are individually cropped and human posture information is annotated to form a third posture data set;

[0089] Combining the first posture data set, the second posture data set and the third posture data set to obtain the training human posture data set;

[0090] The phone call detection dataset is obtained in the following way:

[0091] Obtain images of people making phone calls from a public dataset, use image annotation software to annotate the collected images with rectangular frames, annotate the mobile phone and the hand position overlapping the mobile phone, and label them as "making phone calls" to obtain the first phone call dataset;

[0092] In the target area, a video containing a person making a phone call is obtained through surveillance video, the video is intercepted into a video frame image, and the video frame image is annotated with a rectangular frame using image annotation software, the annotated content is a mobile phone and a hand position that overlaps with the mobile phone, and the label name is "making a phone call" to obtain a second phone call data set;

[0093] Obtain mobile phone pictures from public datasets and the Internet; use image annotation software to annotate the mobile phone pictures with rectangular frames, with the annotation content being the location of the mobile phone and the label name being mobile phone to obtain a third phone call dataset.

[0094] The first phone call data set and the second phone call data set are used to train the phone call detection sub-model, and the third phone call data set is used to train the auxiliary mobile phone detection sub-model.

[0095] The method for detecting phone calls based on posture information reasoning provided by the embodiment of the present application meets the needs of detecting phone calls in specific areas. During the process of acquiring training data, a detection data set for the scene can be produced, and the data set can be expanded and enhanced to adapt to the characteristics of the detection algorithm. The human body posture detection model is used to obtain the key point information of the human body, and the phone call detection model and the lightweight mobile phone auxiliary detection sub-model are used to detect and judge the person's phone calls. Combined with the human body posture information, the posture position and angle analysis of the phone call detection results are performed. Continuous frame judgment can also be used to analyze and determine the final detection result.

[0096] The method provided in this application is described in detail below through specific implementation methods.

[0097] The embodiment of the present application provides a method for detecting phone calls based on posture information reasoning, which aims to solve the problems of distraction, reduced work efficiency and potential safety hazards caused by using mobile phone calls in a specific work area through video surveillance data and related detection algorithm deployment. The method includes the following four points:

[0098] (1) Construct a human posture dataset for a specific area and a phone call detection dataset.

[0099] (2) Construct a neural network algorithm model for phone call detection based on posture information reasoning.

[0100] (3) Construct a human posture analysis algorithm.

[0101] (4) Construct a continuous frame judgment and analysis algorithm.

[0102] The overall flow chart of this method is as follows Figure 2 As shown:

[0103] Construct a human posture dataset for a specific area and a phone call detection dataset.

[0104] The human posture dataset consists of the following three parts:

[0105] (1) Use public human posture datasets.

[0106] (2) Video images containing pedestrians are obtained through surveillance videos in a specific area, and the human posture information of pedestrians in the video is annotated to form a data set.

[0107] (3) Based on the dataset in Part 2, pedestrian images are individually cropped and human posture information is annotated to form a dataset.

[0108] The above three data sets are combined to train a human posture detection model in a specific area.

[0109] The phone call detection dataset consists of the following three parts:

[0110] (1) Collect images from public datasets containing people making phone calls, where a person making a phone call specifically refers to placing a phone next to their ear. Use image annotation software to annotate the collected images with a rectangular frame, with the annotated content being the phone and the hand position overlapping the phone, and the label name being "making a phone call".

[0111] (2) In a specific area, a video containing a person making a phone call is obtained through surveillance video. The person making a phone call specifically refers to an action such as putting a phone to the ear and continuing for a period of time in the video. The video is intercepted into a video frame image, and the video frame image is annotated with a rectangular frame using image annotation software. The annotation content is the phone and the hand position that coincides with the phone, and the label name is "making a phone call".

[0112] (3) Collect smartphone images from public datasets and the Internet. Use image annotation software to annotate the above images with rectangular frames, with the annotated content being the location of the smartphone and the label name being "phone".

[0113] The first and second data sets are used to train a phone call detection model, and the third data set is used to train an auxiliary phone detection model.

[0114] Data enhancement is performed on the human posture dataset and the phone call detection dataset images, and the training set, the test set, and the validation set are obtained by allocating them in a ratio of 7:2:1; specifically, the data enhancement includes image flipping, image rotation, image random cropping, image illumination transformation, and image color transformation.

[0115] Then, a posture information-based call detection neural network algorithm model is constructed. The posture information-based call detection model includes a posture information detection model, a call detection sub-model, and a mobile phone auxiliary detection sub-model.

[0116] The construction of the neural network model for phone detection based on posture information reasoning is as follows Figure 3 as shown.

[0117] The video image frame first enters the posture information detection model for processing to obtain a picture with the position information of the key points of the human posture. At the same time, a preliminary estimate is made for the continuous pictures. If the requirements are met, they are input into the phone call detection sub-model and the mobile phone auxiliary detection sub-model to obtain the phone call detection result of a single frame of the video.

[0118] Use a pre-made image dataset to train a phone detection neural network algorithm model for pose information reasoning.

[0119] The calling detection sub-model uses the yolov8 large-scale target detection model and is trained using image data labeled as calling to detect targets with calling features.

[0120] The mobile phone auxiliary detection sub-model uses the yolov8 lightweight target detection model and is trained using image data labeled as mobile phones to detect mobile phones in images.

[0121] The posture information detection model uses the yolov8 human posture detection model and is trained using image data labeled as posture to detect people in the image and calculate the position coordinates of the key points of their human posture in the image. In order to reduce the amount of parameters and calculations, this method selects the key points as the nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip and right hip.

[0122] Next, we build a human posture analysis algorithm. The human posture analysis algorithm makes a more accurate judgment on the phone call action by calculating the relative position distance and angle change of each key point of the human body in the image. The specific process is as follows:

[0123] The target image that has been judged as a positive sample by the phone call detection model is analyzed for human posture. The first step is distance position analysis. When calculating the distance, the upper left corner of the image is used as the coordinate origin, and the Euclidean distance formula in the two-dimensional image space is used.

[0124] Given the coordinates of two points P1(x1,y1) and P2(x2,y2) in the two-dimensional space of the image, the distance calculation formula from P1 to P2 is as follows:

[0125]

[0126] The coordinate position of the mobile phone in the image has been obtained by the mobile phone assisted detection model. Here, it is assumed that the position of the center point of the mobile phone is Cp, and the coordinates of the human body key points are calculated by the pose information detection model. Here, it is assumed that the coordinate position of the nose is L_n, the coordinate position of the left shoulder is L_ls, the coordinate position of the right shoulder is L_rs, the coordinate position of the left elbow is L_le, the coordinate position of the right elbow is L_re, the coordinate position of the left wrist is L_lw, the coordinate position of the right wrist is L_rw, the distance from the center point of the left hip is L_lh, and the distance from the center point of the right hip is L_rh.

[0127] First, determine whether the mobile phone is in the left hand or the right hand. Calculate the distance D(Cp, L_lw) from the center point position of the mobile phone to the left wrist and the distance D(Cp, L_lr) to the right wrist, and compare their magnitudes. The smaller value indicates that the mobile phone is in the corresponding hand position.

[0128] After determining whether the mobile phone is in the left hand or the right hand, perform the raising hand action determination. When the Y-axis coordinate value of the wrist key point is less than the Y-axis coordinate value of the elbow, it is considered that the hand holding the mobile phone has been raised.

[0129] When the raising hand determination is satisfied, assume that the left hand is raised at this time. According to the above distance formula, enter the first distance determination. Specifically: calculate the distance D(Cp, L_lw) from the center point of the mobile phone to the wrist and the distance D(Cp, L_ls) from the center point of the mobile phone to the left shoulder. If D(Cp, L_lw) < D(Cp, L_ls), then the first distance determination is satisfied.

[0130] After the first distance determination is completed, perform the second distance determination. Specifically: calculate the distance D(Cp, L_n) from the center point of the mobile phone to the nose and the distance D(Cp, L_le) from the center point of the mobile phone to the left elbow. If D(Cp, L_n) < D(Cp, L_le), then the second distance determination is satisfied.

[0131] After the second distance determination is completed, perform the third distance determination. Specifically: calculate the distance D(C_ls, L_le) from the left shoulder to the left elbow and the distance D(Cls, L_lw) from the left shoulder to the left wrist. If D(C_ls, L_le) < D(Cls, L_lw), then the third distance determination is satisfied.

[0132] After the distance determination is completed, perform the angle determination. When calculating the angle, the two-dimensional image vector method is used to calculate the angle. Specifically:

[0133] Give the positions of three coordinate points A1(x1, y1), A2(x2, y2), and A3(x3, y3) in the two-dimensional image.

[0134] Then ∠(A1A2A3) represents the included angle between the straight lines A1A2 and A2A3, and its calculation process is as follows:

[0135] (1) Calculate the vector:

[0136] vector It means pointing from A2 to A1, that is:

[0137]

[0138] vector It means pointing from A2 to A3, that is:

[0139]

[0140] (2) Calculate the dot product between two vectors:

[0141]

[0142] (3) Calculate the modulus of two vectors:

[0143]

[0144] (4) Calculate the cosine of the angle:

[0145]

[0146] (5) Calculate the angle value:

[0147]

[0148] First, the first angle judgment is performed, specifically: calculate the angle between the wrist position, the elbow position, and the shoulder position. Assuming that the mobile phone is held in the left hand at this time, the angle is ∠(L_lwL_leL_ls) according to the above vector method. Determine the size of the angle. If ∠(L_lwL_leL_ls)<90°, the first angle judgment is satisfied, otherwise it is not satisfied.

[0149] After the first angle determination is completed, the second angle determination is entered, specifically: calculate the angle between the center point position of the mobile phone, the shoulder position, and the hip position. Assuming that the mobile phone is held in the left hand at this time, the angle obtained according to the above vector method is ∠(CpL_lsL_lh). Determine the size of the angle. If 90°<∠(CpL_lsL_lh)<180°, the second angle determination is satisfied, otherwise it is not satisfied.

[0150] It should be noted that the above distance position judgment and angle size judgment only need to meet one of the requirements. In addition, considering that the key point positions required for the above distance position judgment and angle size judgment are not detected due to some special reasons in the image, the confidence threshold of the phone call detection model and the mobile phone auxiliary detection model is increased for judgment.

[0151] The overall flow chart of the human posture analysis algorithm constructed by this method is as follows: Figure 4 as shown.

[0152] Finally, we build a continuous frame analysis algorithm. For the action of making a phone call, we set its duration to at least one second. The common surveillance camera frame rate is 30 frames per second, which means 30 images are generated in one second. The specific process of continuous frame judgment and analysis is as follows:

[0153] Image frames that meet the human posture analysis judgment are retained for continuous frame judgment analysis. First, a storage queue Dq is constructed with a maximum capacity of 30, which can store 30 frames of input images. In order to reduce the amount of calculation and storage space, the continuous frame judgment will first make an initial judgment, which can be regarded as the starting point of the phone call action. Continuous frame initial judgment: When the queue has 10 frames of images stored, a judgment is made. If more than 5 frames of images meet the human posture analysis judgment, the judgment is satisfied and the next judgment can be entered. Otherwise, the queue clears all images.

[0154] Then, continuous frame judgment is performed: after the initial continuous frame judgment, the queue continues to receive images. When the queue is full of 30 frames of images, judgment is performed. If more than 20 frames of images meet the human posture analysis judgment, the judgment is satisfied and the result of successful detection of the phone call action is output. Otherwise, the queue clears all images and waits for the image data to be received in the next second. The overall flow chart of continuous frame judgment analysis constructed by this method is as follows: Figure 5 shown.

[0155] In summary, the phone call detection method based on posture information reasoning provided by this application solves the problems of distraction, reduced work efficiency and potential safety hazards caused by using mobile phones to make calls in a specific work area. An intelligent detection system has been established to automatically record and analyze illegal phone calls, provide data support for managers, and facilitate the formulation and implementation of more effective communication behavior norms to create a more efficient and safe working environment. It also solves the problems of high time cost and limited monitoring effect of traditional manual monitoring methods, while improving recognition efficiency and accuracy. It ensures that inappropriate behavior can be effectively curbed, while reducing interference with employees' daily work, and improving the practical application value and user satisfaction of the system.

[0156] See also Figure 6 The embodiment of the present application can also provide a phone call detection device based on posture information reasoning, such as Figure 6 As shown, for executing the above-mentioned method for detecting a phone call based on posture information reasoning, the device may include:

[0157] The video stream acquisition unit 601 is used to acquire the video stream of the monitoring area, and convert the video stream into image frames to obtain the image to be tested;

[0158] A phone call detection result output unit 602, used to input the image to be tested into a phone call detection neural network algorithm based on posture information reasoning, so that the phone call detection neural network algorithm outputs a phone call detection result;

[0159] The calling detection neural network algorithm includes a posture information detection model, a calling detection model, a human posture analysis algorithm, and a continuous frame analysis algorithm; the calling detection model includes a calling detection sub-model and a mobile phone auxiliary detection sub-model;

[0160] The posture information detection model is used to process the image to be tested to obtain a first image with position coordinates of key points of human posture;

[0161] The calling detection sub-model and the mobile phone auxiliary detection sub-model are used to receive the first picture and detect the mobile phone in the first picture, so as to determine that the first picture is a target picture after detecting that the first picture contains a mobile phone;

[0162] The human posture analysis algorithm is used to perform human posture analysis on the relative position distances and angles of various key points of the human body in the target image in combination with the position coordinates of the key points of the human posture, so as to output the phone call detection result according to the analysis result.

[0163] The continuous frame analysis algorithm has continuous frame initial judgment and continuous frame judgment, which are used to analyze the continuous calling detection results to output the final calling detection results.

[0164] The embodiment of the present application may also provide a phone call detection device, the device comprising a processor and a memory:

[0165] The memory is used to store program code and transmit the program code to the processor;

[0166] The processor is used to execute the steps of the above-mentioned method for detecting phone calls based on posture information reasoning according to the instructions in the program code.

[0167] like Figure 7 As shown, a phone call detection device based on posture information reasoning provided by an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.

[0168] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices.

[0169] The processor 10 may call the program stored in the memory 11. Specifically, the processor 10 may execute the operations in the embodiment of the method for detecting a phone call based on inference of posture information.

[0170] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0171] Acquire a video stream of the monitoring area, convert the video stream into an image frame to obtain a picture to be tested; input the picture to be tested into a phone call detection neural network algorithm based on posture information reasoning, so that the phone call detection neural network algorithm outputs a phone call detection result;

[0172] The calling detection neural network algorithm includes a posture information detection model, a calling detection model and a human posture analysis algorithm; the calling detection model includes a calling detection sub-model and a mobile phone auxiliary detection sub-model;

[0173] The posture information detection model is used to process the image to be tested to obtain a first image with position coordinates of key points of human posture;

[0174] The calling detection sub-model and the mobile phone auxiliary detection sub-model are used to receive the first picture and detect the mobile phone in the first picture, so as to determine that the first picture is a target picture after detecting that the first picture contains a mobile phone;

[0175] The human posture analysis algorithm is used to perform human posture analysis on the relative position distances and angles of various key points of the human body in the target image in combination with the position coordinates of the key points of the human posture, so as to output the phone call detection result according to the analysis result.

[0176] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0177] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0178] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0179] Of course, it should be noted that Figure 7 The structure shown does not constitute a limitation on the phone call detection device in the embodiment of the present application. In actual applications, the phone call detection device may include Figure 7 More or fewer components than shown, or combinations of certain components.

[0180] An embodiment of the present application may also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the steps of the above-mentioned phone call detection method based on posture information reasoning.

[0181] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0182] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0183] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0184] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for detecting phone calls based on posture information reasoning, characterized in that: include: Obtaining a video stream of the monitoring area, and converting the video stream into image frames to obtain a picture to be tested; Inputting the image to be tested into a phone call detection neural network algorithm based on posture information reasoning, so that the phone call detection neural network algorithm outputs a phone call detection result; The calling detection neural network algorithm includes a posture information detection model, a calling detection model and a human posture analysis algorithm; the calling detection model includes a calling detection sub-model and a mobile phone auxiliary detection sub-model; The posture information detection model is used to process the image to be tested to obtain a first image with position coordinates of key points of human posture; The calling detection sub-model and the mobile phone auxiliary detection sub-model are used to receive the first picture and detect the mobile phone in the first picture, so as to determine that the first picture is a target picture after detecting that the first picture contains a mobile phone; The human posture analysis algorithm is used to perform human posture analysis on the relative position distances and angles of various key points of the human body in the target image in combination with the position coordinates of the key points of the human posture, so as to output the phone call detection result according to the analysis result.

2. The method for detecting phone calls based on posture information reasoning according to claim 1, characterized in that: The mutual position distance of each key point is calculated by using the Euclidean distance formula of the image two-dimensional space with the upper left corner of the target image as the coordinate origin; the angle is calculated by using the two-dimensional image vector method; The human posture analysis algorithm includes distance determination and angle determination, and if either the distance determination or the angle determination is met, it is determined that there is a phone call in the target image.

3. The method for detecting phone calls based on posture information reasoning according to claim 2, characterized in that: The human body posture key points include nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip and right hip.

4. The method for detecting phone calls based on posture information reasoning according to claim 3, characterized in that: The distance determination includes: Calculate the distance from the center point of the phone to the left wrist and the distance to the right wrist, compare the two distances, and determine that the smaller distance value indicates that the phone is in the corresponding hand position; After determining that the Y-axis coordinate value of the wrist key point corresponding to the hand position is less than the Y-axis coordinate value of the elbow, it is determined that the hand holding the mobile phone has been lifted; Calculate the distance from the center point of the mobile phone to the wrist at the corresponding hand position and the distance from the center point of the mobile phone to the corresponding shoulder at the corresponding hand position. When the distance to the wrist is less than the distance to the corresponding shoulder, determine that the first distance determination is met. Calculate the distance from the center point of the mobile phone to the nose and the distance from the center point of the mobile phone to the corresponding elbow of the corresponding hand position, and when the distance to the nose is less than the distance to the corresponding elbow, determine that the second distance determination is satisfied; Calculate the distance from the corresponding shoulder of the corresponding hand position to the corresponding wrist of the corresponding hand position, and the distance from the corresponding shoulder of the corresponding hand position to the corresponding elbow of the corresponding hand position. When the distance to the corresponding wrist is less than the distance to the corresponding elbow, it is determined that the third distance judgment is met.

5. The method for detecting phone calls based on posture information reasoning according to claim 3, characterized in that: The angle determination includes: Calculate the first angle of the wrist position, the elbow position, and the shoulder position corresponding to the hand position according to the vector method, and determine that the first angle is less than 90°, then determine that the first angle determination is satisfied; Calculate the second angle between the center point position of the mobile phone, the shoulder position corresponding to the hand position, and the hip position corresponding to the hand position, and determine that the second angle is greater than 90° and less than 180°, then determine that the second angle judgment is met.

6. The method for detecting phone calls based on posture information reasoning according to claim 1, characterized in that: Convert the video stream into image frames to obtain a number of pictures to be tested, input the pictures to be tested into a posture information detection model, and store the detection results in a storage queue; Performing initial judgment of continuous frames using a continuous frame analysis algorithm, wherein the initial judgment includes making a judgment when the storage queue stores 10 frames of images, and if more than 5 frames of images meet the judgment of the posture information detection model, obtaining a first picture with the position coordinates of the key points of the human posture and satisfying the judgment condition and proceeding to the next judgment; Otherwise the queue clears all images.

7. The method for detecting phone calls based on posture information reasoning according to claim 6, characterized in that: Inputting the plurality of the first images into the phone call detection sub-model and the mobile phone auxiliary detection sub-model respectively, so that the phone call detection sub-model and the mobile phone auxiliary detection sub-model output the plurality of the phone call detection results; The storage queue receives a number of the call detection results; The continuous frame analysis algorithm is used to perform continuous frame judgment, and the continuous frame judgment includes making a judgment when the storage queue is full of the phone call detection results corresponding to 30 frames of images. If more than 20 frames of images meet the human posture analysis judgment, the result of successfully detecting the phone call action is output; otherwise, the storage queue clears all data and waits to store the data incoming in the next second.

8. The method for detecting phone calls based on posture information reasoning according to claim 1, characterized in that: The data set for training the phone call detection neural network algorithm includes a training human posture data set and a phone call detection data set after data enhancement processing; the data enhancement includes at least image flipping, image rotation, image random cropping, image illumination transformation and image color transformation.

9. The method for detecting phone calls based on posture information reasoning according to claim 8, characterized in that: The training human body posture data set is obtained in the following manner: Acquire a public human posture dataset as a first posture dataset; In the target area, video images containing pedestrians are obtained through surveillance videos, and human posture information of pedestrians in the video is annotated to form a second posture data set; Based on the first posture data set and the second posture data set, images of pedestrians are individually cropped and human posture information is annotated to form a third posture data set; Combining the first posture data set, the second posture data set and the third posture data set to obtain the training human posture data set; The phone call detection dataset is obtained in the following way: Obtain images of people making phone calls from a public dataset, use image annotation software to annotate the collected images with rectangular frames, annotate the mobile phone and the hand position overlapping the mobile phone, and label them as "making phone calls" to obtain the first phone call dataset; In the target area, a video containing a person making a phone call is obtained through surveillance video, the video containing the person making a phone call is intercepted into a video frame image, and the video frame image is annotated with a rectangular frame using image annotation software, the annotated content is a mobile phone and a hand position that coincides with the mobile phone, and the label name is "making a phone call" to obtain a second phone call data set; Obtain mobile phone pictures in public data sets and on the Internet; use image annotation software to annotate the mobile phone pictures with rectangular frames, with the annotation content being the location of the mobile phone and the label name being mobile phone to obtain a third phone call data set; The first phone call data set and the second phone call data set are used to train the phone call detection sub-model, and the third phone call data set is used to train the auxiliary mobile phone detection sub-model.

10. A phone call detection device based on posture information reasoning, characterized in that: The device is used to execute the method for detecting a phone call based on posture information reasoning according to any one of claims 1 to 9, the device comprising: A video stream acquisition unit, used to acquire a video stream of a monitoring area, and convert the video stream into an image frame to obtain a picture to be tested; A phone call detection result output unit, used for inputting the image to be tested into a phone call detection neural network algorithm based on posture information reasoning, so that the phone call detection neural network algorithm outputs a phone call detection result; The calling detection neural network algorithm includes a posture information detection model, a calling detection model, a human posture analysis algorithm and a continuous frame analysis algorithm; the calling detection model includes a calling detection sub-model and a mobile phone auxiliary detection sub-model; The posture information detection model is used to process the image to be tested to obtain a first image with position coordinates of key points of human posture; The calling detection sub-model and the mobile phone auxiliary detection sub-model are used to receive the first picture and detect the mobile phone in the first picture, so as to determine that the first picture is a target picture after detecting that the first picture contains a mobile phone; The human posture analysis algorithm is used to perform human posture analysis on the relative position distances and angles of various key points of the human body in the target image in combination with the position coordinates of the key points of the human posture, so as to output the phone call detection result according to the analysis result.

Images