User call and answer behavior detection method based on deep learning multi-model fusion

Abstract

The present invention discloses a method for detecting user's answering and making phone calls based on deep learning multi-model fusion, obtaining the probability of a person raising his hand and the probability of a phone being located at the head of a human body; inputting the results of a person raising his hand discriminator and a phone discriminator into a person answering and making phone calls discriminator, and finally determining whether a person in a target video stream answers or makes phone calls based on the probability of a person raising his hand and the probability of a phone being located at the head of a human body. The present invention integrates the prediction results of a phone detection model and a human key point detection model, and directly determines that a person answers or makes phone calls when the prediction probability of the calculation formula of any model is greater than a set value, thereby avoiding a single model missing detection that reduces the recall rate of phone call detection, and even when the prediction probabilities of the calculation formulas of the two models are low, the detection results of the two models can be fused, thereby improving the recall rate of the entire algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of oil and gas field operations, and in particular to a method for detecting user's mobile phone receiving and calling behaviors based on deep learning multi-model fusion. Background Art

[0002] Making or receiving phone calls is a very dangerous behavior at the oil and gas field operation site, which seriously threatens the safety of oil and gas field production operations. With the rapid and efficient development of oil and gas fields, there are more and more production and construction well stations in oil and gas fields, and the monitoring area and scale are expanding. At the same time, higher requirements are put forward for the safety control of production and construction sites, and the need to grasp the unsafe behaviors and status of making or receiving phone calls in key areas such as production sites, construction sites, pipeline lines, etc. in real time is becoming more and more urgent.

[0003] One of the traditional methods is to realize real-time viewing and video playback of monitoring point videos through manual remote viewing, which has the difficulties of low monitoring efficiency, heavy supervision tasks, and great management difficulty.

[0004] Another traditional method is to make and receive calls through single model and single frame detection. This method relies on the detection results of a single model and a single frame, which may result in false alarms and missed alarms in complex work sites and has low reliability.

[0005] Therefore, it is necessary to develop a user's mobile phone receiving and calling behavior detection method based on deep learning multi-model fusion to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to design a method for detecting user's mobile phone receiving and making calls based on deep learning multi-model fusion in order to solve the above problems.

[0007] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0008] The method for detecting user's mobile phone receiving and calling behaviors based on deep learning multi-model fusion includes the following steps:

[0009] S1. Build a key point detection model for people, mobile phones and human bodies based on historical images;

[0010] S2, input the image into the person detection model for detection;

[0011] S3, when a person is detected in the image, the maximum bounding rectangle of all objects is calculated, and the maximum bounding rectangle area is intercepted in the original image to generate a person area image;

[0012] S4, inputting the personnel area image into the mobile phone detection model and the human key point detection model to determine whether there is a mobile phone in the personnel area image and to detect the key points of the personnel in the area; proceeding to steps S5 and S6 respectively;

[0013] S5, input the detection results of the mobile phone detection model and the human body key point detection model into the person raising hand action discriminator to obtain the probability of the person raising hand action; proceed to step S7;

[0014] S6, input the detection results of the mobile phone detection model and the human body key point detection model into the mobile phone discriminator to obtain the probability that the mobile phone is located on the human head; proceed to step S7;

[0015] S7. Input the results of the person's hand-raising action discriminator and the mobile phone discriminator into the person's answering and calling discriminator, and finally determine whether there is anyone answering or making a phone call in the target video stream based on the probability of the person's hand-raising action and the probability of the mobile phone being located on the human head.

[0016] The beneficial effects of the present invention are:

[0017] The prediction results of the mobile phone detection model and the human key point detection model are combined. When the prediction probability of the calculation formula of any model is greater than the set value, it is directly determined that a person is answering or making a phone call. This can avoid the reduction of the detection recall rate of answering or making a phone call due to missed detection of a single model. Even when the prediction probability of the calculation formula of the two models is low, the detection results of the two models can be fused to improve the recall rate of the entire algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the method of this application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0022] In the description of the present invention, it should be understood that the terms "upper", "lower", "inside", "outside", "left", "right", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use, or are directions or positional relationships commonly understood by those skilled in the art. These directions or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0023] Furthermore, the terms “first”, “second”, etc. are merely used for distinguishing descriptions and should not be understood as indicating or implying relative importance.

[0024] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0025] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0026] like Figure 1 As shown, the method for detecting user's mobile phone receiving and calling behaviors based on deep learning multi-model fusion includes the following steps:

[0027] S1. Build a key point detection model for people, mobile phones and human bodies based on historical images;

[0028] S2, input the image into the person detection model for detection;

[0029] S3. When a person is detected in the image, the maximum circumscribed rectangle of all targets is calculated, and the maximum circumscribed rectangle area is intercepted in the original image to generate a person area image; in the person area image, the proportion of the target is greatly improved compared with the original image, which can improve the detection accuracy of the mobile phone detection model and the human key point detection model.

[0030] S4, inputting the personnel area image into the mobile phone detection model and the human key point detection model to determine whether there is a mobile phone in the personnel area image and to detect the key points of the personnel in the area; proceeding to steps S5 and S6 respectively;

[0031] S5, input the detection results of the mobile phone detection model and the human body key point detection model into the person raising hand action discriminator to obtain the probability of the person raising hand action; proceed to step S7;

[0032] S6, input the detection results of the mobile phone detection model and the human body key point detection model into the mobile phone discriminator to obtain the probability that the mobile phone is located on the human head; proceed to step S7;

[0033] S7. Input the results of the person's hand-raising action discriminator and the mobile phone discriminator into the person's answering and calling discriminator, and finally determine whether there is anyone answering or making a phone call in the target video stream based on the probability of the person's hand-raising action and the probability of the mobile phone being located on the human head.

[0034] In some embodiments, the historical images include images of various scenes at different time periods at the oil and gas operation site. Each scene is a wellhead, a device area, a process area, a valve room, a purification plant, etc., and each time period is morning, noon, evening, etc. Only in this way can the established model ensure the accuracy of recognition in different scenes.

[0035] In some embodiments, step S5 includes constructing a person raising hand action discriminator, which is used to determine whether the person has raised hand action, wherein the calculation formula of the person raising hand action probability is as follows:

[0036]

[0037]

[0038] Among them, PI keypoint Indicates the probability of a person raising their hand in n consecutive frames, where n is usually three times the video frame rate; i Indicates whether the wrist keypoint coordinates are higher than the neck keypoint in the height direction in the i-th frame. If yes, it is 1, otherwise it is 0; and are the height coordinates of the neck and wrist key points predicted by the human key point detection model in the i-th frame; and is the probability of the neck and wrist key points predicted by the human key point detection model in the i-th frame. This calculation method calculates the probability of a person raising their hand in n consecutive frames, which can avoid the decrease in prediction accuracy caused by false detection and missed detection of a single frame image, and also combines the probability of the neck and wrist key points predicted by the human key point detection model, so that the final predicted probability of raising the hand is more reliable.

[0039] In some embodiments, step S6 includes constructing a mobile phone discriminator, which is used to determine whether the mobile phone is located on the human head, wherein the calculation formula for the probability of the mobile phone being located on the human head is as follows:

[0040]

[0041]

[0042] Among them, Pphone It represents the probability that the mobile phone is located on the human head in the continuous n frames, where n is generally three times the video frame rate; I i Indicates whether the mobile phone coordinates are higher than the neck key point in the height direction in the i-th frame. If yes, it is 1, otherwise it is 0. In the i-th frame, the height coordinates of the neck key points predicted by the human key point detection model are In the i-th frame, the height coordinates of the mobile phone predicted by the mobile phone detection model; Indicates the probability that the mobile phone detection model detects a mobile phone in the i-th frame, and is the probability of the neck and wrist key points predicted by the human key point detection model in the i-th frame. This calculation method calculates the probability of the phone being located on the human head in n consecutive frames, which can avoid the decrease in prediction accuracy caused by false detection and missed detection of single-frame images, and also combines the probabilities of the neck and wrist key points predicted by the human key point detection model, so that the final predicted probability of the phone being located on the human head is more reliable.

[0043] In some embodiments, step S7 includes constructing a person answering or calling discriminator, which is used to determine whether there is a person answering or calling a phone in the target video stream, and the method specifically includes:

[0044] The construction formula is as follows:

[0045]

[0046] Among them, P represents the probability of a person answering or making a phone call in n consecutive frames;

[0047] When detecting P keypoint and P phone When both are less than 0.85, P keypoint A weight of 0.4 is given to P phone The weight of 0.6 is added together to obtain the probability P of a person answering or making a phone call. If P ≥ 0.6, it is determined that a person is detected to be answering or making a phone call at the work site.

[0048] When the probability of a person raising their hand or the probability of the mobile phone being above the person's head for a period of time is greater than 0.85, it is determined that there are people on the work site answering or making phone calls.

[0049] This calculation method combines the prediction results of the mobile phone detection model and the human body key point detection model, which can avoid the decrease in the detection recall rate of mobile phone calls and incoming calls caused by missed detection of a single model. Even when the prediction probability of the calculation formulas of the two models is low, the detection results of the two models can be fused to improve the recall rate of the entire algorithm.

[0050] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. All technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.

Claims

1. A method for detecting user's mobile phone calling and receiving behaviors based on deep learning multi-model fusion, characterized in that: Includes steps: S1. Build a key point detection model for people, mobile phones and human bodies based on historical images; S2, input the image into the person detection model for detection; S3, when a person is detected in the image, the maximum bounding rectangle of all objects is calculated, and the maximum bounding rectangle area is intercepted in the original image to generate a person area image; S4, inputting the personnel area image into the mobile phone detection model and the human key point detection model to determine whether there is a mobile phone in the personnel area image and to detect the key points of the personnel in the area; Go to steps S5 and S6 respectively; S5. Input the detection results of the mobile phone detection model and the human key point detection model into the person raising hand action discriminator to obtain the probability of the person raising hand action; Go to step S7; S6, input the detection results of the mobile phone detection model and the human key point detection model into the mobile phone discriminator to obtain the probability that the mobile phone is located on the human head; proceed to step S7; the calculation formula of the probability that the mobile phone is located on the human head in step S6 is as follows: , , Among them, P phone Indicates the probability that the mobile phone is located on the human head in n consecutive frames, where n is 3 times the video frame rate; I i Indicates whether the mobile phone coordinates are higher than the neck key point in the height direction in the i-th frame. If yes, it is 1, otherwise it is 0. In the i-th frame, the height coordinates of the neck key points predicted by the human key point detection model are In the i-th frame, the height coordinates of the mobile phone predicted by the mobile phone detection model; Indicates the probability that the mobile phone detection model detects a mobile phone in the i-th frame, and is the probability of the neck and wrist key points predicted by the human key point detection model in the i-th frame; S7, input the results of the person's hand-raising action discriminator and the mobile phone discriminator into the person's answering and calling discriminator, and finally determine whether there is anyone answering or calling the phone in the target video stream according to the probability of the person's hand-raising action and the probability of the mobile phone being located on the human head; the method for determining whether there is anyone answering or calling the phone in the target video stream in step S7 specifically includes: The construction formula is as follows: , Among them, P represents the probability of a person answering or making a phone call in n consecutive frames; When detecting P keypoint and P phone When both are less than 0.85, P keypoint A weight of 0.4 is given to P phone The weight of 0.6 is added together to obtain the probability P of a person answering or making a phone call. If P ≥ 0.6, it is determined that a person is detected to be answering or making a phone call at the work site. When the probability of a person raising their hand or the probability of the mobile phone being above the person's head for a period of time is greater than 0.85, it is determined that there are people on the work site answering or making phone calls.

2. According to the method for detecting user's mobile phone receiving and calling behaviors based on deep learning multi-model fusion according to claim 1, it is characterized in that: Historical pictures include pictures of various scenes at different time periods at the oil and gas operation site.

3. The method for detecting user's mobile phone calling and receiving behaviors based on deep learning multi-model fusion according to claim 1 is characterized in that: The calculation formula for the probability of a person raising their hand in step S5 is as follows: , , Among them, P keypoint Indicates the probability of a person raising their hand in n consecutive frames, where n is 3 times the video frame rate; Indicates whether the wrist keypoint coordinates are higher than the neck keypoint in the height direction in the i-th frame. If yes, it is 1, otherwise it is 0; and are the height coordinates of the neck and wrist key points predicted by the human key point detection model in the i-th frame; and is the probability of the neck and wrist key points predicted by the human key point detection model in the i-th frame.

Images