Behavior detection method, device and equipment for remote connection operator and medium

Abstract

The embodiment of the invention discloses a behavior detection method and device for a remote connection operator, equipment and a medium. Comprising the steps of obtaining a plurality of human body images of an operator in a set time period; performing face detection on the plurality of human body images to obtain a face detection result corresponding to each human body image; performing first target detection on the plurality of human body images to obtain a first target detection result corresponding to each human body image; performing second target detection on the plurality of human body images to obtain a second target detection result corresponding to each human body image; and determining a target state and a target behavior of the operator based on the face detection result, the first target detection result and the second target detection result. According to the scheme, the face detection result, the first target detection result and the second target detection result can be accurately obtained, and the target state and the target behavior of the operator can be accurately determined, so that the accuracy of behavior detection of the remote pickup operator is improved, and the safety of remote pickup is guaranteed.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of behavior detection technology, and in particular to a behavior detection method, device, equipment and medium for a remote pick-up operator. Background Art

[0002] As an important tool for passengers to board or disembark, boarding bridges have long been essential equipment at major airports. With the development of technology, in order to improve work efficiency and save human resources, remote pick-up operators are mainly used to remotely control boarding bridges to run to the aircraft, thereby connecting the terminal and the aircraft to enable passengers to board or disembark. During the remote control process, if the remote pick-up operator has abnormal behavior (such as making phone calls, etc.), it will affect the safety of remote pick-up. Therefore, it is particularly important to detect the behavior of the remote pick-up operator. Summary of the invention

[0003] The embodiments of the present invention provide a method, device, equipment and medium for detecting the behavior of a remote pick-up operator, which can improve the accuracy of the behavior detection of the remote pick-up operator, thereby ensuring the safety of the remote pick-up.

[0004] In a first aspect, an embodiment of the present invention provides a behavior detection method for a remote pick-up operator, comprising:

[0005] Acquire multiple human body images of the operator within a set time period;

[0006] Performing facial detection on the multiple human images to obtain a facial detection result corresponding to each human image; wherein the facial detection result includes the facial region of the operator and position information of the facial region in the human image;

[0007] Performing first target detection on the multiple human images to obtain a first target detection result corresponding to each human image; wherein the first target detection result includes whether the first target exists in the human image, and if the first target exists, the first target detection result also includes position information of the first target in the human image;

[0008] Performing second target detection on the multiple human images to obtain a second target detection result corresponding to each human image; wherein the second target detection result includes whether the second target exists in the human image, and if the second target exists, the second target detection result also includes position information of the second target in the human image;

[0009] The target state and target behavior of the operator are determined based on the facial detection result, the first target detection result, and the second target detection result corresponding to each human body image.

[0010] In a second aspect, an embodiment of the present invention further provides a behavior detection device for a remote pick-up operator, the device comprising:

[0011] An image acquisition module, used to acquire multiple human body images of the operator within a set time period;

[0012] A face detection module, configured to perform face detection on the plurality of human images to obtain a face detection result corresponding to each human image; wherein the face detection result includes a face region of the operator and position information of the face region in the human image;

[0013] A first target detection module, configured to detect a first target on the multiple human images, and obtain a first target detection result corresponding to each human image; wherein the first target detection result includes whether the first target exists in the human image, and if the first target exists, the first target detection result also includes position information of the first target in the human image;

[0014] A second target detection module, configured to detect a second target on the plurality of human images, and obtain a second target detection result corresponding to each human image; wherein the second target detection result includes whether the second target exists in the human image, and if the second target exists, the second target detection result also includes position information of the second target in the human image;

[0015] A target behavior determination module is used to determine the target state and target behavior of the operator based on the facial detection result corresponding to each human body image, the first target detection result, and the second target detection result.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the behavior detection method of the remote pick-up operator described in the embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the behavior detection method for a remote pick-up operator described in an embodiment of the present invention when executed.

[0021] The embodiment of the present invention discloses a behavior detection method, device, equipment and medium for a remote pick-up operator, comprising: acquiring multiple human body images of the operator within a set time period; performing facial detection on the multiple human body images to obtain a facial detection result corresponding to each human body image; wherein the facial detection result includes the facial area of ​​the operator and the position information of the facial area in the human body image; performing first target detection on the multiple human body images to obtain a first target detection result corresponding to each human body image; wherein the first target detection result includes whether the first target exists in the human body image, and if the first target exists, the first target detection result also includes the position information of the first target in the human body image; performing second target detection on the multiple human body images to obtain a second target detection result corresponding to each human body image; wherein the second target detection result includes whether the second target exists in the human body image, and if the second target exists, the second target detection result also includes the position information of the second target in the human body image; determining the target state and target behavior of the operator based on the facial detection result corresponding to each human body image, the first target detection result and the second target detection result. The above technical solution specifically considers detecting the behavior of the remote pick-up operator, thereby obtaining multiple human images of the operator within a set time period; at the same time, it also considers performing facial detection, first target detection, and second target detection on multiple human images, and can accurately obtain the facial detection results, first target detection results, and second target detection results corresponding to each human image; finally, the operator's target state and target behavior can be accurately determined based on the facial detection results, first target detection results, and second target detection results. Compared with the prior art, the technical solution provided in this embodiment can accurately obtain the facial detection results, first target detection results, and second target detection results corresponding to each human image, and can accurately determine the operator's target state and target behavior, thereby improving the accuracy of the behavior detection of the remote pick-up operator and ensuring the safety of the remote pick-up. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of a behavior detection method for a remote pick-up operator in Embodiment 1 of the present invention;

[0023] Figure 2 is a flow chart of a behavior detection method for a remote pick-up operator in Embodiment 2 of the present invention;

[0024] Figure 3 It is a structural schematic diagram of a behavior detection device for a remote pick-up operator in Embodiment 3 of the present invention;

[0025] Figure 4 It is a structural schematic diagram of an electronic device in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0027] Embodiment 1

[0028] Figure 1 This is a flow chart of a behavior detection method for a remote pick-up operator provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of detecting the behavior of a remote pick-up operator. The method can be executed by a behavior detection device for a remote pick-up operator. The device can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc. Specifically, it includes the following steps:

[0029] S110, acquiring a plurality of body images of the operator within a set time period.

[0030] In this embodiment, the application scenario of the present invention is a scenario where a remote pick-up operator remotely controls an aerobridge. Specifically, the behavior of the remote pick-up operator is detected during the process from the beginning of remote control of the aerobridge to the end of control. For example, if the time period during which the remote pick-up operator remotely controls the aerobridge is t1 to tn, the behavior of the remote pick-up operator is detected during the period from t1 to tn.

[0031] In accordance with the above description, the set time period can be understood as the time period during which the remote pick-up operator remotely controls the boarding bridge. The operator is the remote pick-up operator, and the human body image can be understood as an image containing the operator's body parts (such as the head, limbs, etc.), and the image also contains the operator's behavior.

[0032] Optionally, an image acquisition device may be used to acquire multiple human body images of the operator within a set time period, and the human body images may be stored in a target mapping table, and then multiple human body images of the operator within the set time period may be acquired from the target mapping table. The image acquisition device may be a camera. In the target mapping table, the operator, the set time period, and the human body images are in a one-to-one correspondence.

[0033] S120: Perform facial detection on multiple human images to obtain a facial detection result corresponding to each human image.

[0034] In this embodiment, the face detection result includes the operator's facial region and the position information of the facial region in the human image. The facial region can be understood as a rectangular frame region containing the operator's face, and the position information of the facial region in the human image can be understood as the position coordinates of the four corners of the rectangular frame region in the human image.

[0035] Optionally, performing facial detection on multiple human images to obtain facial detection results corresponding to each human image may be performed by inputting multiple human images into a pre-trained facial detection model to obtain facial detection results corresponding to each human image.

[0036] Furthermore, the training method of the face detection model can be: obtaining a sample set of human images and the real face detection results corresponding to the sample set; inputting the sample set into the face detection model to obtain predicted face detection results; determining the target loss function based on the predicted face detection results and the real face detection results, and reversely adjusting the parameters in the face detection model based on the target loss function to realize the training of the face detection model.

[0037] Continuing with the above description, the real face detection result includes the real face area and the position information of the real face area in the human body image; the predicted face detection result includes the predicted face area and the position information of the predicted face area in the human body image. Further, the method of determining the target loss function based on the predicted face detection result and the real face detection result can be: determining a first loss function based on the predicted face area and the real face area; determining a second loss function based on the position information of the predicted face area in the human body image and the position information of the real face area in the human body image; and performing weighted summation of the first loss function and the second loss function to obtain the target loss function. Optionally, the loss function can be a mean square error loss function.

[0038] S130: Perform first target detection on multiple human body images to obtain a first target detection result corresponding to each human body image.

[0039] In this embodiment, the first target can be understood as a target in the human image, such as a cigarette, and further, the first target detection result can be a cigarette detection result. The first target detection result includes whether the first target exists in the human image, and if the first target exists, the first target detection result also includes the position information of the first target in the human image. The position information of the first target in the human image can be understood as the position coordinates of the four corners of the rectangular frame area containing the first target in the human image.

[0040] Optionally, the first target detection is performed on multiple human images to obtain the first target detection result corresponding to each human image by inputting the multiple human images into a pre-trained first target detection model to obtain the first target detection result corresponding to each human image.

[0041] Furthermore, the training method of the first target detection model may be: obtaining a sample set of human body images and a real first target detection result corresponding to the sample set; inputting the sample set into the first target detection model to obtain a predicted first target detection result; determining a loss function based on the predicted first target detection result and the real first target detection result, and reversely adjusting the parameters in the first target detection model based on the loss function to implement the training of the first target detection model. Optionally, the loss function may be a cross entropy loss function. The first target detection model may be a cigarette detection model.

[0042] S140: Perform second target detection on multiple human body images to obtain a second target detection result corresponding to each human body image.

[0043] In this embodiment, the second target can be understood as a target in the human image, such as a mobile phone. Further, the second target detection result can be a mobile phone detection result. The second target detection result includes whether the second target exists in the human image, and if the second target exists, the second target detection result also includes the position information of the second target in the human image. Among them, the position information of the second target in the human image can be understood as the position coordinates of the four corners of the rectangular frame area containing the second target in the human image.

[0044] Optionally, the second target detection is performed on multiple human images to obtain the second target detection result corresponding to each human image by inputting multiple human images into a pre-trained second target detection model to obtain the second target detection result corresponding to each human image.

[0045] Furthermore, the training method of the second target detection model can be: obtaining a sample set of human body images and the real second target detection results corresponding to the sample set; inputting the sample set into the second target detection model to obtain the predicted second target detection results; determining the loss function based on the predicted second target detection results and the real second target detection results, and reversely adjusting the parameters in the second target detection model based on the loss function to realize the training of the second target detection model. Optionally, the loss function can be a cross entropy loss function. The second target detection model can be a mobile phone detection model.

[0046] S150, determining the target state and target behavior of the operator based on the facial detection result, the first target detection result, and the second target detection result corresponding to each human body image.

[0047] In this embodiment, the target state can be understood as the state of the operator, such as fatigue, etc., and the target behavior can be understood as the behavior of the operator, such as making a phone call, etc.

[0048] Furthermore, the method of determining the operator's target state and target behavior based on the facial detection result, the first target detection result and the second target detection result corresponding to each human body image can be: determining the operator's target state according to the facial detection result; determining the operator's behavior towards the first target according to the first target detection result; determining the operator's behavior towards the second target according to the second target detection result.

[0049] The technical solution of this embodiment is to obtain multiple human images of the operator within a set time period; perform facial detection on the multiple human images to obtain facial detection results corresponding to each human image; perform first target detection on the multiple human images to obtain first target detection results corresponding to each human image; perform second target detection on the multiple human images to obtain second target detection results corresponding to each human image; determine the target state and target behavior of the operator based on the facial detection results, first target detection results and second target detection results corresponding to each human image. The above technical solution specifically considers the detection of the behavior of the remote pick-up operator, thereby obtaining multiple human images of the operator within a set time period; at the same time, it also considers facial detection, first target detection and second target detection on multiple human images, and can accurately obtain facial detection results, first target detection results and second target detection results corresponding to each human image; finally, the target state and target behavior of the operator can be accurately determined based on the facial detection results, first target detection results and second target detection results. The technical solution provided in this embodiment can accurately obtain the facial detection results, the first target detection results and the second target detection results corresponding to each human body image, and can accurately determine the operator's target state and target behavior, thereby improving the accuracy of behavior detection of remote pick-up operators and ensuring the safety of remote pick-up.

[0050] Embodiment 2

[0051] Figure 2 This is a flow chart of a behavior detection method for a remote pick-up operator provided in Embodiment 2 of the present invention. Based on the above embodiment, the method includes the following steps:

[0052] S210: Acquire multiple human body images of the operator within a set time period.

[0053] S220: Perform facial detection on multiple human body images to obtain facial detection results corresponding to each human body image.

[0054] S230: Perform first target detection on multiple human body images to obtain a first target detection result corresponding to each human body image.

[0055] S240: Perform second target detection on multiple human body images to obtain a second target detection result corresponding to each human body image.

[0056] S250: Determine a first target state and a second target state of the operator based on the face detection result.

[0057] In this embodiment, the first target state may include fatigue or non-fatigue; the second target state may include concentration or non-concentration.

[0058] In this embodiment, one implementation of determining the first target state and the second target state of the operator based on the face detection result can be described as the following steps:

[0059] a1) determining position information corresponding to a plurality of key points in the facial region;

[0060] In this embodiment, the multiple key points may include eye key points, mouth key points, nose tip key points and face key points. The position information may be understood as the position coordinates of the key points.

[0061] In this embodiment, the method for determining the position information corresponding to the multiple key points in the facial region may be: using a key point detection algorithm to extract the multiple key points in the facial region and determine the position information of the multiple key points in the facial region. The key point detection algorithm may be a facial-landmarks algorithm.

[0062] b1) determining a first target state of the operator based on the position information corresponding to the eye key point and the position information corresponding to the mouth key point;

[0063] Optionally, the eye key points may include the upper eye key points, the lower eye key points, the left eye key points and the right eye key points; the mouth key points may include the upper mouth key points, the lower mouth key points, the left mouth key points and the right mouth key points. Exemplarily, the upper eye key points, the lower eye key points, the left eye key points and the right eye key points are respectively recorded as a1, a2, a3, a4. The upper mouth key points, the lower mouth key points, the left mouth key points and the right mouth key points are respectively recorded as b1, b2, b3, b4.

[0064] In this embodiment, the method of determining the first target state of the operator based on the position information corresponding to the eye key point and the position information corresponding to the mouth key point can be described as the following steps:

[0065] (b11) determining the width and height of the eye based on the position information of the key points above the eye, the key points below the eye, the key points on the left side of the eye, and the key points on the right side of the eye;

[0066] For example, assuming that the position information of the key point a1 on the upper side of the eye, the key point a2 on the lower side of the eye, the key point a3 on the left side of the eye, and the key point a4 on the right side of the eye are respectively: (x1, y1), (x2, y2), (x3, y3), (x4, y4), then the distance between the key point a3 on the left side of the eye and the key point a4 on the right side of the eye is determined according to the position information (x3, y3) of the key point a3 on the left side of the eye and the position information (x4, y4) of the key point a4 on the right side of the eye, and the distance between a3 and a4 is taken as the width of the eye. Among them, the width W1 of the eye is:

[0067]

[0068] Continuing with the above description, the distance between the upper key point a1 and the lower key point a2 is determined based on the position information (x1, y1) of the upper key point a1 and the position information (x2, y2) of the lower key point a2, and the distance between a1 and a2 is taken as the height of the eye. The height H1 of the eye is:

[0069]

[0070] (b12) determining the width and height of the mouth based on the position information of the key points on the upper side of the mouth, the key points on the lower side of the mouth, the key points on the left side of the mouth, and the key points on the right side of the mouth;

[0071] For example, assuming that the position information of the key point b1 on the upper side of the mouth, the key point b2 on the lower side of the mouth, the key point b3 on the left side of the mouth, and the key point b4 on the right side of the mouth are respectively: (z1, k1), (z2, k2), (z3, k3), (z4, k4), then the distance between the key point b3 on the left side of the mouth and the key point b4 on the right side of the mouth is determined according to the position information (z3, k3) of the key point b3 on the left side of the mouth and the position information (z4, k4) of the key point b4 on the right side of the mouth, and the distance between b3 and b4 is used as the width of the mouth. Among them, the width W2 of the mouth is:

[0072]

[0073] Continuing with the above description, the distance between the upper key point b1 and the lower key point b2 is determined based on the position information (z1, k1) of the upper key point b1 and the position information (z2, k2) of the lower key point b2, and the distance between b1 and b2 is taken as the height of the mouth. The height H2 of the mouth is:

[0074]

[0075] (b13) determining the state of the eye based on the ratio between the width and height of the eye;

[0076] The state of the eyes includes an open state or a closed state.

[0077] In this embodiment, the method of determining the state of the eye based on the ratio between the width and height of the eye can be: obtaining a first set threshold; if the ratio between the width and height of the eye is greater than or equal to the first set threshold, the state of the eye is a closed state; if the ratio between the width and height of the eye is less than the first set threshold, the state of the eye is an open state. The first set threshold is a threshold set based on experience. Exemplarily, the ratio between the width and height of the eye is W1 / H1.

[0078] (b14) determining a state of the mouth based on a ratio between a width and a height of the mouth;

[0079] The state of the mouth includes an open state or a closed state;

[0080] In this embodiment, the method of determining the state of the mouth based on the ratio between the width and height of the mouth can be: obtaining a second set threshold; if the ratio between the width and height of the mouth is greater than or equal to the second set threshold, the state of the mouth is a closed state; if the ratio between the width and height of the mouth is less than the second set threshold, the state of the mouth is an open state. The second set threshold is a threshold set based on experience. Exemplarily, the ratio between the width and height of the mouth is W2 / H2.

[0081] (b15) Determine a first target state of the operator based on the state of the eyes and the state of the mouth.

[0082] In this embodiment, the method of determining the operator's first target state based on the state of the eyes and the state of the mouth can be: if the state of the eyes is open and the state of the mouth is closed, the operator's first target state is not tired; if the state of the eyes is closed and / or the state of the mouth is open, the operator's first target state is tired.

[0083] c1) determining a second target state of the operator based on the position information corresponding to the nose tip key point and the position information corresponding to the facial key points.

[0084] Optionally, the facial key points include upper facial key points, lower facial key points, left facial key points, and right facial key points.

[0085] Exemplarily, the key points on the upper side of the face, the key points on the lower side of the face, the key points on the left side of the face, and the key points on the right side of the face are respectively recorded as c1, c2, c3, and c4, and the key point on the tip of the nose is recorded as d.

[0086] In this embodiment, the method of determining the second target state of the operator based on the position information corresponding to the nose tip key point and the position information corresponding to the face key point can be described as the following steps:

[0087] (c11) determining a first distance between the nose tip key point and the upper face key point and a second distance between the nose tip key point and the lower face key point based on position information of the nose tip key point, the upper face key point and the lower face key point;

[0088] For example, assuming that the position information of the nose tip key point d, the upper face key point c1, and the lower face key point c2 are (x5, y5), (x6, y6), and (x7, y7), respectively, the first distance between the nose tip key point d and the upper face key point c1 is The second distance between the nose tip key point d and the lower face key point c2 is

[0089] (c12) determining a third distance between the nose tip key point and the left side face key point and a fourth distance between the nose tip key point and the right side face key point based on the position information of the nose tip key point, the left side face key point and the right side face key point;

[0090] For example, assuming that the position information of the key point c3 on the left side of the face and the key point c4 on the right side of the face are (x8, y8) and (x9, y9) respectively, the third distance between the key point d on the nose tip and the key point c3 on the left side of the face is The fourth distance between the nose tip key point d and the right side face key point c4 is

[0091] (c13) determining a first facial state based on a ratio between the first distance and the second distance;

[0092] The first facial state includes a nodding state or a non-nodding state;

[0093] In this embodiment, the method of determining the first facial state based on the ratio between the first distance and the second distance may be: obtaining a third set threshold; if the difference between the ratio between the first distance and the second distance and the fixed value is less than or equal to the third set threshold, the first facial state is a non-nodding state; if the difference between the ratio between the first distance and the second distance and the fixed value is greater than the third set threshold, the first facial state is a nodding state. The fixed value may be 1, and the third set threshold is a threshold set based on experience.

[0094] (c14) determining a second facial state based on a ratio between the third distance and the fourth distance;

[0095] The second facial state includes a head-turning state or a head-not-turning state;

[0096] In this embodiment, the method of determining the second facial state based on the ratio between the third distance and the fourth distance may be: obtaining a fourth set threshold; if the difference between the ratio between the third distance and the fourth distance and the fixed value is less than or equal to the fourth set threshold, the second facial state is a head-not-turned state; if the difference between the ratio between the third distance and the fourth distance and the fixed value is greater than the fourth set threshold, the second facial state is a head-turned state. The fixed value may be 1, and the fourth set threshold is a threshold set based on experience.

[0097] (c15) Determining a second target state of the operator based on the first facial state and the second facial state.

[0098] In this embodiment, the method of determining the operator's second target state based on the first facial state and the second facial state can be: if the first facial state is a non-nodding state and the second facial state is a non-turning head state, the operator's second target state is concentration; if the first facial state is a nodding state and / or the second facial state is a turning head state, the operator's second target state is a lack of concentration.

[0099] S260. Determine a first target behavior of the operator based on the first target detection result.

[0100] The first target behavior includes the operator's mouth touching the first target or not touching the first target; illustratively, the operator's mouth touching the first target can be understood as the operator smoking; the operator's mouth not touching the first target can be understood as the operator not smoking.

[0101] In this embodiment, one implementation method of determining the operator's first target behavior based on the first target detection result can be described as: if the first target detection result is that the first target does not exist in the human body image, then the operator's first target behavior is that the operator's mouth does not touch the first target.

[0102] In this embodiment, another implementation of determining the first target behavior of the operator based on the first target detection result can be described as the following steps:

[0103] a2) if the first target detection result is that the first target exists in the human image, determining a first center position of the first target based on position information of the first target in the human image;

[0104] Exemplarily, the position information of the first target in the human body image is the position coordinates of the four corners of the rectangular frame area containing the first target in the human body image. For example, the position coordinates of the four corners of the rectangular frame area containing the first target are (x10, y10), (x11, y11), (x12, y12), and (x13, y13). Then the first center position is

[0105] b2) Determine the first target behavior of the operator based on the first center position and the position information of the key points above the mouth.

[0106] In this embodiment, the method of determining the first target behavior of the operator based on the position information of the first center position and the key point on the upper side of the mouth can be: determining the distance between the first center position and the key point on the upper side of the mouth based on the position information of the first center position and the key point on the upper side of the mouth; if the distance between the first center position and the key point on the upper side of the mouth is less than or equal to the fifth set threshold, the first target behavior of the operator is that the mouth touches the first target; if the distance between the first center position and the key point on the upper side of the mouth is greater than the fifth set threshold, the first target behavior of the operator is that the mouth does not touch the first target. The fifth set threshold is a threshold set based on experience.

[0107] S270: Determine a second target behavior of the operator based on at least one of the second target detection result and the position information of the facial region in the human body image.

[0108] The second target behavior includes the operator's face touching the second target or not touching the second target. For example, the operator's face touching the second target can be understood as the operator is making a phone call; the operator's face not touching the second target can be understood as the operator is not making a phone call.

[0109] In this embodiment, one implementation method of determining the operator's second target behavior based on at least one of the second target detection result and the position information of the facial area in the human image can be described as: if the second target detection result is that the second target does not exist in the human image, then the operator's second target behavior is that the operator's face does not touch the second target.

[0110] In this embodiment, another implementation of determining the second target behavior of the operator based on at least one of the second target detection result and the position information of the facial region in the human body image can be described as the following steps:

[0111] a3) if the second target detection result is that the second target exists in the human body image, determining a second center position of the second target based on the position information of the second target in the human body image;

[0112] Exemplarily, the position information of the second target in the human body image is the position coordinates of the four corners of the rectangular frame area containing the second target in the human body image. For example, the position coordinates of the four corners of the rectangular frame area containing the second target are: (x14, y14), (x15, y15), (x16, y16), (x17, y17), then the second center position is:

[0113] b3) determining a second target behavior of the operator based on the second center position and position information of the facial area in the human body image.

[0114] Exemplarily, the position information of the facial region in the human body image is the position coordinates of the four corners of the rectangular frame region containing the operator's face in the human body image, assuming that the position coordinates are: upper left corner (x18, y18), upper right corner (x19, y19), lower left corner (x20, y20), lower right corner (x21, y21), and the second center position Denoted as (r1, r2).

[0115] Continuing with the above description, the method for determining the operator's second target behavior based on the second center position and the position information of the facial area in the human body image can be: if the second target is on the right side of the facial area, then if the difference between r1 and x19 is less than the sixth set threshold and r2 is within the first set range [y21, y19], then the second target behavior is that the operator's face contacts the second target, otherwise, the second target behavior is that the operator's face does not contact the second target; if the second target is on the left side of the facial area, then if the difference between r1 and x18 is less than the sixth set threshold and r2 is within the second set range [y20, y18], then the second target behavior is that the operator's face contacts the second target, otherwise, the second target behavior is that the operator's face does not contact the second target.

[0116] Further, after determining the second target behavior of the operator based on at least one of the second target detection result and the position information of the facial area in the human body image, it also includes: if the first target state of the operator is fatigue, the second target state is inattention, the first target behavior is the operator's mouth touching the first target and / or the second target behavior is the operator's face touching the second target, and the duration is less than or equal to the set value, then the operator is warned; if the first target state of the operator is fatigue, the second target state is inattention, the first target behavior is the operator's mouth touching the first target and / or the second target behavior is the operator's face touching the second target, and the duration is greater than the set value, then the operator's control of the boarding bridge is stopped urgently. Exemplarily, the set value can be 5 seconds.

[0117] The technical solution of this embodiment is to obtain multiple body images of the operator within a set time period; perform facial detection on the multiple body images to obtain a facial detection result corresponding to each body image; perform first target detection on the multiple body images to obtain a first target detection result corresponding to each body image; perform second target detection on the multiple body images to obtain a second target detection result corresponding to each body image; determine the first target state and the second target state of the operator based on the facial detection results; determine the first target behavior of the operator based on the first target detection result; determine the second target behavior of the operator based on at least one of the second target detection result and the position information of the facial area in the body image. The above technical solution specifically considers detecting the behavior of the remote pick-up operator, thereby obtaining multiple human images of the operator within a set time period; at the same time, it also considers performing facial detection, first target detection, and second target detection on multiple human images, and can accurately obtain facial detection results, first target detection results, and second target detection results corresponding to each human image; then, the operator's first target state and second target state are determined based on the facial detection results, the operator's first target behavior is determined based on the first target detection results, and the operator's second target behavior is determined based on at least one of the second target detection results and the position information of the facial area in the human image, and the operator's first target state, second target state, first target behavior, and second target behavior can be accurately determined. The technical solution provided by this embodiment can accurately obtain the facial detection results, first target detection results, and second target detection results corresponding to each human image, and can accurately determine the operator's first target state, second target state, first target behavior, and second target behavior, thereby improving the accuracy of the behavior detection of the remote pick-up operator and ensuring the safety of the remote pick-up.

[0118] Embodiment 3

[0119] Figure 3 A schematic diagram of a behavior detection device for a remote pick-up operator provided in Embodiment 3 of the present invention is shown in FIG. Figure 3 As shown, the device comprises:

[0120] An image acquisition module 310 is used to acquire a plurality of human body images of an operator within a set time period;

[0121] A face detection module 320 is used to perform face detection on the multiple human images to obtain a face detection result corresponding to each human image; wherein the face detection result includes the face area of ​​the operator and the position information of the face area in the human image;

[0122] A first target detection module 330 is configured to detect a first target on the plurality of human images to obtain a first target detection result corresponding to each human image; wherein the first target detection result includes whether the first target exists in the human image, and if the first target exists, the first target detection result also includes position information of the first target in the human image;

[0123] A second target detection module 340 is used to detect a second target on the multiple human images to obtain a second target detection result corresponding to each human image; wherein the second target detection result includes whether the second target exists in the human image, and if the second target exists, the second target detection result also includes position information of the second target in the human image;

[0124] The target behavior determination module 350 is used to determine the target state and target behavior of the operator based on the facial detection result corresponding to each human body image, the first target detection result, and the second target detection result.

[0125] Optionally, the target behavior determination module 350 may specifically include:

[0126] a target state determination unit, configured to determine a first target state and a second target state of the operator based on the face detection result; wherein the first target state includes fatigue or non-fatigue; and the second target state includes concentration or non-concentration;

[0127] A first target behavior determination unit, configured to determine a first target behavior of the operator based on the first target detection result; wherein the first target behavior includes whether the operator's mouth touches the first target or does not touch the first target;

[0128] A second target behavior determination unit is used to determine the second target behavior of the operator based on the second target detection result and at least one of the position information of the facial area in the human body image; wherein the second target behavior includes the operator's face contacting the second target or not contacting the second target.

[0129] Optionally, the target state determination unit may specifically include:

[0130] A position information determination subunit, used to determine position information corresponding to a plurality of key points in the facial region; wherein the key points include eye key points, mouth key points, nose tip key points and face key points;

[0131] A first target state determination subunit, configured to determine a first target state of the operator based on the position information corresponding to the eye key point and the position information corresponding to the mouth key point;

[0132] The second target state determination subunit is used to determine the second target state of the operator based on the position information corresponding to the nose tip key point and the position information corresponding to the facial key point.

[0133] Optionally, the eye key points include key points on the upper side of the eye, key points on the lower side of the eye, key points on the left side of the eye, and key points on the right side of the eye; the mouth key points include key points on the upper side of the mouth, key points on the lower side of the mouth, key points on the left side of the mouth, and key points on the right side of the mouth.

[0134] Optionally, the first target state determination subunit may be specifically used to:

[0135] Determine the width and height of the eye based on the position information of the upper key point of the eye, the lower key point of the eye, the left key point of the eye, and the right key point of the eye;

[0136] Determine the width and height of the mouth based on the position information of the key points on the upper side of the mouth, the key points on the lower side of the mouth, the key points on the left side of the mouth, and the key points on the right side of the mouth;

[0137] Determining the state of the eye based on the ratio between the width and the height of the eye; wherein the state of the eye includes an open state or a closed state;

[0138] Determining a state of the mouth based on a ratio between a width and a height of the mouth; wherein the state of the mouth includes an open state or a closed state;

[0139] A first target state of the operator is determined based on the state of the eyes and the state of the mouth.

[0140] Optionally, the facial key points include upper facial key points, lower facial key points, left facial key points and right facial key points.

[0141] Optionally, the second target state determination subunit may be specifically used to:

[0142] Determine a first distance between the nose tip key point and the upper face key point and a second distance between the nose tip key point and the lower face key point based on position information of the nose tip key point, the upper face key point and the lower face key point;

[0143] Determine, based on the position information of the nose tip key point, the left side face key point and the right side face key point, a third distance between the nose tip key point and the left side face key point and a fourth distance between the nose tip key point and the right side face key point;

[0144] Determine a first facial state based on a ratio between the first distance and the second distance; wherein the first facial state includes a nodding state or a non-nodding state;

[0145] Determining a second facial state based on a ratio between the third distance and the fourth distance; wherein the second facial state includes a head-turned state or a head-unturned state;

[0146] A second target state of the operator is determined based on the first facial state and the second facial state.

[0147] Optionally, the first target behavior determination unit may be specifically configured to:

[0148] If the first target detection result is that the first target does not exist in the human body image, then the first target behavior of the operator is that the operator's mouth does not touch the first target; or;

[0149] If the first target detection result is that the first target exists in the human image, determining a first center position of the first target based on position information of the first target in the human image;

[0150] A first target behavior of the operator is determined based on the first center position and the position information of the key points above the mouth.

[0151] Optionally, the second target behavior determination unit may be specifically configured to:

[0152] If the second target detection result is that the second target does not exist in the human body image, then the second target behavior of the operator is that the operator's face does not touch the second target; or;

[0153] If the second target detection result is that the second target exists in the human image, determining a second center position of the second target based on position information of the second target in the human image;

[0154] A second target behavior of the operator is determined based on the second center position and position information of the facial area in the human body image.

[0155] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

[0156] Embodiment 4

[0157] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0158] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0159] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0160] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a behavior detection method for a remote pick-up operator.

[0161] In some embodiments, a behavior detection method for a remote pick-up operator may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the behavior detection method for a remote pick-up operator described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute a behavior detection method for a remote pick-up operator in any other appropriate manner (e.g., by means of firmware).

[0162] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0164] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0165] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0166] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0167] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0168] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements a behavior detection method for a remote pick-up operator as provided in any embodiment of the present application.

[0169] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0170] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0171] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A behavior detection method for a remote pick-up operator, characterized in that: include: Acquire multiple human body images of the operator within a set time period; Performing facial detection on the multiple human images to obtain a facial detection result corresponding to each human image; wherein the facial detection result includes the facial region of the operator and position information of the facial region in the human image; Performing first target detection on the multiple human images to obtain a first target detection result corresponding to each human image; wherein the first target detection result includes whether the first target exists in the human image, and if the first target exists, the first target detection result also includes position information of the first target in the human image; Performing second target detection on the multiple human images to obtain a second target detection result corresponding to each human image; wherein the second target detection result includes whether the second target exists in the human image, and if the second target exists, the second target detection result also includes position information of the second target in the human image; The target state and target behavior of the operator are determined based on the face detection result, the first target detection result, and the second target detection result corresponding to each human body image.

2. The method according to claim 1, characterized in that Determining the target state and target behavior of the operator based on the face detection result, the first target detection result, and the second target detection result corresponding to each human body image includes: Determine a first target state and a second target state of the operator based on the face detection result; wherein the first target state includes fatigue or non-fatigue; and the second target state includes concentration or non-concentration; Determine a first target behavior of the operator based on the first target detection result; wherein the first target behavior includes whether the operator's mouth touches the first target or does not touch the first target; The second target behavior of the operator is determined based on at least one of the second target detection result and the position information of the facial area in the human body image; wherein the second target behavior includes the operator's face contacting the second target or not contacting the second target.

3. The method according to claim 2, characterized in that Determining a first target state and a second target state of the operator based on the face detection result includes: Determine position information corresponding to a plurality of key points in the facial region; wherein the key points include eye key points, mouth key points, nose tip key points and face key points; Determining a first target state of the operator based on the position information corresponding to the eye key point and the position information corresponding to the mouth key point; The second target state of the operator is determined based on the position information corresponding to the nose tip key point and the position information corresponding to the facial key point.

4. The method according to claim 3, characterized in that The eye key points include the upper eye key points, the lower eye key points, the left eye key points and the right eye key points; the mouth key points include the upper mouth key points, the lower mouth key points, the left mouth key points and the right mouth key points; Determining the first target state of the operator based on the position information corresponding to the eye key point and the position information corresponding to the mouth key point includes: Determine the width and height of the eye based on the position information of the upper key point of the eye, the lower key point of the eye, the left key point of the eye, and the right key point of the eye; Determine the width and height of the mouth based on the position information of the key points on the upper side of the mouth, the key points on the lower side of the mouth, the key points on the left side of the mouth, and the key points on the right side of the mouth; Determining the state of the eye based on the ratio between the width and the height of the eye; wherein the state of the eye includes an open state or a closed state; Determining a state of the mouth based on a ratio between a width and a height of the mouth; wherein the state of the mouth includes an open state or a closed state; A first target state of the operator is determined based on the state of the eyes and the state of the mouth.

5. The method according to claim 3, characterized in that: The facial key points include upper facial key points, lower facial key points, left facial key points and right facial key points; Determining the second target state of the operator based on the position information corresponding to the nose tip key point and the position information corresponding to the facial key point includes: Determine a first distance between the nose tip key point and the upper face key point and a second distance between the nose tip key point and the lower face key point based on position information of the nose tip key point, the upper face key point and the lower face key point; Determine, based on the position information of the nose tip key point, the left side face key point and the right side face key point, a third distance between the nose tip key point and the left side face key point and a fourth distance between the nose tip key point and the right side face key point; Determine a first facial state based on a ratio between the first distance and the second distance; wherein the first facial state includes a nodding state or a non-nodding state; Determining a second facial state based on a ratio between the third distance and the fourth distance; wherein the second facial state includes a head-turned state or a head-unturned state; A second target state of the operator is determined based on the first facial state and the second facial state.

6. The method according to claim 4, characterized in that Determining a first target behavior of the operator based on the first target detection result includes: If the first target detection result is that the first target does not exist in the human body image, then the first target behavior of the operator is that the operator's mouth does not touch the first target; or; If the first target detection result is that the first target exists in the human image, determining a first center position of the first target based on position information of the first target in the human image; A first target behavior of the operator is determined based on the first center position and the position information of the key points above the mouth.

7. The method according to claim 2, characterized in that Determining a second target behavior of the operator based on at least one of the second target detection result and the position information of the facial region in the human body image includes: If the second target detection result is that the second target does not exist in the human body image, then the second target behavior of the operator is that the operator's face does not touch the second target; or; If the second target detection result is that the second target exists in the human image, determining a second center position of the second target based on position information of the second target in the human image; A second target behavior of the operator is determined based on the second center position and position information of the facial area in the human body image.

8. A behavior detection device for a remote pick-up operator, characterized in that: include: An image acquisition module, used to acquire multiple human body images of the operator within a set time period; A face detection module, configured to perform face detection on the plurality of human images to obtain a face detection result corresponding to each human image; wherein the face detection result includes a face region of the operator and position information of the face region in the human image; A first target detection module, configured to detect a first target on the multiple human images, and obtain a first target detection result corresponding to each human image; wherein the first target detection result includes whether the first target exists in the human image, and if the first target exists, the first target detection result also includes position information of the first target in the human image; A second target detection module, configured to detect a second target on the plurality of human images, and obtain a second target detection result corresponding to each human image; wherein the second target detection result includes whether the second target exists in the human image, and if the second target exists, the second target detection result also includes position information of the second target in the human image; A target behavior determination module is used to determine the target state and target behavior of the operator based on the facial detection result corresponding to each human body image, the first target detection result, and the second target detection result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the behavior detection method for a remote pick-up operator according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the behavior detection method for a remote pick-up operator according to any one of claims 1 to 7 when executed.

Images