Vehicle-mounted driver behavior safety detection method and system based on multi-view visual perception

Abstract

The invention provides a vehicle-mounted driver behavior safety detection method and system based on multi-view visual perception. The system for detecting a target behavior comprises: a first camera for collecting a first image of a first shooting area, the first shooting area comprising a face of a target; the second camera is used for collecting a second image of a second shooting area, the second shooting area is larger than the first shooting area, and the second shooting area comprises the upper body of the target; the processing equipment is used for receiving and processing the first image to obtain a first processing result, and the first processing result is associated with the face of the target; the second image is received and processed to obtain a second processing result, and the second processing result is associated with the upper body of the target and an object in the environment where the target is located; and fusing the first processing result and the second processing result to determine the behavior of the target.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and automotive electronic safety technology, and in particular to a driver behavior safety detection scheme for in-vehicle environments based on multi-camera and deep learning technology. Background Technology

[0002] With the development of artificial intelligence technology, vision-based driver monitoring systems (DMS) are becoming increasingly popular. Existing solutions mostly use a single camera to monitor the driver in order to detect fatigue driving (such as yawning and dozing off) and distracted behaviors (such as talking on the phone and smoking).

[0003] However, such solutions have inherent drawbacks: limited by the camera's field of view (FOV), a single camera struggles to simultaneously capture high-precision facial details (for fatigue analysis) and a complete image of the driver's upper body (for large-scale motion recognition). When the camera is positioned on the center console and aimed at the face, it struggles to capture movements of other parts of the body, such as hands under the steering wheel or knees, making it difficult to detect dangerous actions like hands leaving the steering wheel or placing hands on knees to look at a phone. Placing the camera on the A-pillar or roof to obtain a wider field of view reduces the resolution of the facial image, affecting the accuracy of eye and mouth recognition, leading to missed and false alarms.

[0004] Therefore, there is an urgent need in this field for a detection scheme that can take into account both local details and global actions, so as to comprehensively improve the accuracy and reliability of driver behavior safety detection. Summary of the Invention

[0005] This disclosure provides a method and system for detecting driver behavior safety in a vehicle environment. According to a first aspect of this disclosure, a system for detecting target behavior is provided. The system includes: a first camera for acquiring a first image of a first shooting area, the first shooting area including the face of the target; a second camera for acquiring a second image of a second shooting area, wherein the second shooting area is larger than the first shooting area and includes the upper body of the target; and a processing device for: receiving and processing the first image to obtain a first processing result, wherein the first processing result is associated with the face of the target; receiving and processing the second image to obtain a second processing result, wherein the second processing result is associated with the upper body of the target and objects in the environment in which the target is located; and fusing the first processing result and the second processing result to determine the behavior of the target.

[0006] In some embodiments, fusing the first processing result and the second processing result to determine the behavior of the target includes: determining a corresponding score based on at least one of the first processing result and / or at least one of the second processing result; and determining the behavior of the target based on a weighted sum of the scores.

[0007] In some embodiments, the first processing result includes at least one of the following: state classification information of the eyes and / or mouth, head posture information, and facial key point location information. The second processing result includes at least one of the following: location information of objects in the environment where the target is located, upper body key point location information of the target, and action classification information.

[0008] In some embodiments, the upper body key point location information of the second processing result includes some or all facial features, wherein the facial features in the second processing result are used or not used when fusing the first processing result and the second processing result to determine the behavior of the target.

[0009] In some embodiments, the target's behavior includes driving while fatigued. The weighted fusion of the first processing result and the second processing result to determine the target's behavior specifically includes: calculating a score based on at least one of eye and / or mouth state classification information, head posture information, facial key point location information, target action classification information, and upper body key point location information; and determining whether the target's behavior constitutes driving while fatigued based on the weighted sum of the scores.

[0010] In some embodiments, a score is calculated based on at least one of the following: eye and / or mouth state classification information, head posture information, facial key point location information, target action classification information, and upper body key point location information, including at least one of the following: calculating a score based on the confidence level of the eye and / or mouth state classification; calculating a score based on the head pitch angle; calculating a score based on the coordinates of the eye and / or mouth key points; calculating a score based on the confidence level of the target action classification; and calculating a score based on the coordinates of the upper body key points.

[0011] In some embodiments, the target's behavior includes illegal driving. The fusion of the first processing result and the second processing result to determine the target's behavior includes: calculating a score based on at least one of the following: eye and / or mouth state classification information, head posture information, facial key point location information, location information of objects in the target's environment, upper body key point location information, and target's action classification information; and determining whether the target's behavior constitutes illegal driving based on a weighted sum of the scores.

[0012] In some embodiments, the processing device is configured to run a first deep learning neural network to obtain the first processing result. The first deep learning neural network includes: a first shared backbone network for extracting facial features of the target, and two or more network heads, including at least: a classification head for determining state classification information of the eyes and / or mouth based on the extracted facial features, and a first regression head for determining head pose information and facial key point location information of the target based on the extracted facial features. The processing device is configured to run a second deep learning neural network to obtain a second processing result. The second deep learning neural network includes: a second shared backbone network for extracting features in the second shooting region, and three or more network heads, including at least: a detection head for determining the category information and location information of objects in the environment based on the extracted features of the second shooting region, a second regression head for determining the upper body key point location information of the target based on the extracted features of the second shooting region, and a semantic classification head for determining the action classification information of the target based on the extracted features of the second shooting region.

[0013] In some embodiments, the target is a driver located inside a vehicle, and the objects in the environment include objects within the vehicle that interact with the driver.

[0014] In some embodiments, the target's actions are categorized as follows: lying down, holding the control components of a vehicle, smoking, eating, and using personal electronic devices.

[0015] In some embodiments, the system further includes an alarm device for issuing an alarm based on the behavior of the target.

[0016] In some embodiments, the alarm has different levels determined based on different behaviors of the target.

[0017] In some embodiments, the system further includes an intervention device for intervening in the behavior of the target based on the target's behavior.

[0018] According to a second aspect of this disclosure, a method for detecting target behavior is provided, the method comprising: acquiring a first image of a first shooting area via a first camera, the first shooting area including the face of the target; acquiring a second image of a second shooting area via a second camera, wherein the second shooting area is larger than the first shooting area and includes the upper body of the target; receiving and processing the first image via a processing device to obtain a first processing result, wherein the first processing result is associated with the face of the target; receiving and processing the second image via the processing device to obtain a second processing result, wherein the second processing result is associated with the upper body of the target and objects in the environment in which the target is located; and fusing the first processing result and the second processing result via the processing device to determine the behavior of the target.

[0019] The system disclosed herein incorporates combined input from at least two cameras, enabling multi-view fusion for driver behavior safety detection. The behavior safety detection method and system disclosed herein offer advantages in the following specific aspects: First, through the collaborative work of dual cameras, both detailed facial features and macroscopic upper body movements can be simultaneously covered, resulting in more comprehensive and accurate detection of various behaviors and eliminating the blind spots caused by a single camera. Second, by processing facial images and upper body images independently, feature interference between multiple tasks is avoided, significantly improving the accuracy of each task. Third, through multimodal information fusion and decision-making, cross-validation using information from different sources compensates for information deficiencies, significantly reducing the possibility of misjudgment based on a single signal (e.g., sudden changes in illumination may lead to facial recognition errors even when upper body behavior appears normal), thus improving the robustness and reliability of the system.

[0020] Furthermore, the behavioral safety detection method and system provided in this disclosure can adopt an architecture of "dual camera collaboration + deep learning neural network + multimodal decision module". The images obtained by the dual cameras are processed independently through the corresponding deep learning neural network, realizing dedicated network for dedicated use. That is, the facial analysis network focuses on high-precision facial features and posture regression, while the upper body network focuses on object and action recognition. The information they emphasize to detect is different, avoiding feature interference between multiple tasks of a single network.

[0021] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0022] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0023] Figure 1A A block diagram of a system for detecting target behavior according to an embodiment of the present disclosure is shown.

[0024] Figure 1B A block diagram of a face detection network according to an embodiment of the present disclosure is shown.

[0025] Figure 1C A block diagram of an upper body detection network according to an embodiment of the present disclosure is shown.

[0026] Figure 2 A flowchart illustrating a method for detecting target behavior according to an embodiment of the present disclosure is shown. Detailed Implementation

[0027] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0028] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0029] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0030] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0031] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0032] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0033] Figure 1A A block diagram of a system for detecting target behavior according to an embodiment of the present disclosure is shown. In one implementation, the system may be implemented in a vehicle. Figure 1A As shown, the detection system 100 may include: a first camera 110, a second camera 120, a processing device 130, and an optional alarm / intervention device 140.

[0034] The first camera 110 is used to capture images of a first imaging area, which includes the target's face. In a preferred embodiment, the target is the driver of the vehicle, and the first imaging area includes the driver's face. Since the first camera 110 is dedicated to face detection, it is desirable that the driver's face occupies a large portion of the first imaging area to capture a greater amount of accurate detail information. The first camera 110 is preferably mounted above the vehicle's dashboard, behind the steering wheel, or to the side of the rearview mirror, facing and close to the driver's face, to ensure that the target's face is always within the first imaging area and to reduce the possibility of the target's face being obscured.

[0035] The second camera 120 is used to capture images of a second shooting area. The second camera 120 is designed to obtain a wider field of view for more comprehensive information acquisition; therefore, the second shooting area is larger than the first shooting area of ​​the first camera 110. The second shooting area includes the upper body of the target, that is, at least the target's head, neck, shoulders, arms, and hands. The second shooting area may also include a portion of the target's environment. For example, if the target is the driver of a vehicle, the second shooting area may include components of the vehicle that can be operated by the driver (such as the steering wheel and gearshift) and other objects that may interact with the driver (such as mobile phones, cigarettes, etc.) to capture the driver's interactions with these objects. The second camera 120 is preferably mounted on the A-pillar or top of the vehicle, where its field of view covers the driver's upper body, to ensure a more comprehensive shooting area and reduce the possibility of camera obstruction.

[0036] The first camera 110 and the second camera 120 may each employ various image acquisition technologies, such as complementary metal-oxide-semiconductor (CMOS) sensors or electro-coupled device (CCD) sensors, to convert captured optical signals into electrical signals; this disclosure does not limit this. The first camera 110 and the second camera 120 may each include an analog-to-digital converter (ADC) to convert analog electrical signals into digital image data. The first camera 110 and the second camera 120 may each include encoding and compression circuitry to convert the digital image data into a format suitable for transmission. The first camera 110 and the second camera 120 may each include corresponding communication interfaces to transmit image data to the processing device 130 via wired (e.g., MIPI CSI-2, Ethernet, etc.) or wireless (e.g., Wi-Fi, 5G, etc.) means.

[0037] The processing device 130 processes images from the first camera 110 and the second camera 120 respectively, fuses the processing results, and finally determines the target's behavior based on the fused result. In a preferred embodiment, the processing device 130 has a first detection module specifically for receiving and processing image data from the first camera 110, a second detection module specifically for receiving and processing image data from the second camera 120, and a fusion decision module 136 for fusing and making a final determination. The first and second detection modules preferably use deep learning neural networks to process the received image data, and are therefore also referred to as a "face detection network" 132 and an "upper body detection network" 134, respectively. The processing device 130 can implement the functions of its modules by running code (computer-readable program instructions) on processors such as CPUs and GPUs. Each module can also be implemented by running code on its sub-devices.

[0038] like Figure 1BAs shown, in some embodiments, the face detection network 132 has a shared backbone network 1322, which is used to extract facial features of the target from received image data from the first camera 110. This backbone network can be a convolutional neural network (CNN), a Transformer-type backbone network (e.g., Transformer Encoder), a graph neural network (e.g., GCN), etc., and this disclosure is not limited thereto. The face detection network 132 includes at least two network heads: a classification head 1324 and a regression head 1326. The facial features extracted by the shared backbone network 1322 are output to the classification head 1324 and the regression head 1326, respectively. The classification head 1324 determines state classification information for certain body parts (e.g., eyes and mouth) based on the facial features extracted by the shared backbone network 1322. The regression head 1326 determines the pose information of the target's head and the position information of key points on the face based on the facial features extracted by the shared backbone network 1322. The information obtained from each head is transmitted to the fusion decision module 136 as the processing result of the face detection network 132 for further processing.

[0039] like Figure 1C As shown, in some embodiments, the upper body detection network 134 has a shared backbone network 1342, which is used to extract features from the received image data from the second camera 120 in a second shooting region, including features of the target's upper body and features of objects in the environment (e.g., a steering wheel, a mobile phone). The shared backbone networks 1322 and 1342 can be the same network, i.e., the face detection network 132 and the upper body detection network 134 use the same deep learning neural network to extract features from their respective received images; or they can be different networks. The upper body detection network 134 also includes at least three network heads: a detection head 1344, a regression head 1346, and a semantic classification head 1348. The detection head 1344 determines the category and location information of the corresponding object based on the features extracted from the second shooting region by the shared backbone network 1342, including features of objects in the environment. The regression head 1346 determines the location information of key points of the upper body based on the features extracted from the second shooting region by the shared backbone network 1342, including features of the target's upper body. The semantic classification head 1348 determines action classification information for the target based on features extracted from the second imaging region by the shared backbone network 1342, including features of the target's upper body and features of objects in the environment. The information obtained by each head is transmitted to the fusion decision module 136 as the processing result of the upper body detection network 134 for further processing.

[0040] The fusion decision module 136 receives the processing results from the face detection network 132 and the upper body detection network 134 respectively, and fuses these processing results as described below to determine the target's behavior. For example, the fusion decision module 136 can detect unsafe behaviors of the driver, such as driving while fatigued or making phone calls.

[0041] Optionally, the target behavior determined by the fusion decision module 136 is encoded and transmitted as data to the alarm / intervention device 140. The alarm / intervention device 140 determines whether an alarm needs to be issued based on the target behavior indicated by the received data. For example, if the received data indicates that the target is performing normal driving behavior such as turning the steering wheel or adjusting the sun visor, no alarm needs to be issued. Conversely, if the received data indicates that the target is performing unsafe behavior such as making a phone call or eating, an alarm needs to be issued to the driver, supervisor, or other user.

[0042] The first detection module, the second detection module, and the fusion module can be trained using sample datasets (e.g., sample images and labels) based on training methods in related technologies. The first detection module, the second detection module, and the fusion module can be trained independently or jointly. This application does not restrict the training method.

[0043] The following will refer to Figure 2 The specific steps of the method 200 for detecting target behavior executed by the detection system 100 are described in detail.

[0044] The detection method 200 begins with step S210. In step S210, a first image of a first shooting area is acquired by a first camera 110, and a second image of a second shooting area is acquired by a second camera 120. As described above, the first image is a facial image of the target; the second image is an upper body image of the target, and also includes other objects in the environment. In this disclosure, "first image" and "second image" can each represent a single frame or a series of image frames. In a preferred embodiment, the first image and the second image are synchronized in time. The first camera 110 and the second camera 120 can each acquire one or more frames at the same point in time. The first camera 110 and the second camera 120 can also each acquire a series of image frames at the same frequency starting at the same point in time, such that for each image frame in the series of image frames from the first camera 110, a corresponding image frame can be found in the series of image frames from the second camera 120. By synchronizing the first image or the second image in time, it can be ensured that the acquired images reflect the target's actions and state at the same time, thereby improving the accuracy and reliability of behavior detection. In another embodiment, the first image and the second image may also be asynchronous in time, as long as they reflect the same behavior of the target. For example, a time difference of less than 100 ms between the first and second images is considered acceptable as it will not have a substantial impact on the final behavior detection result.

[0045] In step S220, the first image is received and processed by the processing device 130 to obtain a first processing result.

[0046] Specifically, the face detection network 132 in the processing device 130 receives the first image. The received first (facial) image is first input into the shared backbone network 1322 of the face detection network 132 for feature extraction; image feature extraction can be performed using techniques known in the art, and this disclosure does not limit this. The features extracted from the first image include key points of the target face as defined by standards such as iBUG and AFLW, such as eyebrows, eyes, bridge of the nose, and tip of the nose. These features will be output from the shared backbone network 1322 to the classification head 1324 and the regression head 1326 for further processing.

[0047] The classification head 1324 determines state classification information for specific parts of the target face based on facial features extracted by the shared backbone network 1322. In embodiments of this disclosure, these specific parts include the eyes and mouth, as these two areas are particularly important for behavior detection related to vehicle driving; other areas, such as eyebrows and jaw, may also be included depending on the specific application requirements, and this disclosure does not impose any limitations on this. As a result of the processing by the classification head 1324, multiple state classification information can be obtained for different types of states of different parts. For example, the state classification information for the eyes may include classification information regarding the state of the eyes being occluded / unoccluded, and classification information regarding the state of the eyes being closed / open. The state classification information for the mouth may include classification information regarding the state of the mouth being occluded / unoccluded, and classification information regarding the state of the mouth being closed / open. The classification head 1324 provides the state classification information in the form of confidence scores. For example, a confidence score of 0.1 for a closed mouth state indicates a very low probability of the mouth being closed, meaning that the mouth is actually very likely to be "open." A confidence score of 0.9 for a closed mouth state indicates that the mouth is very likely to be "closed." To avoid confusion between normal behaviors like speaking and states of fatigue like yawning, the confidence score can be configured to be very low (e.g., 0) when the mouth opening is less than a threshold, and greater than 0 when the mouth opening is greater than a threshold.

[0048] The regression head 1326 determines the target's head pose information and facial key point location information based on facial features extracted by the shared backbone network 1322. The facial key point location information can be represented as the coordinate positions of each key point in the first image. In a preferred embodiment, a Cartesian coordinate system is constructed with the lower left corner of the first image as the origin; alternatively, a Cartesian coordinate system or a polar coordinate system can be constructed based on other positions in the first image or specific positions on the target's face, and this disclosure does not impose any limitations on this. Similarly, based on multiple facial key points extracted by the shared backbone network 1322, the regression head can determine the target's head pose, including the head's upward and downward angles, left and right angles, and tilt angles, corresponding to pitch, yaw, and roll angles, respectively.

[0049] The first processing result obtained by performing step S220 is associated with the target's face. This first processing result may include the outputs of the classification head and regression head described above, specifically including at least one of the following: state classification information for the eyes and / or mouth, head pose information, and facial key point location information. In another embodiment, depending on the specific application scenario, the first processing result for the facial image may also include other information, such as state classification information for other body parts.

[0050] In step S230, the processing device 130 receives and processes the second image to obtain the second processing result.

[0051] The upper body detection network 134 in the processing device 130 receives the second image. The received second image (upper body image) is first input into the shared backbone network 1342 in the upper body detection network 134 for feature extraction; image feature extraction can be achieved using techniques known in the art, and this disclosure does not limit it. The features extracted from the second image include key points of various parts of the target's upper body (e.g., head, neck, shoulders, arms, hands, etc.); and also key points of other objects in the target's environment that may interact with the target (e.g., steering wheel, mobile phone). These features will be output from the shared backbone network 1342 to the detection head 1344, regression head 1346, and semantic classification head 1348 for further processing.

[0052] The detection head 1344 is used to determine the category and location information of objects in the environment based on features of the second imaging area extracted by the shared backbone network 1342. In some embodiments, the detection head 1344 may, for example, employ an anchor-based technique or an anchor-free technique to identify the object category and determine the object's location in the environment. Specifically, the determined object location is expressed as the coordinate position of the object's bounding box (bbox) in a coordinate system. Similar to the embodiments described above, a Cartesian coordinate system or a polar coordinate system may be constructed based on a specific location in the second image, a specific location on the upper body of the target, or a specific location on an object in the environment; this disclosure does not impose any limitations on this.

[0053] The regression head 1346 determines the positional information of key points (e.g., head, neck, shoulders, arms, hands, etc.) of the target's upper body based on features extracted from the second imaging region by the shared backbone network 1342. Similar to the function of the regression head 1326, the key point positional information can be represented as the coordinate positions of each key point in the second image. The coordinate system used to determine these coordinate positions is similar to the coordinate system settings described above for facial key point position information and object position information in the environment, and will not be repeated here. In a preferred embodiment, the regression head 1346 and the detection head 1344 use the same coordinate system to determine the positions of objects in the environment and the coordinate positions of key points on the target's upper body, facilitating subsequent calculations.

[0054] The semantic classification head 1348 determines the classification of the target's actions based on features extracted from the second imaging region by the shared backbone network 1342. Specifically, the semantic classification head 1348 can output the probability for each action category by transforming the features of the second imaging region into vectors, performing fully connected mapping, and normalizing the probabilities. The semantic classification head 1348 can provide multiple action classification information for different action categories, such as lying down, holding the control parts of a vehicle, smoking, eating, using personal electronic products, and other actions that may affect the safety of the operation performed by the target. Similar to the classification head 1324 mentioned above, the semantic classification head 1348 provides action classification information in the form of action classification confidence. For example, the action classification confidence of the action "lying down" is 0.1, indicating that the probability of the target performing the "lying down" action is very small, that is, it actually means that the target is very likely not lying down; the action classification confidence of the action "lying down" is 0.9, indicating that there is a high probability that the target did indeed perform the lying down action.

[0055] The second processing result obtained by executing step S230 is associated with the upper body of the target and objects in the environment. This second processing result may include the outputs of the detection head, regression head, and semantic classification head, specifically including at least one of the following: location information of objects in the target's environment, location information of key points on the upper body, and action classification information of the target. In another embodiment, depending on the specific application scenario, the second processing result for the upper body image may also include other information, such as pose information of environmental objects and state classification information of specific parts of the upper body.

[0056] The above steps S220 and S230 can be used with Figure 2 The different orders shown can be performed simultaneously, as long as the corresponding first and second processing results can be obtained.

[0057] After both steps S220 and S230 are completed, step S240 is performed. In step S240, the first processing result and / or the second processing result are fused by the processing device 130 to determine the behavior of the target.

[0058] Specifically, such as Figure 1AAs shown, the first processing result from the face detection network 132 and the second processing result from the upper body detection network 134 are both transmitted to the fusion decision module 136 for further processing. As described above, the first and second processing results may each include one or more pieces of information, such as the state of facial organs (from the classification head 1324), head pose and facial key point positions (from the regression head 1326), bounding boxes of objects in the environment (from the detection head 1344), the position of specific parts of the upper body (from the regression head 1346), and the classification of upper body movements (from the semantic classification head 1348), etc. In order to comprehensively utilize this information to finally determine the target's behavior, this information needs to be appropriately fused. Such fusion may include scoring each piece of information in the first and second processing results separately, then weighting the scores and calculating a weighted sum to obtain the final score used to determine the target's behavior.

[0059] For example, regarding classification information about the closed / open state of the eyes from classification head 1324, the confidence value can be directly used as the score, or the reciprocal of the confidence value or the absolute value of the difference between the confidence value and 1 can be used as the score, or other numerical calculations can be performed based on the confidence value to obtain the score; this disclosure does not limit this. Similarly, scores can be given for the states of eyes being occluded / unoccluded, mouth being closed / unoccluded, and mouth being occluded / unoccluded. For head posture from regression head 1326, scores can be given, for example, by comparing the head pitch angle with a reference pitch angle or comparing the head roll angle with a reference roll angle, and the score indicates the size of the head pitch angle or roll angle. For facial key point positions from regression head 1326, scores can be given, for example, by calculating the relative positions of the facial key points, or by comparing the positions of the facial key points with corresponding reference positions, and the score indicates the degree of opening and closing of facial organs (e.g., eyes, mouth, etc.). For the position of objects in the environment from the detection head 1344 and the position of upper body keypoints from the regression head 1346, scoring can be performed, for example, by calculating the relative position (e.g., whether they overlap) between the object's bounding box and the upper body keypoints, or by calculating the relative position between the object's bounding box or upper body keypoints and the facial keypoints from the regression head 1326. The score indicates the degree to which the object is close to or far from specific parts of the upper body (e.g., hands, mouth, ears). For action classification information from the semantic classification head 1348, scoring can be performed in a manner similar to that used for facial state classification information, and will not be repeated here. It is understood that other scoring methods can be used for different application scenarios for the information in the first and second processing results, as long as they are convenient for quantitative evaluation in the final behavior detection; this disclosure does not impose any limitations on this.

[0060] In step S240, after obtaining multiple scores based on the information from the first and second processing results, different scores can be selected for fusion to determine the final behavior for different behaviors to be detected. An example of fusion for a sample behavior is given below.

[0061] The behaviors to be detected include fatigued driving. For this behavior, the following scores can be fused. For example, the eye-closed / open state score reflects whether the target's eyes are open, a key factor in determining fatigued driving. Similarly, the vertical distance between key points on the upper and lower contours of the eyes also reflects whether the target's eyes are open; therefore, scores based on the relative positions of key points of the eyes are also considered. Additionally, if the eyes are obstructed, it may negatively impact the accuracy of fatigued driving assessment; therefore, the eye-obstructed / unobstructed state score is also considered. Similar to the above scores regarding eye-closed / open state and key points of the eyes, the mouth-closed / open state score and key points of the mouth may reflect the target's yawning state and are therefore also considered. Head posture may reflect whether the head is tilted to one side; upper body detection, such as the positional information of key points on the shoulders, can reflect whether the target's body is tilted to one side; the confidence level of action classification can indicate the probability of the target bending over or crouching; therefore, scores related to this information are also considered. All or some of the above items can be selected for weighted fusion. For example, only the scores of the state classification of the eyes and / or mouth, the score of the head pitch angle, and the scores of key points of the eyes and / or mouth can be used for fusion to determine whether the target is in a state of fatigued driving.

[0062] To integrate the multiple scores mentioned above into a final score used to determine whether a target is driving while fatigued, weights are assigned to these score distributions. For example, scores indicating whether the eyes are obscured or not, and scores indicating whether the eyes are closed or open, can have higher weights; while scores indicating the location information of key shoulder points can have lower weights.

[0063] After assigning weights to each score, the weighted scores are summed to obtain the final score for the behavior of "fatigue driving." The validity of the detection of this behavior is determined by comparing the final score with a reference value. For example, if the final score is greater than or equal to the reference value, the behavior detection is considered valid, meaning the target is driving while fatigued. If the final score is less than the reference value, the behavior detection is considered invalid, meaning the target is not driving while fatigued.

[0064] The behaviors to be detected also include distracting or unauthorized behaviors, such as making or receiving phone calls, smoking, lying down, or taking both hands off the steering wheel. The first and second processing results can be combined for different behaviors to determine whether the corresponding behavior has occurred. The items used in the first and second processing results in the fusion can differ for different behaviors.

[0065] For example, regarding the act of making and receiving phone calls, the following scores can be fused. For instance, if the bounding box of the "phone" object detected by the detection head 1344 coincides with the position of the ear keypoint determined by the regression head 1326, it indicates that the target may be making or receiving a phone call. Therefore, the score obtained by calculating the relative position of the object's bounding box and the facial keypoint is taken into consideration. Note that calculating this score may involve transformations between different coordinate systems from different images; for the sake of highlighting the focus of this disclosure, this is not described in detail herein. Similarly, the score obtained by calculating the relative position of the object (phone)'s bounding box and the upper body keypoint (hand) is also taken into consideration. In addition, the semantic classification head 1348 itself can provide a confidence score for classifying the action of "making and receiving phone calls"; the score based on this confidence score is also taken into consideration. Then, similar to the above embodiments, these scores can be weighted, the weighted scores can be summed, and the final summed score can be compared with a reference value to determine whether the action of "making and receiving phone calls" has occurred. For example, facial information may show the eyes looking down, but it cannot make an accurate judgment. By combining it with upper body information in a weighted manner, it is possible to capture the driver's hand waving their phone under the steering wheel.

[0066] Distracting or unauthorized behaviors can also include "smoking." For this behavior, the following scores can be fused. For example, if the bounding box of a "cigarette" object detected by the detection head 1344 coincides with the position of a keypoint about the mouth determined by the regression head 1326, it indicates that the target may be smoking. Therefore, the score obtained by calculating the relative position of the object's bounding box to the facial keypoint is taken into consideration. Similarly, the score obtained by calculating the relative position of the object's (cigarette) bounding box to the upper body keypoint (hand) is also taken into consideration. Since the mouth is necessarily closed when holding a cigarette, the score based on the confidence level of the mouth-closed / open state classification is taken into consideration. Additionally, the semantic classification head 1348 itself can provide a confidence level for classifying the action of "smoking"; the score based on this confidence level is also taken into consideration. Then, similar to the above embodiment, these scores can be weighted, the weighted scores can be summed, and the final summed score can be compared with a reference value to determine whether the target has engaged in the behavior of "smoking." Distractions can also include physical discomfort. For example, facial information may capture changes in the target's facial expressions (such as slightly opening the mouth or occasionally closing the eyes), but it is not possible to make an accurate judgment. By weighted fusion of upper body information, if the driver's hand is placed on a certain part of the body, it can be determined that the driver is experiencing physical discomfort.

[0067] Distracted or unauthorized behaviors can also include "lying down." For this behavior, the following scores can be fused. For example, the head posture determined by regression head 1326, specifically the head pitch angle, can be an important indicator of lying down behavior; therefore, the score calculated based on the head pitch angle (e.g., the difference between the head pitch angle and a reference pitch angle) is taken into consideration. A low shoulder position can also reflect that the target is lying down; therefore, the score calculated based on upper body keypoints is also taken into consideration. If the bounding box position of an object such as a steering wheel detected by detection head 1344 coincides with the position of an arm or shoulder keypoint determined by regression head 1346, it may also indicate that the target is lying down. Therefore, the score obtained by calculating the relative position of the object's bounding box and upper body keypoints is also taken into consideration. Additionally, semantic classification head 1348 itself can provide a confidence score for classifying the action "lying down"; the score based on this confidence score is also taken into consideration. Then, similar to the above embodiments, weights can be assigned to these scores, the weighted scores can be summed, and the final summed score can be compared with a reference value to determine whether the "lying down" behavior has been performed. In other words, if only the first camera is used, the facial information cannot be captured of the driver, and an accurate judgment cannot be made. The upper body information needs to be combined to capture the driver's current lying down state.

[0068] Distracted or unauthorized behaviors can also include "hands off the steering wheel." For this behavior, scores can be fused together. For example, if the bounding box of the steering wheel object detected by detection head 1344 does not touch or overlap with the position of the hand keypoints determined by regression head 1346, it may also indicate that the target's hands have left the steering wheel. The relative relationship between the positions of facial keypoints determined by regression head 1326 and the positions of hand keypoints determined by regression head 1346 can also be calculated, and a score can be calculated based on this relative positional relationship to consider, for example, whether the target's hands are in contact with their face. Additionally, semantic classification head 1348 itself can provide a confidence level for the action classification of "hands off the steering wheel"; a score based on this confidence level is also considered. Then, similar to the above embodiments, these scores can be weighted, the weighted scores can be summed, and the final summed score can be compared with a reference value to determine whether the "hands off the steering wheel" behavior has occurred.

[0069] The detection of the above behaviors is merely an illustrative embodiment and is not intended to limit the scope of this disclosure. For example, different scores for different items in the first and second processing results can be selected and weighted differently as needed to detect other different behaviors (such as "one hand off the steering wheel", "chatting", "eating", "applying makeup", etc.).

[0070] Optionally, the detection method 200 may also include step S250. In step S250, an alarm is issued by the alarm / intervention device 140 based on the behavior of the target determined in step S240. For example, if unsafe behavior such as making a phone call or eating is detected in step S240, an alarm needs to be issued. The alarm may be sent to the target (e.g., the driver) in the form of sound, such as by sounding an alarm in the driver's cab of the vehicle. The alarm may also be issued to the target in the form of optical (e.g., flashing dashboard icons) or tactile (e.g., vibration of the steering wheel), or by the vehicle's driver assistance system through interface prompts or voice commands. Alternatively, the alarm command may be sent wirelessly to a remote location and, in an appropriate manner, to persons other than the target (e.g., supervisors).

[0071] In some embodiments, different alarm levels can be set according to the application scenario and for different detected behaviors. For example, in a factory environment, a higher alarm level is set for the behavior of "smoking," in which case an alarm sound and flashing lights will be used simultaneously to alert the user; while a lower alarm level is set for behaviors such as "taking both hands off the steering wheel," which is only alerted by illuminating a specific icon on the dashboard. In another driving scenario, such as on a highway, a higher alarm level is set for the behavior of "drowsy driving," in which case an alarm sound and flashing lights will be used simultaneously to alert the user; while a lower alarm level is set for behaviors such as "smoking," which is only alerted by the vibration of the steering wheel. It is understood that other alarm levels and alarm methods can also be set, and this disclosure does not limit this.

[0072] In some embodiments, at step S250, the alarm / intervention device 140 also intervenes based on the behavior of the target determined in step S240. For example, the alarm / intervention device 140 may send instructions to components of the vehicle to control these components to intervene if intervention is deemed necessary. For example, if the behavior of "fatigue driving" is determined in step S240, the alarm / intervention device 140 may automatically lower the windows and / or automatically play music to keep the driver alert. The alarm / intervention device 140 may also be connected to an advanced driver assistance system (ADAS), and the ADAS may send instructions to the corresponding components to intervene. For example, if the behavior of "hands off the steering wheel" is determined in step S240, the ADAS may control the current driving direction of the steering mechanism components and control the actuation components to decelerate. It is understood that other intervention methods may be set for other behaviors, and this disclosure does not limit this.

[0073] In some embodiments, the detection system 100 can be implemented entirely within the vehicle, meaning that cameras 110 and 120, processing device 130, and alarm / intervention device 140 are all vehicle-mounted devices, and the corresponding detection method 200 is also executed entirely within the vehicle. In another embodiment, the detection system 100 can also be implemented in a distributed manner. For example, cameras 110 and 120 and alarm / intervention device 140 can be installed in the vehicle, while processing device 130 can be located at a remote computing center, and the two can be connected wirelessly to jointly complete the detection method 200; thus, steps S210 and 250 of the detection method 200 can be performed at the vehicle, while steps S220-S240 are completed at the computing center.

[0074] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0075] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0076] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0077] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods and apparatus (systems) according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0078] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0079] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0081] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant regions.

[0082] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A system for detecting target behavior, characterized in that, include: A first camera is used to capture a first image of a first shooting area, the first shooting area including the face of the target; A second camera is used to capture a second image of a second shooting area, wherein the second shooting area is larger than the first shooting area, and the second shooting area includes the upper body of the target; Processing equipment, used for: The first image is received and processed to obtain a first processing result, wherein the first processing result is associated with the face of the target; The second image is received and processed to obtain a second processing result, wherein the second processing result is associated with the upper body of the target and objects in the environment in which the target is located; and The behavior of the target is determined by fusing the first processing result and the second processing result.

2. The system according to claim 1, characterized in that, The step of fusing the first processing result and the second processing result to determine the behavior of the target includes: A corresponding score is determined based on at least one of the first processing results and / or at least one of the second processing results; and The behavior of the target is determined by the weighted sum of the scores.

3. The system according to claim 2, characterized in that, The first processing result includes at least one of the following: state classification information of the eyes and / or mouth, head posture information, and facial key point location information; and The second processing result includes at least one of the following: the location information of objects in the environment where the target is located, the location information of key points of the upper body, and the action classification information of the target.

4. The system according to claim 3, characterized in that, The upper body key point location information of the second processing result includes some or all facial features, wherein the facial features in the second processing result are used or not used when the first processing result and the second processing result are fused to determine the behavior of the target.

5. The system according to claim 3, characterized in that, The target's behavior includes driving while fatigued. The step of fusing the first processing result and the second processing result to determine the behavior of the target includes: Scores are calculated based on at least one of the following: eye and / or mouth status classification information, head posture information, facial key point location information, target motion classification information, and upper body key point location information; and The weighted sum of the scores determines whether the target's behavior constitutes fatigued driving.

6. The system according to claim 5, characterized in that, The scoring is calculated based on at least one of the following: eye and / or mouth state classification information, head posture information, facial key point location information, target action classification information, and upper body key point location information, including at least one of the following: Scores are calculated based on the confidence level of the classification of the condition of the eyes and / or mouth; The score is calculated based on the head's pitch angle; Scoring is calculated based on the coordinates of key points around the eyes and / or mouth; The score is calculated based on the confidence level of the target's action classification; as well as The score is calculated based on the coordinates of key points on the upper body.

7. The system according to claim 3, characterized in that, The target's behavior includes illegal driving. The step of fusing the first processing result and the second processing result to determine the behavior of the target includes: Scores are calculated based on at least one of the following: eye and / or mouth state classification information, head posture information, facial key point location information, object location information in the target's environment, upper body key point location information, and target motion classification information; and The weighted sum of the scores determines whether the target's behavior constitutes a violation of driving rules.

8. The system according to claim 3, characterized in that, The target's action classification includes: Lie prone, The control components that hold the vehicle. Smoking Eating, and Use personal electronic products.

9. The system according to claim 1, characterized in that, The processing device is configured as follows: Running a first deep learning neural network to obtain the first processing result, the first deep learning neural network includes: A first shared backbone network is used to extract the facial features of the target, and Two or more network headers, including at least: A classification head is used to determine the state classification information of the eyes and / or mouth based on the extracted facial features, and The first regression head is used to determine the target's head pose information and facial key point location information based on the extracted facial features, and Run a second deep learning neural network to obtain a second processing result, the second deep learning neural network comprising: The second shared backbone network is used to extract features from the second captured region, and Three or more network headers, including at least: The detection head is used to determine the category and location information of objects in the environment based on the features extracted from the second imaging area. The second regression head is used to determine the location information of key points on the upper body of the target based on the extracted features of the second shooting area, and A semantic classification head is used to determine the action classification information of the target based on the features extracted from the second shooting area.

10. The system according to claim 1, characterized in that, The target is the driver located inside the vehicle, and the objects in the environment include objects within the vehicle that interact with the driver.

11. The system according to claim 1, characterized in that, It also includes alarm devices for issuing alarms based on the target's behavior.

12. The system according to claim 11, characterized in that, The alert has different levels determined based on the different behaviors of the target.

13. The system according to claim 1, characterized in that, It also includes intervention devices for intervening in the behavior of the target based on the target's behavior.

14. A method for detecting target behavior, characterized in that, include: A first image of a first shooting area is captured by a first camera, the first shooting area including the face of the target; A second image is captured by a second camera in a second shooting area, wherein the second shooting area is larger than the first shooting area and includes the upper body of the target; The first image is received and processed by a processing device to obtain a first processing result, wherein the first processing result is associated with the face of the target; The processing device receives and processes the second image to obtain a second processing result, wherein the second processing result is associated with the upper body of the target and objects in the environment in which the target is located; and The behavior of the target is determined by fusing the first processing result and the second processing result using the processing device.

15. The method according to claim 14, characterized in that, The step of fusing the first processing result and the second processing result to determine the behavior of the target includes: A corresponding score is determined based on at least one of the first processing results and / or at least one of the second processing results; and The behavior of the target is determined by the weighted sum of the scores.

16. The method according to claim 14, characterized in that, The first processing result includes at least one of the following: state classification information of the eyes and / or mouth, head posture information, and facial key point location information; and The second processing result includes at least one of the following: the location information of objects in the environment where the target is located, the location information of key points of the upper body, and the action classification information of the target.

17. The method according to claim 16, characterized in that, The upper body key point location information of the second processing result includes some or all facial features. When fusing the first processing result and the second processing result to determine the behavior of the target, the facial features in the second processing result may or may not be used.

18. The method according to claim 16, characterized in that, The target's behavior includes driving while fatigued. The step of fusing the first processing result and the second processing result to determine the behavior of the target includes: Scores are calculated based on at least one of the following: eye and / or mouth status classification information, head posture information, facial key point location information, target motion classification information, and upper body key point location information; and The weighted sum of the scores determines whether the target's behavior constitutes fatigued driving.

19. The method according to claim 18, characterized in that, The scoring is calculated based on at least one of the following: eye and / or mouth state classification information, head posture information, facial key point location information, target action classification information, and upper body key point location information, including at least one of the following: Scores are calculated based on the confidence level of the classification of the condition of the eyes and / or mouth; The score is calculated based on the head's pitch angle; Scoring is calculated based on the coordinates of key points around the eyes and / or mouth; The score is calculated based on the confidence level of the target's action classification; as well as The score is calculated based on the coordinates of key points on the upper body.

20. The method according to claim 16, characterized in that, The target's behavior includes illegal driving. The step of fusing the first processing result and the second processing result to determine the behavior of the target includes: Scores are calculated based on at least one of the following: eye and / or mouth state classification information, head posture information, facial key point location information, object location information in the target's environment, upper body key point location information, and target motion classification information; and The weighted sum of the scores determines whether the target's behavior constitutes a violation of driving rules.

21. The method according to claim 16, characterized in that, The target's action classification includes: Lie prone, The control components that hold the vehicle. Smoking Eating, and Use personal electronic products.

22. The method according to claim 14, characterized in that, The method further includes: Running a first deep learning neural network to obtain the first processing result, the first deep learning neural network includes: A first shared backbone network is used to extract the facial features of the target, and Two or more network headers, including at least: A classification head is used to determine the state classification information of the eyes and / or mouth based on the extracted facial features, and The first regression head is used to determine the target's head pose information and facial key point location information based on the extracted facial features, and Running a second deep learning neural network to obtain the second processing result, the second deep learning neural network includes: The second shared backbone network is used to extract features from the second captured region, and Three or more network headers, including at least: The detection head is used to determine the category and location information of objects in the environment based on the features extracted from the second imaging area. The second regression head is used to determine the location information of key points on the upper body of the target based on the extracted features of the second shooting area, and A semantic classification head is used to determine the action classification information of the target based on the features extracted from the second shooting area.

23. The method according to claim 14, characterized in that, The target is the driver located inside the vehicle, and the objects in the environment include objects within the vehicle that interact with the driver.

24. The method according to claim 14, characterized in that, It also includes issuing alerts based on the target's behavior.

25. The method according to claim 24, characterized in that, The alert has different levels determined based on the different behaviors of the target.

26. The method according to claim 14, characterized in that, It also includes intervening in the target's behavior based on the target's behavior.

Images