Target behavior processing method and device, vehicle and storage medium

Abstract

The present disclosure relates to a target behavior processing method and device, a vehicle and a storage medium. The method comprises: a target image extraction step of extracting a target image for detecting a target behavior from a to-be-tested image or a to-be-tested video frame; a target image instance segmentation step of performing instance segmentation on the target image to obtain a first segmentation result of a target human body and a second segmentation result of a target object in the target image, wherein the first segmentation result comprises shape information of the target human body, and the second segmentation result comprises shape information of the target object; a target behavior detection step of performing human-object interaction (HOI) detection based on feature information fused with the instance segmentation process to determine interaction information of the target human body and the target object; and a target behavior determination step of determining whether the target human body has performed the target behavior based on the interaction information. Thus, the accuracy of target behavior detection can be improved, and the false detection rate of target behavior can be reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and particularly relates to a target behavior processing method and device, a carrier and a storage medium. BACKGROUND

[0002] In the scenario of driver behavior detection, the currently commonly used method is to extract, by a target detection method, cigarette information in an image, and then to determine whether there is a smoking behavior.

[0003] However, since the target detection method obtains less features of the cigarette and the cigarette has a small volume, the detection accuracy of behaviors such as smoking is poor, and more than that, in driving scenarios such as industrial vehicles with complex cockpit backgrounds, the target detection is more likely to cause mis-detection. SUMMARY

[0004] Therefore, the present disclosure provides a target behavior processing method and device, a carrier and a storage medium, so as to improve the accuracy of target behavior detection and reduce the mis-detection rate of target behavior.

[0005] According to a first aspect of the present disclosure, a target behavior processing method is provided, comprising: a step of extracting a target image for detecting the target behavior from a to-be-detected image or a to-be-detected video frame; a step of performing instance segmentation on the target image to obtain a first segmentation result of a target human body and a second segmentation result of a target object in the target image, wherein the first segmentation result comprises shape information of the target human body, and the second segmentation result comprises shape information of the target object; a step of performing human-object interaction HOI detection based on feature information fused with the instance segmentation process to determine interaction information of the target human body and the target object; and a step of determining whether the target human body has performed the target behavior based on the interaction information.

[0006] According to a second aspect of the present disclosure, a target behavior processing device is provided, comprising: an extraction module configured to extract a target image for detecting the target behavior from a to-be-detected image or a to-be-detected video frame; an acquisition module configured to perform instance segmentation on the target image to obtain a first segmentation result of a target human body and a second segmentation result of a target object in the target image, wherein the first segmentation result comprises shape information of the target human body, and the second segmentation result comprises shape information of the target object; a detection module configured to perform human-object interaction HOI detection based on feature information fused with the instance segmentation process to determine interaction information of the target human body and the target object; and a determination module configured to determine whether the target human body has performed the target behavior based on the interaction information.

[0007] According to a third aspect of the present disclosure, a processing apparatus of a target behavior is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above processing method when executing the instructions stored in the memory.

[0008] According to a fourth aspect of the present disclosure, a vehicle is provided, which is configured with a driver monitoring system (DMS) carrying the processing apparatus according to the second aspect or the third aspect.

[0009] According to a fifth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, which stores computer program instructions, wherein the computer program instructions are executed by a processor to implement the above processing method.

[0010] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above processing method.

[0011] Through the above processing method and apparatus of a target behavior, vehicle and storage medium, the target image is instance segmented to obtain respective instance segmentation results of the target human body and the target object in the target image, HOI detection is performed based on the feature information generated in the instance segmentation process to determine the interaction information of the target human body and the target object corresponding to the shape information included in the instance segmentation results, and whether the target human body performs the target behavior is determined based on the interaction information. Therefore, the accuracy of target behavior detection can be improved, and the false detection rate of target behavior can be reduced.

[0012] Among them, the features of the target human body and the target object instance segmentation pixel level are extracted, and the interaction relationship of the target human body and the target object pair is predicted, the instance segmentation task and the interaction relationship prediction task share the features extracted by the part CNN model, which has the following advantages: lightweight backbone network design, small amount of calculation, short time consumption, and is very suitable for application scenarios with limited computing power such as vehicle-mounted equipment; Unlike many existing methods based on target detection technology, the present disclosure extracts pixel-level semantic information and deeper human interaction semantic information, which is more robust; and is easy to extend to detection of other abnormal driving behaviors such as eating and drinking; under the condition of maintaining a certain TPR recall rate (recall / true positive rate), the false detection rate is low, supplemented by a filtering module, further reducing the occurrence of false detection, especially in industrial vehicle driving with complex cockpit background.

[0013] Other features and aspects of the present disclosure will become apparent from a detailed description of exemplary embodiments with reference to the following drawings. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0015] Figure 1 is a flowchart of a processing method of a target behavior according to an embodiment of the present disclosure.

[0016] Figure 2 is a flowchart of an extraction step according to an embodiment of the present disclosure.

[0017] Figure 3 is a schematic diagram of a clipping manner of a target image according to an embodiment of the present disclosure.

[0018] Figure 4 is a schematic diagram of a principle of performing instance segmentation and HOI detection according to an embodiment of the present disclosure.

[0019] Figure 5 is a schematic diagram of a result example of smoking and phone call behavior detection according to an embodiment of the present disclosure.

[0020] Figure 6 is a schematic diagram of internal structures of a backbone network and a feature pyramid network according to an embodiment of the present disclosure.

[0021] Figure 7 is a flowchart of a processing method of a target behavior according to an embodiment of the present disclosure.

[0022] Figure 8 is a block diagram of a processing apparatus of a target behavior according to an embodiment of the present disclosure.

[0023] Figure 9 is a block diagram of a processing apparatus of a target behavior according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Various exemplary embodiments, features, and aspects of the present disclosure will be explained hereinafter with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0025] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0026] In addition, for a better illustration of the present disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that the present disclosure can be practiced without certain specific details. In some instances, well-known methods, apparatuses, elements and circuits have not been described in detail in order to avoid obscuring the present disclosure.

[0027] With the continuous increase of the number of motor vehicles, the incidence of road safety accidents remains high, endangering the safety of drivers, passengers and pedestrians. The application of driver monitoring system (DMS) is becoming more and more widespread. As an important means of active safety, it is popularized on more and more motor vehicles. Especially for engineering and commercial vehicles, the harm caused by frequent safety accidents is greater. By installing DMS on vehicles, these safety accidents can be effectively prevented, and life and property losses can be avoided.

[0028] Behaviors such as smoking and making phone calls are typical violations of dangerous driving behaviors. The detection technology of smoking and making phone calls based on convolutional neural networks (CNN) has become a standard in DMS, which has the advantages of low cost and high accuracy compared with traditional detection methods.

[0029] In related technologies, the method of target detection is generally used to extract cigarette and mobile phone information in images, and then to determine whether there is smoking and making phone call behavior. However, due to the characteristics of few features and small volume of cigarettes, and the fact that phones are easy to be blocked, the accuracy of smoking and making phone call behavior detection needs to be further improved. Especially for some industrial scenes with complex cockpit backgrounds, smoking and making phone call behavior detection is prone to false detection, which greatly interferes with production.

[0030] Therefore, the present disclosure provides a target behavior detection method and device. Unlike the general driver behavior analysis mainly completed by target detection, the present disclosure detects smoking and making phone call behavior based on the method of instance segmentation and human object interaction (HOI), which enhances the robustness of the algorithm and improves the accuracy of smoking and making phone call behavior detection. Especially for complex driving environments such as industrial scenes, the false detection rate can be reduced.

[0031] Of course, the detection method and device of the present disclosure can be applied not only to the detection of the driver's smoking and phone calling behavior in the driving scenario, but also to the detection of the driver's smoking and phone calling behavior in other vehicles in the driving scenario, such as trains, high-speed rails, airplanes, etc. Moreover, the detection method and device of the present disclosure can be applied not only to the detection of the driver's smoking and phone calling behavior, but also to the detection of other behaviors of the driver of the vehicle, such as eating, drinking, holding a phone, watching a mobile phone video and / or browsing mobile phone information, and single-handed or double-handed leaving the steering wheel, etc. In addition, the detection method and device of the present disclosure can be applied not only to the driving scenario, but also to places where smoking is prohibited, such as gas stations, natural gas plants, factories where smoking is prohibited (for example, factories where flammable, explosive, and other dangerous goods are placed), etc. Of course, it can also be applied to other scenarios where the target behavior of the user needs to be monitored.

[0032] Figure 1 FIG. 1 is a flowchart of a processing method of a target behavior according to an embodiment of the present disclosure. In a possible implementation, the processing method can be performed by a DMS carried by a vehicle. As shown in FIG. 1, the processing method can include the following steps. Figure 1

[0033] Step S110 (extraction step), extracting a target image for detecting the target behavior from a to-be-detected image or a to-be-detected video frame.

[0034] In this embodiment, the to-be-detected image is one frame of image in a sequence of image frames, which is a set of a series of images having a chronological order, and can be specifically a plurality of frames of image continuously captured by an image capturing device, such as a plurality of frames of image continuously captured by a camera. The to-be-detected video frame is one frame of image in a sequence of video frames, which is a set of a series of images having a chronological order, and can be specifically a sequence of video frames in a video captured by an image capturing device. A region capable of being used to detect whether a human body has performed a target behavior can be extracted from the image frame or the video frame as a target image. For example, assuming that the processing method is used to detect the driver's smoking behavior, a region covering the driver's face, hands, body and key points thereof, and a cigarette can be extracted from the image frame or the video frame as the target image. For example, assuming that the processing method is used to detect the driver's phone calling behavior, a region covering the driver's face, hands, body and key points thereof, and a phone can be extracted from the image frame or the video frame as the target image. For example, assuming that the processing method is used to detect the driver's single-handed leaving the steering wheel behavior, a region covering the driver's face, hands, body and key points thereof, and a steering wheel can be extracted from the image frame or the video frame as the target image.

[0035] ​In one possible implementation, image frames or video frames acquired by the image acquisition device can be preprocessed, such as by noise reduction, and the target image can be extracted from the preprocessed image frames or video frames. This can improve the accuracy of behavior detection.

[0036] Step S120 (acquisition step): Instance segmentation is performed on the target image to obtain a first segmentation result of the target human body and a second segmentation result of the target object in the target image, wherein the first segmentation result includes the shape information of the target human body and the second segmentation result includes the shape information of the target object.

[0037] In this embodiment, after extracting the target image, instance segmentation can be performed on the target image to obtain instance segmentation results for the target human body and the target object. The instance segmentation results may include, but are not limited to, shape information of the target human body and shape information of the target object, wherein the shape information of the target object is the shape of its edges. In one possible implementation, the first segmentation result further includes the first bounding box position information of the target human body, and the second segmentation result further includes the object bounding box position information of the target object and the classification information of the target object. The object bounding box position information is the position information of the bounding rectangle of the target object, and the classification information is information that indicates what kind of object the target object is, such as the target object being a cigarette, a mobile phone, etc.

[0038] It should be understood that the bounding box of the target object outputs the approximate position of the target object, while the shape information of the target object outputs the precise position of the target object. Therefore, compared with the bounding box of the target object, the shape information of the target object is the more precise position of the target object.

[0039] like Figure 5 As shown, the shape formed by the outline of the cigarette's edge represents the cigarette's shape information. The green rectangle is the circumscribed rectangle of the cigarette. It's clear that the green rectangle, compared to the cigarette's outline, encompasses part of the face and fingers. Similarly, the shape formed by the outline of the phone's edge represents the phone's shape information. The blue rectangle is the circumscribed rectangle of the cigarette. It's clear that the blue rectangle, compared to the phone's outline, encompasses part of the fingers and clothing. Therefore, compared to the bounding box of the target object, the shape information of the target object provides a more precise location.

[0040] In one possible implementation, step S120 may include: inputting the target image into a target model, performing instance segmentation via the instance segmentation branch of the target model, and outputting the first segmentation result and the second segmentation result.

[0041] In this embodiment, the target model is a CNN model, which includes an instance segmentation branch. The instance segmentation branch is based on the principles of yolox+yolact, and the backbone network uses the custom lightweight network structure shown in FIG. 1, such as extracting pixel-level instance segmentation feature information for the body (taking the upper body in the driving environment), cigarettes, and mobile phones. The segmentation branch outputs the first target box position, the second target box position (cigarettes and mobile phones), classification information, and shape information (mask). Figure 6

[0042] As shown in FIG. 1, the instance segmentation is performed via the instance segmentation branch of the CNN model, and the shape information of the cigarette, the bounding rectangle of the cigarette, the shape information of the mobile phone, the bounding rectangle of the mobile phone, and the bounding rectangle of the human body are output. Figure 5

[0043] Step S130 (detection step), based on the feature information fused by the instance segmentation process, the human interaction HOI detection is performed to determine the interaction information of the target human body and the target object.

[0044] In this embodiment, after obtaining the instance segmentation result, the feature information fused by the instance segmentation process can be used to predict the interaction relationship between the target human body corresponding to the shape information included in the first instance segmentation result and the target object corresponding to the shape information included in the second instance segmentation result, such as Figure 5 As shown in FIG. 1, the interaction relationship between the human body and the cigarette, and the interaction relationship between the human body and the mobile phone are predicted.

[0045] In one possible implementation, step S130 can include: using the feature information fused by the instance segmentation process via the prediction branch of the target model to perform HOI detection to obtain interaction information, wherein the interaction information includes the position information of the interaction point between the target human body and the target object, the interaction category of the target human body and the target object, and the position offset of the interaction point to the target human body and the target object.

[0046] In this embodiment, the target model is a CNN model, which further includes an HOI relationship prediction branch. The HOI relationship prediction branch can be implemented based on an improved interaction point prediction algorithm. The position information of the interaction point is the coordinate of the center position of the line connecting the center point of the target human body and the center point of the target object in each human-object pair. This embodiment directly uses the regression of the interaction point coordinates in yolox to achieve the advantages of the point coordinate regression of the yolox framework, while combining the image pixel-level features, which greatly improves the prediction accuracy of the interaction point.

[0047] ​​The embodiment adds an interaction prediction branch on the basis of the yolox framework, including interaction point position prediction, interaction category prediction and interaction vector offset prediction. The interaction point position prediction is directly trained by regressing point coordinates using smooth-L1 loss. The interaction prediction branch is established on the basis of the instance segmentation branch, fully utilizes the global pixel-level features, learns a set of corresponding weight components by fusing the features of, for example, 32 prototype masks in the instance segmentation branch, and then summarizes the features and weights by multiplication. On this basis, a corresponding head is designed to predict the coordinates of the interaction points, the interaction categories and the interaction offsets. The feature information fused in the instance segmentation process is the feature of the 32 prototype masks.

[0048] The overall structural schematic diagram is shown in Figure 4 The input target image is processed by a CNN backbone network to generate a set of features. The last three layers of features (p3, p4, p5) are fused by a feature pyramid network (FPN) to generate a feature pyramid. The internal structure diagrams of the backbone network and the FPN are shown in Figure 6 The c3, c4 and c5 in the internal structure of the FPN are feature maps corresponding to the convolution layers with downsampling multiples of 8, 16 and 32, respectively. The p3, p4 and p5 are the outputs of the above feature maps after FPN operation. On this basis, an instance segmentation head and an interaction relationship head are constructed to generate corresponding prediction branches. Figure 4 The orange module represents the instance segmentation branch, and the gray module represents the interaction relationship prediction branch. In the instance segmentation branch, the mask cofficents are processed by NMS and multiplied by, for example, 32 prototypes generated by the prototype network (protonet), which is the shape information of the target. Then, the final shape information is obtained by cutting and threshold filtering according to the information of the bounding box (bbox).

[0049] In the interaction relationship prediction branch, the predicted shape weight is multiplied by the shape information of the 32 prototypes, and then concatenated with the feature pyramid. On this basis, the interaction category, the interaction point position and the interaction offset are predicted. Finally, four target interaction categories are output: holding a cigarette, smoking, holding a mobile phone and making a phone call. The output results are shown in Figure 5As shown, the class, bbox and shape information of the cigarette (cigaratte) are predicted by the instance segmentation branch, the class, bbox and shape information of the cellphone are predicted by the instance segmentation branch, the interaction center point between the person and the cigarette, the interaction class (smoke) and the position offset from the interaction center point to the person and the cigarette are predicted by the interaction branch, the interaction center point between the person and the cellphone, the interaction class (make a phone call) and the position offset from the interaction center point to the person and the cellphone are predicted by the interaction branch.

[0050] The specific training method of the CNN model is as follows: the data set is labeled by combining artificial and automatic labels (auto-labels), and the position rectangular frame, center point and instance segmentation labeling information of the human body, cigarette and cellphone, and the interaction relationship classification, interaction point position and offset information are generated to form the final data set. When the training starts, the images in the data set are used as input images, the size is adjusted to the input size (640*640) of the model, and after a certain data enhancement (such as random left-right flipping, random Mosaic, etc.) and normalization, the input is input into the CNN model, the corresponding feature map is generated, and the corresponding prediction value is generated through the head. Then, the loss between the prediction value and the labeling GroundTruth is calculated. Specifically, the label assignment is completed by SimOTA, the bbox loss in the instance segmentation branch adopts IOU loss, the target confidence loss and the target classification loss both adopt classification cross-entropy BCE loss, and the mask loss adopts pixel-level classification cross-entropy loss. The interaction point position loss and the interaction vector offset loss in the interaction prediction branch both adopt Smooth-L1 loss, and the interaction class loss adopts classification cross-entropy loss. The final loss is the sum of the products of all the above losses and the assigned weights. The gradient descent algorithm is used to adjust the parameters to optimize the loss. The Mosaic data enhancement is turned off at the last 15 cycles (epochs). The entire training process described above is carried out on the yolox framework.

[0051] As described above, in a possible implementation, using the feature information fused by the instance segmentation process via the prediction branch of the target model for HOI detection to obtain the interaction information can include: multiplying the feature information by the corresponding weight, and performing feature fusion on the product in the manner of a feature pyramid to obtain the position information of the interaction point, the interaction class and the position offset.

[0052] Step S140 (determination step), based on the interaction information, determining whether the target human body has performed the target behavior.

[0053] In this embodiment, whether the target human body has performed the target behavior can be determined according to the position information of the interaction point, the interaction category, and the position offset. As shown in Figure 5 the center point of the cigarette and the body, and the cigarette being in the mouth, the human body is classified as being smoking; and according to the center point of the mobile phone and the body, and the mobile phone being beside the ear, the human body is classified as making a phone call. Thus, it can be determined that the human body is smoking and making a phone call, that is, the human body has performed the smoking behavior and the phone call behavior.

[0054] According to this embodiment, instance segmentation is performed on the target image to obtain respective instance segmentation results of the target human body and the target object in the target image, HOI detection of human interaction is performed based on feature information fused by the instance segmentation process to determine interaction information of the target human body and the target object corresponding to respective shape information included in the instance segmentation results, and whether the target human body has performed a target behavior is determined based on the interaction information.

[0055] Therefore, this embodiment extracts features of instance segmentation pixels of the target human body and the target object, and simultaneously predicts interaction relationships of the target human body and the target object, the instance segmentation task and the interaction relationship prediction task share features extracted by a part of the CNN model, and have the following advantages: lightweight backbone network design, small amount of calculation, short time consumption, and are very suitable for application scenarios such as vehicle-mounted devices with limited computing power; unlike many existing methods based on target detection technology, this disclosure extracts pixel-level semantic information and deeper human interaction semantic information, and is more robust; and is easy to extend to detection of other abnormal driving behaviors such as eating and drinking behaviors; has a low false detection rate while maintaining a certain TPR recall rate (recall / true positive rate).

[0056] Compared with the prior art, obtaining the bounding rectangle of the target object artificially judges the possible existence of target behavior, which may have false detection and does not consider the interaction relationship between the target object and the target human body, this embodiment obtains more accurate shape information of the target object in addition to the rectangle of the target object, thereby increasing the supervision information in the shape dimension, and simultaneously, the interaction prediction branch is also increased, and the supervision information in the interaction relationship dimension is also increased, so that the accuracy of the target behavior can be improved and the false detection rate can be reduced. For example, under a vehicle-mounted system, assuming that the remaining personnel of the vehicle deliver / place a cigarette to the driver's mouth, which is not an abnormal behavior, the shape information of the target human body of this embodiment can be used to exclude this behavior, so that false detection can be avoided.

[0057] In a possible implementation manner, as shown in Figure 2 the extraction step S110 can include:

[0058] Step S111, obtaining a current image frame or a current video frame collected by the image collection device as the to-be-tested image or to-be-tested video frame.

[0059] In this embodiment, the current image frame or the current video frame can be obtained from the image collection device such as a camera as the to-be-tested image or to-be-tested video frame.

[0060] Step S112, obtaining a first target frame corresponding to a human body related part in the to-be-tested image or to-be-tested video frame, the human body related part including a face and a body of the human body.

[0061] In this embodiment, any suitable manner can be used to obtain the first target frame in the to-be-tested image or to-be-tested video frame, for example, a target detection manner can be used to obtain the first target frame in the to-be-tested image or to-be-tested video frame, or a pre-trained model such as a CNN model can be used to obtain the first target frame in the to-be-tested image or to-be-tested video frame. In the case of obtaining the first target frame by using the CNN model, the to-be-tested image or to-be-tested video frame can be input into the pre-trained model such as the CNN model, and the CNN model outputs a face rectangular frame and a body rectangular frame. Of course, a hand rectangular frame and body key point position information can also be output.

[0062] In one possible implementation, face, body and key point detection are performed on the current image frame or the current video frame, and a face rectangular frame position and a body rectangular frame position are output. On the basis of the principle of the yolox detection algorithm framework, a self-defined Figure 6 As shown in the lightweight backbone and head design implementation, the head design is simplified to improve the inference speed, and only one layer of 3*3 convolution layer and one layer of 1*1 convolution layer are used.

[0063] Step S113, determining whether the first target frame contains a face and a body.

[0064] If it is determined in step S113 that the first target frame contains a face and a body, for example, the CNN model outputs a face rectangular frame and a body rectangular frame, it indicates that a person is present at this time, and the target behavior detection can be continued on the current image frame or the current video frame. Therefore, the following step S114 can be performed. Otherwise, for example, the CNN model does not output a face rectangular frame and a body rectangular frame, it indicates that no one is present at this time, and the target behavior detection does not need to be continued on the current image frame or the current video frame. Therefore, the following step S115 can be performed. That is, whether the human body such as the driver is off the seat is determined according to whether the face and the body are contained in the image frame or the video frame. If the driver is off the seat, the next image frame or the next video frame is returned to step S111 for detection.

[0065] In step S114, the target image is extracted from the to-be-tested image or the to-be-tested video frame according to the position information of the first target frame.

[0066] In this embodiment, if valid face and body information is detected, the first target frame can be expanded by a predetermined proportion to obtain a second target frame, and the target image can be cropped from the to-be-tested image or the to-be-tested video frame along the second target frame. That is, the first target frame region can be expanded by a certain proportion, the expanded region can be used as the target image, and the target image can be cropped from the to-be-tested image or the to-be-tested video frame.

[0067] Therefore, compared with directly using the to-be-tested image or the to-be-tested video frame for target behavior detection, using the target image cropped from the to-be-tested image or the to-be-tested video frame for target behavior detection can reduce the image range for target behavior detection, exclude a part of irrelevant background information, and thus can reduce the required computing power for target behavior detection and avoid excessive computing power consumption. In addition, compared with directly using the first target frame for target behavior detection, using the second target frame obtained by expanding the first target frame outward for target behavior detection can ensure that the target object such as a hand, a cigarette, a mobile phone, or the like is within the target image.

[0068] In a possible implementation, the predetermined proportion can be dynamically adjusted according to the computing power of a device that executes the processing method.

[0069] In this embodiment, the smaller the computing power of the device that executes the processing method of the present disclosure is, the smaller the predetermined proportion is under the premise that the target image can be used to detect whether the target human body performs the target behavior. Therefore, it can be avoided that the required computing power is excessively large due to the complex or excessively large background, and thus it can be avoided that a system such as a vehicle-mounted system with small computing power cannot support excessively large computing power.

[0070] In a possible implementation, expanding the first target frame outward by a predetermined proportion to obtain a second target frame includes: expanding the first target frame upward and downward by a first value and expanding the first target frame leftward and rightward by a second value to obtain the second target frame, wherein the width of the second target frame is longer than the width of the first target frame by two times the first value, and the height of the second target frame is longer than the height of the first target frame by two times the second value. For example, the first value and the second value are both 0.1.

[0071] In this embodiment, as Figure 3As shown, the first target frame of blue has a width of w and a height of h, and the first target frame of blue can be expanded by 0.1h upward and downward and by 0.1w left and right, so that a red rectangular frame is obtained by expanding the first target frame, and the to-be-tested image or the to-be-tested video frame can be cropped according to the position information of the red rectangular frame, and then a target image (also referred to as a target to-be-tested region) can be obtained, which is the region shown by the red dashed frame in Figure 3 It should be understood that if the expansion has reached the edge (boundary) of the to-be-tested image or the to-be-tested video frame in a certain direction, the expansion in the direction can be stopped.

[0072] In step S115, the next image frame or the next video frame collected by the image collection device is obtained as the to-be-tested image.

[0073] In this embodiment, since the target behavior detection is not required for the current image frame or the current video frame in the unmanned state, the target behavior detection can be performed on the next image frame or the next video frame, so that the next image frame or the next video frame can be re-obtained as the to-be-tested image, and steps S111-S115 can be re-executed until the face and the body are included in the first target frame.

[0074] In a possible implementation manner, as shown in Figure 7 The processing method can include steps S710-S750. Steps S710, S720, and S750 can refer to the foregoing description of steps S110, S120, and S140, and will not be described here. In step S730, according to the shape information of the target object, the segmentation result in which the similarity between the target object and the standard object is lower than a threshold is filtered from the second segmentation result.

[0075] In this embodiment, the image predicted to have four interaction categories can be further filtered, and the filtering is specifically implemented as follows: The shape information of the target object generated by the instance segmentation branch is matched with a group of standard object shape information, and the halcon shape matching algorithm is used to filter out low-similarity instance samples.

[0076] Correspondingly, as shown in Figure 7 In step S740, based on the feature information fused with the feature information generated by the instance segmentation process, the human interaction HOI detection is performed to determine the interaction information of the target object corresponding to the shape information included in the filtered second segmentation result and the target human body. That is, the HOI detection can be performed based on the filtered instance segmentation result.

[0077] In a possible implementation, after the determining step S140 or S750, the processing method further includes: after the processing method is performed on the current image frame or the current video frame, continuing to perform the processing method on the continuous M image frames or the continuous M video frames including the current image frame or the current video frame; if the target human body is detected to perform the target behavior from N image frames or N video frames in the M image frames or the M video frames, and M is greater than or equal to N, performing alarm processing for the target behavior; if M is less than N, not performing alarm processing for the target behavior. N is determined according to the frame transmission number per second of the device performing the processing method.

[0078] In this embodiment, considering that the images of some frames are unstable, such as image blurring / shaking, in order to exclude false detection, smoothing processing can be used. That is, smoothing processing can be performed on the filtered result, and if N frames in the continuous M image frames or video frames detect the corresponding target behavior, and M is greater than or equal to N, alarm for this type of target behavior is performed, where N can be dynamically adjusted according to the overall frame transmission number per second (Frames Per Second, FPS). For example, if 5 frames in 10 consecutive frames detect the target behavior, alarm can be performed; if the computing power of the device is high, which can be 3-4 fps, N can be set to be larger.

[0079] Therefore, with the aid of smoothing processing, the embodiment can further improve the accuracy of behavior detection, and can further reduce the occurrence of false detection, which is particularly advantageous in industrial vehicle driving with a complex cockpit background.

[0080] In a possible implementation, after the determining step S140 or S750, the processing method further includes: if the tolerance for the target behavior is lower than a threshold, performing alarm processing in response to the determination that the target human body performs the target behavior in the determining step; if the tolerance for the target behavior is not lower than the threshold, performing alarm processing in response to the determination that the target human body performs the target behavior for a continuous predetermined number of times.

[0081] If the requirement for the target behavior is strict, that is, the tolerance is low, such as not tolerating smoking, alarm processing is performed as long as smoking behavior is detected in one image frame or video frame. If the requirement for the target behavior is relatively strict, that is, the tolerance is relatively low, such as slightly tolerating smoking, alarm processing is performed after smoking behavior is detected in 2 or 3 continuous image frames or video frames.

[0082] In a possible implementation, the alarm processing mode includes at least one of a pop-up window alarm, an audio alarm, a video alarm, an SMS alarm, a WeChat alarm, and an email alarm. Of course, any other suitable alarm mode can also be used.

[0083] Figure 8 is a block diagram of a processing apparatus of a target behavior according to an embodiment of the present disclosure. As shown in Figure 8 the processing apparatus 800 can include an extraction module 810, an acquisition module 820, a detection module 830, and a determination module 840. The extraction module 810 is configured to extract a target image for detecting the target behavior from a to-be-detected image. The acquisition module 820 is connected with the extraction module 810 and configured to perform instance segmentation on the target image to acquire a first segmentation result of a target human body and a second segmentation result of a target object in the target image, where the first segmentation result includes shape information of the target human body, and the second segmentation result includes shape information of the target object. The detection module 830 is connected with the acquisition module 820 and configured to perform human interaction HOI detection based on feature information fused from an instance segmentation process to determine interaction information of the target human body and the target object. The determination module 840 is connected with the detection module 830 and configured to determine whether the target human body has performed the target behavior based on the interaction information.

[0084] In a possible implementation, the extraction module 810 is configured to acquire a current image frame or a current video frame collected by an image collection apparatus as the to-be-detected image or to-be-detected video frame, acquire a first target box corresponding to a human body related part in the to-be-detected image or to-be-detected video frame, where the human body related part includes a face and a body of the human body, and extract the target image from the to-be-detected image or to-be-detected video frame according to position information of the first target box.

[0085] In a possible implementation, the extraction module 810 is configured to determine whether the first target box contains a face and a body, extract the target image from the to-be-detected image or to-be-detected video frame according to position information of the first target box if it is determined that the first target box contains a face and a body, and acquire a next image frame or a next video frame collected by the image collection apparatus as the to-be-detected image if it is determined that the first target box does not contain a face and a body.

[0086] In a possible implementation, the extraction module 810 is configured to expand the first target box outward by a predetermined proportion to obtain a second target box, and crop the target image from the to-be-detected image along the second target box.

[0087] In a possible implementation, the extraction module 810 is configured to: expand the first target frame upward and downward by a first value and expand the first target frame leftward and rightward by a second value to obtain the second target frame, where a width of the second target frame is longer than a width of the first target frame by twice the first value, and a height of the second target frame is longer than a height of the first target frame by twice the second value.

[0088] In a possible implementation, the acquisition module 820 is configured to: input the target image into a target model, perform instance segmentation via an instance segmentation branch of the target model, and output the first segmentation result and the second segmentation result.

[0089] In a possible implementation, the detection module 830 is configured to: perform HOI detection using the feature information via a prediction branch of the target model to obtain interaction information, where the interaction information includes position information of an interaction point between the target human body and the target object, an interaction category of the target human body and the target object, and a position offset of the interaction point to the target human body and the target object, and accordingly, the determination module 840 is configured to: determine whether the target human body performs the target behavior according to the position information of the interaction point, the interaction category, and the position offset.

[0090] In a possible implementation, the detection module 830 is configured to: multiply the feature information by a corresponding weight, and perform feature fusion on the product in a feature pyramid manner to obtain the position information of the interaction point, the interaction category, and the position offset.

[0091] In a possible implementation, the processing apparatus 800 further includes a filtering module (not shown) configured to: filter, according to shape information of the target object, segmentation results in which a similarity between the target object and a standard object is lower than a threshold from the second segmentation result, and accordingly, the detection module 830 is configured to: perform HOI detection based on the feature information to determine interaction information of the target human body and a target object corresponding to shape information included in the filtered second segmentation result.

[0092] In a possible implementation, the processing apparatus further comprises an alarm module (not shown), wherein the processing apparatus performs processing on continuous M image frames or continuous M video frames including the current image frame or the current video frame in sequence; the target human body is detected to perform the target behavior from N image frames or N video frames in the M image frames or the M video frames, wherein if M is greater than or equal to N, the alarm module performs alarm processing for the target behavior; if M is less than N, the alarm module does not perform alarm processing for the target behavior.

[0093] In a possible implementation, the processing apparatus 800 further comprises an alarm module (not shown), configured to perform alarm processing in response to the determination module determining that the target human body performs the target behavior if the tolerance for the target behavior is lower than a threshold; and perform alarm processing in response to the determination module determining that the target human body performs the target behavior for a predetermined number of times consecutively if the tolerance for the target behavior is not lower than the threshold.

[0094] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules for performing the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0095] The embodiments of the present disclosure also provide a computer-readable storage medium having computer program instructions stored therein, and the computer program instructions are executed by a processor to implement the above processing method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0096] The embodiments of the present disclosure also provide a target behavior processing apparatus, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above processing method when executing the instructions stored in the memory.

[0097] The embodiments of the present disclosure also provide a computer program product, comprising computer readable code or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in a processor of an electronic device, the processor in the electronic device executes the above processing method.

[0098] Figure 9 Fig. 19 is a block diagram of a target behavior processing apparatus 1900 according to an exemplary embodiment. For example, the apparatus 1900 can be provided as a terminal device, such as a vehicle-mounted terminal device. Referring to Fig. 19, the apparatus 1900 can include a processor 1901, a memory 1902, and a communication interface 1903. Figure 9The apparatus 1900 includes a processing component 1922, which is further composed of one or more processors, and a memory resource represented by the memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-mentioned processing method.

[0099] The apparatus 1900 can further include a power supply component 1926 configured to perform power management of the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input / output interface 1958 (I / O interface). The apparatus 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0100] In an exemplary embodiment, a non-transitory computer readable storage medium, such as the memory 1932 including computer program instructions, is also provided, which can be executed by the processing component 1922 of the apparatus 1900 to complete the above-mentioned processing method.

[0101] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0102] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0103] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0104] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0105] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0106] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0107] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0108] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0109] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are within the scope of the disclosure. The selection of terms is intended to best describe the principles of the embodiments, practical application, or technical improvements in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing target behavior, characterized in that, include: The extraction step involves extracting a target image from the image to be tested or the video frame to be tested for detecting the target behavior. The acquisition step involves performing instance segmentation on the target image to obtain a first segmentation result of the target human body and a second segmentation result of the target object in the target image. The first segmentation result includes the shape information of the target human body, and the second segmentation result includes the shape information of the target object, wherein the shape information of the target object is the shape of the edge of the target object. The detection step involves performing Human Interaction HOI detection based on the feature information generated by the instance segmentation process to determine the interaction information between the target human body corresponding to the shape information included in the first segmentation result and the target object corresponding to the shape information included in the second segmentation result. The determination step involves, based on the interaction information, determining whether the target human body has performed the target behavior. The detection step includes: multiplying the feature information with the corresponding weights, and performing feature fusion on the product using a feature pyramid method to determine the interaction information.

2. The processing method according to claim 1, characterized in that, The extraction steps include: The current image frame or current video frame acquired by the image acquisition device is used as the image or video frame to be tested; Obtain a first target bounding box corresponding to a human body part in the image or video frame to be tested, wherein the human body part includes the face and body of the human body; Based on the position information of the first target bounding box, the target image is extracted from the image to be tested or the video frame to be tested.

3. The processing method according to claim 2, characterized in that, After obtaining the first target bounding box, the extraction step further includes: Determine whether the first target box contains a face and a body; If it is determined that the first target box contains a face and a body, then the target image is extracted from the image to be tested or the video frame to be tested based on the position information of the first target box. If it is determined that the first target frame does not contain a face and body, then the next image frame or the next video frame acquired by the image acquisition device is obtained as the image to be tested or the video frame to be tested.

4. The processing method according to claim 2, characterized in that, Extracting the target image from the image to be tested or the video frame to be tested based on the position information of the first target bounding box includes: The first target bounding box is expanded outward by a predetermined ratio to obtain a second target bounding box. The image to be tested or the video frame to be tested is cropped along the second target bounding box, and the cropped image is the target image.

5. The processing method according to claim 4, characterized in that, The predetermined ratio can be dynamically adjusted according to the computing power of the device executing the processing method.

6. The processing method according to claim 4, characterized in that, The second target box is obtained by expanding the first target box outward by a predetermined ratio, including: The first target box is expanded vertically by a first value and horizontally by a second value to obtain the second target box. The width of the second target box is twice the width of the first target box by the first value, and the height of the second target box is twice the height of the first target box by the second value.

7. The processing method according to claim 6, characterized in that, Both the first value and the second value are 0.

1.

8. The processing method according to claim 1, characterized in that, The acquisition steps include: The target image is input into the target model, and instance segmentation is performed through the instance segmentation branch of the target model to output the first segmentation result and the second segmentation result.

9. The processing method according to claim 8, characterized in that, The first segmentation result also includes the first target bounding box position information of the target human body, and the second segmentation result also includes the object bounding box position information of the target object and the classification information of the target object.

10. The processing method according to claim 8, characterized in that, The detection step includes: performing HOI detection using the feature information via the prediction branch of the target model to obtain interaction information, wherein the interaction information includes the positional information of the interaction points between the target human body and the target object, the interaction category between the target human body and the target object, and the positional offset of the interaction points from the target human body and the target object. Accordingly, The determination step includes: determining whether the target human body has performed the target behavior based on the location information of the interaction point, the interaction category, and the location offset.

11. The processing method according to claim 10, characterized in that, Using the feature information to perform HOI detection via the prediction branch of the target model to obtain interaction information includes: The feature information is multiplied by the corresponding weight, and the product is fused using a feature pyramid method to obtain the location information of the interaction point, the interaction category, and the location offset.

12. The processing method according to claim 10, characterized in that, The location information of the interaction point is the coordinates of the center position of the line connecting the center point of the target human body and the center point of the target object for each character.

13. The processing method according to claim 1, characterized in that, Prior to the detection step, the processing method further includes: filtering out segmentation results from the second segmentation results where the similarity between the target object and the standard object is lower than a threshold, based on the shape information of the target object; accordingly, The detection step includes: performing HOI detection based on the feature information to determine the interaction information of the target human body and the target object corresponding to the shape information included in the filtered second segmentation result.

14. The processing method according to claim 13, characterized in that, Following the determination step, the processing method further includes: After the processing method is executed on the current image frame or the current video frame, the processing method is executed sequentially on M consecutive image frames or M consecutive video frames, including the current image frame or the current video frame. Detecting the target human body performing the target behavior from N image frames or N video frames out of the M image frames or M video frames. If M is greater than or equal to N, then an alarm is triggered for the target behavior. If M is less than N, no alarm will be triggered for the target behavior.

15. The processing method according to claim 14, characterized in that, N is determined based on the number of frames transmitted per second by the device performing the processing method.

16. The processing method according to claim 1, characterized in that, Following the determination step, the processing method further includes: If the tolerance for the target behavior is below a threshold, an alarm is triggered in response to the determination step in which the target human body is found to have performed the target behavior. If the tolerance for the target behavior is not lower than a threshold, an alarm is triggered in response to the determination that the target human body has performed the target behavior a predetermined number of times.

17. The processing method according to any one of claims 14-16, characterized in that, The alarm handling includes at least one of the following: pop-up alarms, audio alarms, video alarms, SMS alarms, WeChat alarms, and email alarms.

18. A processing device for a target behavior, characterized in that, include: The extraction module is used to extract a target image from the image to be tested or the video frame to be tested for detecting the target behavior; The acquisition module is used to perform instance segmentation on the target image to obtain a first segmentation result of the target human body and a second segmentation result of the target object in the target image, wherein the first segmentation result includes the shape information of the target human body and the second segmentation result includes the shape information of the target object, and the shape information of the target object is the shape of the edge of the target object; The detection module is used to perform human interaction HOI detection based on the feature information generated by the instance segmentation process to determine the interaction information between the target human body corresponding to the shape information included in the first segmentation result and the target object corresponding to the shape information included in the second segmentation result. The determination module is used to determine, based on the interaction information, whether the target human body has performed the target behavior. The detection module multiplies the feature information with the corresponding weights and performs feature fusion on the product using a feature pyramid to determine the interaction information.

19. A processing device for a target behavior, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the processing method of any one of claims 1 to 17 when executing instructions stored in the memory.

20. A vehicle equipped with a driver monitoring system (DMS), characterized in that, The DMS is equipped with the processing device according to claim 18 or 19.

21. The vehicle according to claim 20, characterized in that, The target behavior includes at least one of the following: the driver smoking, the driver drinking water, the driver making a phone call, or the driver eating.

22. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the processing method according to any one of claims 1 to 17.

Images