Target behavior processing method and device, carrying tool and storage medium

Abstract

The invention relates to a target behavior processing method and device, a carrying tool and a storage medium, and the method comprises the steps: an extraction step: extracting a target image used for detecting a target behavior from a to-be-detected image or a to-be-detected video frame; an obtaining step: performing instance segmentation on the target image to obtain a first segmentation result of a target human body and a second segmentation result of a target object in the target image, the first segmentation result including shape information of the target human body, and the second segmentation result including shape information of the target object; a detection step: based on the feature information generated by fusing the instance segmentation process, performing character interaction HOI detection to determine the interaction information of the target human body and the target object; and a determination step of determining whether the target human body performs the target behavior based on the interaction information. Therefore, the accuracy of target behavior detection can be improved, and the false detection rate of the target behavior can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method and device for processing target behavior, a carrier, and a storage medium. Background Art

[0002] In the scenario of driver behavior detection, the currently commonly used method is to extract information such as cigarettes in the image through target detection methods, and then determine whether there is smoking behavior.

[0003] However, since the target detection method obtains fewer features of cigarettes and the cigarettes are small in size, the detection accuracy of behaviors such as smoking is poor. What's worse, in driving scenarios such as industrial vehicles with complex cockpit backgrounds, target detection is more prone to misdetection. Summary of the invention

[0004] In view of this, the present disclosure proposes a method and device for processing target behavior, a carrier and a storage medium, so as to improve the accuracy of target behavior detection and reduce the false detection rate of target behavior.

[0005] According to a first aspect of the present disclosure, a target behavior processing method is provided, comprising: an extraction step of extracting a target image for detecting the target behavior from an image to be tested or a video frame to be tested; an acquisition step of performing instance segmentation on the target image to obtain a first segmentation result of a target human body and a second segmentation result of a target object in the target image, wherein the first segmentation result includes shape information of the target human body, and the second segmentation result includes shape information of the target object; a detection step of performing human interaction HOI detection based on feature information generated by an instance segmentation process to determine interaction information between the target human body and the target object; and a determination step of determining whether the target human body has performed the target behavior based on the interaction information.

[0006] According to a second aspect of the present disclosure, a device for processing a target behavior is provided, comprising: an extraction module, used to extract a target image for detecting the target behavior from an image to be tested or a video frame to be tested; an acquisition module, used to perform instance segmentation on the target image to obtain a first segmentation result of a target human body and a second segmentation result of a target object in the target image, wherein the first segmentation result includes shape information of the target human body, and the second segmentation result includes shape information of the target object; a detection module, used to perform human interaction HOI detection based on feature information generated by an instance segmentation process to determine interaction information between the target human body and the target object; and a determination module, used to determine whether the target human body has performed the target behavior based on the interaction information.

[0007] According to a third aspect of the present disclosure, a target behavior processing device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above-mentioned processing method when executing the instructions stored in the memory.

[0008] According to a fourth aspect of the present disclosure, there is provided a vehicle, which is equipped with a driver monitoring system DMS, and the DMS is equipped with the processing device according to the second aspect or the third aspect.

[0009] According to a fifth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above-mentioned processing method when executed by a processor.

[0010] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned processing method.

[0011] Through the above-mentioned target behavior processing method and device, carrier and storage medium, the target image is instance segmented to obtain the instance segmentation results of the target human body and the target object in the target image, and the human interaction HOI detection is performed based on the feature information generated by the instance segmentation process to determine the interaction information of the target human body and the target object corresponding to the shape information included in these instance segmentation results, and whether the target human body performs the target behavior is determined based on the interaction information. Therefore, the accuracy of target behavior detection can be improved, and the false detection rate of target behavior can be reduced.

[0012] Among them, pixel-level features of target human and target object instance segmentation are extracted, and the interaction relationship prediction of the target human and target object is performed at the same time. The instance segmentation task and the interaction relationship prediction task share some features extracted by the CNN model, which has the following advantages: lightweight backbone network design, small calculation amount, short time consumption, and is very suitable for application scenarios with limited computing power such as vehicle-mounted equipment; different from many existing methods based on target detection technology, the present invention extracts pixel-level semantic information and deeper human interaction semantic information, which is more robust; and it is easy to expand to other abnormal driving behavior detection such as eating, drinking and other behaviors; while maintaining a certain TPR recall rate (recall / true positive rate), the false detection rate is low, and with the help of a filtering module, the occurrence of false detection is further reduced, which is particularly advantageous in industrial vehicle driving with a complex cockpit background.

[0013] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0015] Figure 1 is a flowchart of a method for processing a target behavior according to an embodiment of the present disclosure.

[0016] Figure 2 is a flowchart of an extraction step according to an embodiment of the present disclosure.

[0017] Figure 3 is a schematic diagram of a method for cropping a target image according to an embodiment of the present disclosure.

[0018] Figure 4 It is a schematic diagram of the principle of performing instance segmentation and HOI detection according to an embodiment of the present disclosure.

[0019] Figure 5 is a schematic diagram of an example result of smoking and phone call behavior detection according to an embodiment of the present disclosure.

[0020] Figure 6 It is a schematic diagram of the internal structure of a backbone network and a feature pyramid network according to an embodiment of the present disclosure.

[0021] Figure 7 is a flowchart of a method for processing a target behavior according to an embodiment of the present disclosure.

[0022] Figure 8 It is a block diagram of a target behavior processing device according to an embodiment of the present disclosure.

[0023] Fig. 9 It is a block diagram of a target behavior processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0025] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0026] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0027] With the continuous increase in the number of motor vehicles, the incidence of road safety accidents remains high, endangering the safety of drivers, passengers and pedestrians. The application of Driver Monitoring System (DMS) is becoming more and more widespread. As an important means of active safety, it has been popularized in more and more motor vehicles. Especially for engineering and commercial vehicles, the frequent occurrence of safety accidents causes greater harm. By installing DMS on the vehicle, these safety accidents can be effectively prevented from happening, avoiding loss of life and property.

[0028] Behaviors such as smoking and making phone calls are typical violations of dangerous driving behavior. The detection technology for smoking and making phone calls based on convolutional neural networks (CNN) has become standard in DMS. Compared with traditional detection methods, it has the advantages of low cost and high accuracy.

[0029] In the related art, the target detection method is generally used to extract the information of cigarettes and mobile phones in the image, and then determine whether there is smoking and calling behavior. However, due to the characteristics of cigarettes such as few features and small size, and the easy obstruction of phones, the accuracy of smoking and calling behavior detection needs to be further improved. Especially for some industrial scenes with complex cockpit backgrounds, smoking and calling behavior detection is prone to false detection, causing great interference to production.

[0030] To this end, the present invention provides a target behavior detection method and device. Different from the general driver behavior analysis mainly completed by target detection, the present invention detects smoking and making phone calls based on instance segmentation and human object interaction (HOI) methods, which enhances the robustness of the algorithm and improves the accuracy of smoking and making phone calls detection, especially for complex driving environments such as industrial scenarios, and can reduce the false detection rate.

[0031] Of course, the detection method and device disclosed in the present invention can be applied not only to the detection of smoking and phone calls by drivers of vehicles in driving scenarios, but also to the detection of smoking and phone calls by drivers of other vehicles in driving scenarios, such as vehicles such as trains, high-speed trains, and airplanes. Moreover, in addition to being applied to the detection of smoking and phone calls by drivers of vehicles, the detection method and device disclosed in the present invention can also be applied to other behavior detections of drivers of vehicles, such as eating, drinking, holding a phone, watching mobile phone videos and / or browsing mobile phone information, leaving the steering wheel with one hand or both hands, etc. In addition, in addition to being applied to driving scenarios, the detection method and device disclosed in the present invention can also be applied to places such as non-smoking areas, such as gas stations, natural gas plants, factories where smoking is prohibited (for example, factories where dangerous goods such as flammable and explosive items are placed), etc., and of course, it should also be applied to other scenarios where the user's target behavior needs to be monitored.

[0032] Figure 1 FIG. 1 is a flow chart of a method for processing a target behavior according to an embodiment of the present disclosure. In a possible implementation, the processing method may be executed by a DMS carried by a vehicle. Figure 1 As shown, the processing method may include the following steps.

[0033] Step S110 (extraction step), extracting a target image for detecting the target behavior from the image to be tested or the video frame to be tested.

[0034] In this embodiment, the image to be tested is one of the frames in the image frame sequence, and the image frame sequence is a collection of a series of images with a temporal order, and can specifically be multiple frames of images continuously collected by an image acquisition device, such as multiple frames of images continuously collected by a camera. The video frame to be tested is one of the frames in the video frame sequence, and the video frame sequence is a collection of a series of images with a temporal order, and can specifically be a video frame sequence in a video collected by an image acquisition device. An area that can be used to detect whether a human body has performed a target behavior can be extracted from the image frame or video frame as a target image. For example, assuming that the processing method is used to detect the driver's smoking behavior, the area covering the driver's face, hands, body and key points, and cigarettes can be extracted from the image frame or video frame as a target image. For example, assuming that the processing method is used to detect the driver's phone call behavior, the area covering the driver's face, hands, body and key points, and the phone can be extracted from the image frame or video frame as a target image. For another example, assuming that the processing method is used to detect the driver's behavior of leaving the steering wheel with one hand, the area covering the driver's face, hands, body and key points, and the steering wheel can be extracted from the image frame or video frame as a target image.

[0035] In a possible implementation, the image frames or video frames collected by the image acquisition device may be preprocessed such as noise reduction, and the target image may be extracted from the preprocessed image frames or video frames, thereby improving the accuracy of behavior detection.

[0036] Step S120 (acquisition step), performing instance segmentation on the target image to obtain a first segmentation result of a target human body and a second segmentation result of a target object in the target image, wherein the first segmentation result includes shape information of the target human body, and the second segmentation result includes shape information of the target object.

[0037] In this embodiment, after the target image is extracted, the target image can be instance segmented to obtain instance segmentation results of the target human body and the target object. The instance segmentation result may include but is not limited to the shape information of the target human body and the shape information of the target object, wherein the shape information of the target object is the shape of the edge of the target object. In a possible implementation, the first segmentation result also includes the first target frame position information of the target human body, and the second segmentation result also includes the object frame position information of the target object and the classification information of the target object. Among them, the object frame position information is the position information of the circumscribed rectangular frame of the target object, and the classification information is information that can indicate what the target object is, such as the target object is a cigarette, a mobile phone, etc.

[0038] It should be understood that the rectangular frame of the target object outputs the rough position of the target object, while the shape information of the target object outputs the precise position of the target object. Therefore, compared with the circumscribed rectangular frame of the target object, the shape information of the target object is a more precise position of the target object.

[0039] like Figure 5 As shown in the figure, the shape formed by the outline of the cigarette edge is the shape information of the cigarette, and the green rectangular frame is the circumscribed rectangular frame of the cigarette. It is obvious that the green rectangular frame covers part of the face and fingers compared with the outline of the cigarette edge. The shape formed by the outline of the mobile phone edge is the shape information of the mobile phone, and the blue rectangular frame is the circumscribed rectangular frame of the cigarette. It is obvious that the blue rectangular frame covers part of the fingers and clothes compared with the outline of the mobile phone edge. Therefore, compared with the rectangular frame of the target object, the shape information of the target object is a more accurate position of the target object.

[0040] In a possible implementation, step S120 may include: inputting the target image into a target model, performing instance segmentation via an instance segmentation branch of the target model, and outputting the first segmentation result and the second segmentation result.

[0041] In this embodiment, the target model is a CNN model, which includes an instance segmentation branch. The instance segmentation branch is based on the yolox+yolact principle, and the backbone network adopts Figure 6 The custom lightweight network structure shown, for example, extracts pixel-level instance segmentation feature information for the body (upper body in a driving environment), cigarettes, and mobile phones. The segmentation branch outputs the first target box position, the second target box position (cigarettes and mobile phones), classification information, and shape information (mask).

[0042] like Figure 5 As shown, instance segmentation is performed through the instance segmentation branch of the CNN model and outputs the shape information of the cigarette, the circumscribed rectangular frame of the cigarette, the shape information of the mobile phone, the circumscribed rectangular frame of the mobile phone, and the circumscribed rectangular frame of the human body.

[0043] Step S130 (detection step), based on the feature information generated by the instance segmentation process, human interaction HOI detection is performed to determine the interaction information between the target human body and the target object.

[0044] In this embodiment, after obtaining the instance segmentation result, the feature information generated by the fusion instance segmentation process can be used to predict the interaction relationship between the target human body corresponding to the shape information included in the first instance segmentation result and the target object corresponding to the shape information included in the second instance segmentation result, such as Figure 5 As shown, the interaction relationship between the human body and cigarettes, and the interaction relationship between the human body and mobile phones are predicted.

[0045] In one possible implementation, step S130 may include: performing HOI detection via the prediction branch of the target model using feature information generated by an instance segmentation process to obtain interaction information, wherein the interaction information includes position information of an interaction point between the target human body and the target object, an interaction category between the target human body and the target object, and a position offset from the interaction point to the target human body and the target object.

[0046] In this embodiment, the target model is a CNN model, which also includes a HOI relationship prediction branch. The HOI relationship prediction branch can be implemented based on an improved interaction point prediction algorithm, and the position information of the interaction point is the coordinates of the center position of the line connecting the center point of the target human body and the center point of the target object in each character. This embodiment directly uses the regression of the interaction point coordinates in yolox to achieve this, taking advantage of the yolox framework for point coordinate regression, while combining the image pixel-level features, greatly improving the prediction accuracy of the interaction point.

[0047] This embodiment adds an interaction prediction branch based on the yolox framework, including the position prediction of the interaction point, the interaction category prediction and the interaction vector offset prediction, wherein the interaction point position prediction is directly trained by regressing the point coordinates using smooth-L1 loss. The interaction prediction branch is based on the instance segmentation branch, making full use of the global pixel-level features. By fusing the features of, for example, 32 prototype masks (prototype shapes) in the instance segmentation branch, a set of corresponding weight components is learned, and then the features and weights are summarized by multiplication. On this basis, the corresponding head is designed to predict the coordinates of the interaction point, the interaction category and the interaction offset. Among them, the feature information generated by the instance segmentation process is fused as the features of the 32 prototype masks.

[0048] The overall structural principle diagram is as follows Figure 4 As shown in the figure, the input target image is passed through the CNN backbone network to generate a set of features, and the last three layers of features (p3, p4, p5) are fused by the Feature Pyramid Networks (FPN) to generate a feature pyramid. The internal structure of the backbone network and FPN is shown in the figure. Figure 6 As shown in the figure, c3, c4, and c5 in the internal structure of FPN are the feature maps corresponding to the downsampling multiples of 8, 16, and 32 in the convolution layer, respectively. p3, p4, and p5 are the outputs of the above feature maps after the FPN operation. On this basis, the corresponding prediction branches are generated by constructing the instance segmentation head and the interaction relationship head. Figure 4 The orange module in the figure represents the instance segmentation branch, and the gray module represents the interaction relationship prediction branch. The mask cofficents in the instance segmentation branch are processed by NMS and multiplied by, for example, 32 prototypes generated by the prototype network to obtain the shape information of the target. The final shape information is obtained by cropping and threshold filtering based on the bounding box (bbox) information.

[0049] In the interaction relationship prediction branch, the predicted shape weight is multiplied by the 32 prototype shape information, and then fused with the feature pyramid (concat), on this basis, the interaction category, interaction point location and interaction offset are predicted. Finally, four target interaction categories are output: holding a cigarette, smoking, holding a mobile phone, and making a phone call. The output results are as follows: Figure 5As shown in the figure, after prediction by the CNN model, the instance segmentation branch predicts the category, bbox and shape information of cigarettes, the category, bbox and shape information of mobile phones, and the interaction branch predicts the interaction center point between people and cigarettes, the interaction category is smoking, and the position offset from the interaction center point to people and cigarettes, the interaction center point between people and mobile phones, the interaction category (making a phone call), and the position offset from the interaction center point to people and mobile phones.

[0050] The specific training method of the CNN model is as follows: the dataset is annotated by combining manual and automatic labeling (auto-label) to generate the location rectangles, center points and instance segmentation annotation information of human bodies, cigarettes, and mobile phones, as well as the interaction relationship classification, interaction point location and offset information to form the final dataset. At the beginning of training, the images in the dataset are used as input images, and the size (resize) is adjusted to the input size of the model (640*640). After certain data enhancement (such as random left-right flipping, random Mosaic, etc.) and normalization, they are input into the CNN model to generate the corresponding feature map and the corresponding prediction value generated by the head, and then the loss (loss) between the prediction value and the annotation GorundTruth is calculated. Specifically, the label assignment is completed by SimOTA. The bbox loss in the instance segmentation branch uses IOU loss, the target confidence loss and the target classification loss both use classification cross entropy BCE loss, and the mask loss uses pixel-level classification cross entropy loss. The interaction point position loss and interaction vector offset loss in the interaction prediction branch both use Smooth-L1 loss, and the interaction category loss uses classification cross entropy loss. The final loss is the sum of all the above losses multiplied by the assigned weights. The parameters are adjusted through the gradient descent algorithm to achieve the purpose of optimizing the loss. Mosaic data enhancement is turned off in the last 15 cycles (epochs). The entire training process described above is carried out on the yolox framework.

[0051] As described above, in one possible implementation, performing HOI detection via the prediction branch of the target model using feature information generated by an instance segmentation process to obtain interaction information may include: multiplying the feature information with corresponding weights, and performing feature fusion on the product by means of a feature pyramid to obtain the position information of the interaction point, the interaction category, and the position offset.

[0052] Step S140 (determination step), based on the interaction information, determining whether the target human body has performed the target behavior.

[0053] In this embodiment, it can be determined whether the target human body has performed the target behavior based on the position information of the interaction point, the interaction category and the position offset. Figure 5 As shown, the person can be classified as smoking according to the center point of the cigarette and the body, and the cigarette is already in the mouth, and the person can be classified as making a phone call according to the center point of the mobile phone and the body, and the mobile phone is next to the ear. Thus, it can be determined that the person is smoking and making a phone call, that is, the person is smoking and making a phone call.

[0054] According to this embodiment, instance segmentation is performed on the target image to obtain instance segmentation results of the target human body and the target object in the target image, and human interaction HOI detection is performed based on the feature information generated by the instance segmentation process to determine the interaction information of the target human body and the target object corresponding to the shape information included in these instance segmentation results, and based on the interaction information, it is determined whether the target human body has performed the target behavior.

[0055] Therefore, this embodiment extracts pixel-level features for instance segmentation of the target human body and the target object, and predicts the interaction relationship between the target human body and the target object at the same time. The instance segmentation task and the interaction relationship prediction task share some features extracted by the CNN model, and has the following advantages: lightweight backbone network design, small amount of calculation, short time consumption, and is very suitable for application scenarios with limited computing power such as vehicle-mounted equipment; different from many existing methods based on target detection technology, the present invention extracts pixel-level semantic information and deeper level of character interaction semantic information, which is more robust; and it is easy to expand to other abnormal driving behavior detection such as eating, drinking and other behaviors; the false detection rate is low while maintaining a certain TPR recall rate (recall / true positive rate).

[0056] Compared with the prior art, which obtains the target object's circumscribed rectangular frame to artificially judge the target behavior, there may be false detection and does not consider the interaction relationship between the target object and the target body. In addition to the rectangular frame of the target object, this embodiment also obtains more accurate shape information of the target object, thereby increasing the supervision information of the shape dimension. At the same time, it also adds an interaction prediction branch, and increases the supervision information of the interaction relationship dimension, so that the accuracy of the target behavior can be improved and the false detection rate can be reduced. For example, in a vehicle-mounted system, assuming that the rest of the vehicle's personnel handing a cigarette to / putting it in the driver's mouth is not an abnormal behavior, the shape information of the target body in this embodiment can be used to exclude such behavior, thereby avoiding false detection.

[0057] In one possible implementation, Figure 2 As shown, the extraction step S110 may include:

[0058] Step S111 : acquiring a current image frame or a current video frame acquired by an image acquisition device as the image to be tested or the video frame to be tested.

[0059] In this embodiment, the current image frame or the current video frame may be acquired from an image acquisition device such as a camera as the image to be tested or the video frame to be tested.

[0060] Step S112, obtaining a first target frame corresponding to a human body-related part in the image to be tested or the video frame to be tested, wherein the human body-related part includes the face and body of the human body.

[0061] In this embodiment, any suitable method can be used to obtain the first target frame in the image to be tested or the video frame to be tested, such as the target detection method can be used to obtain the first target frame in the image to be tested or the video frame to be tested, or a pre-trained model such as a CNN model can be used to obtain the first target frame in the image to be tested or the video frame to be tested. In the case of using a CNN model to obtain the first target frame, the image to be tested or the video frame to be tested can be input into a pre-trained model such as a CNN model, and the CNN model outputs a face rectangular frame and a body rectangular frame, and of course, a hand rectangular frame and body key point position information can also be output.

[0062] In a possible implementation, face, body and key point detection is performed on the current image frame or the current video frame, and the face rectangle frame position and the body rectangle frame position are output. Based on the principle of the yolox detection algorithm framework, a custom Figure 6 The lightweight backbone network and head design shown in the figure simplifies the head design to improve the inference speed, and only uses 1 layer of 3*3 convolution layer and 1 layer of 1*1 convolution layer.

[0063] Step S113, determining whether the first target frame includes a face and a body.

[0064] If it is determined in step S113 that the first target frame contains a face and a body, for example, the CNN model outputs a face rectangular frame and a body rectangular frame, it means that it is in a human state at this time, and the target behavior detection can continue to be performed on the current image frame or the current video frame, so the following step S114 can be executed. Otherwise, for example, if the CNN model does not output a face rectangular frame and a body rectangular frame, it means that it is in an unmanned state at this time, and there is no need to continue to perform target behavior detection on the current image frame or the current video frame, so the following step S115 can be executed. That is, whether the human body, such as the driver, is out of position is determined based on whether the image frame or the video frame contains a face and a body. If so, return to step S111 to continue to detect the next image frame or the next video frame.

[0065] Step S114: extracting the target image from the image to be tested or the video frame to be tested according to the position information of the first target frame.

[0066] In this embodiment, if valid face and body information is detected, the first target frame can be expanded outward by a predetermined ratio to obtain a second target frame, and the image to be tested or the video frame to be tested is cropped along the second target frame, and the cropped image is the target image. That is, the first target frame area can be expanded outward by a certain ratio, and the expanded area is used as the target image, which can be cropped from the image to be tested or the video frame to be tested.

[0067] Therefore, compared with directly using the image to be tested or the video frame to be tested for target behavior detection, using the target image cropped from the image to be tested or the video frame to be tested for target behavior detection can narrow the image range for target behavior detection and exclude some irrelevant background information, thereby reducing the computing power required for target behavior detection and avoiding excessive computing power consumption. In addition, compared with directly using the first target frame for target behavior detection, using the second target frame obtained by expanding the first target frame for target behavior detection can ensure that target objects such as hands, cigarettes, mobile phones and other objects are within the target image.

[0068] In a possible implementation, the predetermined ratio can be dynamically adjusted according to the computing power of a device executing the processing method.

[0069] In this embodiment, the smaller the computing power of the device executing the processing method of the present invention, the smaller the predetermined ratio will be while ensuring that the target image can be used to detect whether the target human body has performed the target behavior. This can avoid the situation where the background covered is complex or too large, resulting in excessive computing power being required, thereby avoiding the situation where systems with less computing power, such as vehicle-mounted systems, are unable to support excessive computing power.

[0070] In a possible implementation, expanding the first target frame outward by a predetermined ratio to obtain the second target frame includes: expanding the first target frame vertically by a first value and horizontally by a second value to obtain the second target frame, wherein the width of the second target frame is twice as long as the width of the first target frame, and the height of the second target frame is twice as long as the height of the first target frame. For example, the first value and the second value are both 0.1.

[0071] In this embodiment, Figure 3As shown in FIG. 1 , the width of the blue first target frame is w and the height is h. The blue first target frame can be expanded by 0.1h up and down and by 0.1w left and right. Thus, a red rectangular frame is obtained by expanding the first target frame. The image to be tested or the video frame to be tested can be cropped according to the position information of the red rectangular frame, and then the target image (also called the target area to be tested) can be obtained, which is Figure 3 It should be understood that if the extension has reached an edge (boundary) in a certain direction of the image to be tested or the video frame to be tested, the extension in that direction may be stopped.

[0072] Step S115, obtaining the next image frame or the next video frame acquired by the image acquisition device as the image to be tested.

[0073] In this embodiment, since there is no need to perform target behavior detection on the current image frame or the current video frame in an unmanned state, target behavior detection can be performed on the next image frame or the next video frame. Therefore, the next image frame or the next video frame can be re-acquired as the image to be tested, and steps S111-S115 can be re-executed until the face and body are included in the first target frame.

[0074] In one possible implementation, Figure 7 As shown, the processing method may include steps S710-S750. Among them, steps S710, S720, and S750 can refer to the description of steps S110, S120, and S140 above, and will not be repeated here. In step S730, based on the shape information of the target object, the segmentation results whose similarity between the target object and the standard object is lower than a threshold are filtered out from the second segmentation results.

[0075] In this embodiment, the images predicted to have four interaction categories can be further filtered. Specifically, the shape information of the target object generated by the instance segmentation branch is matched with a set of standard object shape information, and the halcon shape matching algorithm is used to filter out instance samples with low similarity.

[0076] Accordingly, if Figure 7 As shown, in step S740, based on the feature information generated by the instance segmentation process, human interaction HOI detection is performed to determine the interaction information of the target human body and the target object corresponding to the shape information included in the filtered second segmentation result. That is, HOI detection can be performed based on the filtered instance segmentation result.

[0077] In a possible implementation, after the determination step S140 or S750, the processing method may further include: after executing the processing method on the current image frame or the current video frame, continuing to sequentially execute the processing method on M consecutive image frames or M consecutive video frames including the current image frame or the current video frame; detecting that the target human body has performed the target behavior in N image frames or N video frames among the M image frames or M video frames, and if M is greater than or equal to N, performing an alarm process for the target behavior; if M is less than N, not performing an alarm process for the target behavior. N is determined according to the number of frames transmitted per second of the device executing the processing method.

[0078] In this embodiment, considering that some frames are unstable, such as blurred / shaky images, smoothing can be used to eliminate false detections. That is, the filtered results can be smoothed. If N frames out of M consecutive frames of images or videos have detected the corresponding target behavior, where M is greater than or equal to N, an alarm for such target behavior will be issued, where N can be dynamically adjusted based on the overall frame rate (Frames Per Second, FPS) of the picture. For example, if 5 out of 10 consecutive frames have detected the target behavior, an alarm can be issued; if the computing power of the device is high, which can be 3-4fps, N can be set larger.

[0079] Therefore, this embodiment, supplemented by smoothing processing, can further improve the accuracy of behavior detection and can further reduce the occurrence of false detection, which is particularly advantageous in industrial vehicle driving with a complex cockpit background.

[0080] In one possible implementation, after the determination step S140 or S750, the processing method further includes: if the tolerance for the target behavior is lower than a threshold, performing an alarm process in response to determining in the determination step that the target human body has performed the target behavior; if the tolerance for the target behavior is not lower than a threshold, performing an alarm process in response to determining that the target human body has performed the target behavior for a predetermined number of consecutive times.

[0081] If the target behavior is strictly required, that is, the tolerance is low, such as smoking cannot be tolerated, then an alarm is issued as long as smoking behavior is detected in one image frame or video frame. If the target behavior is strictly required, that is, the tolerance is low, such as smoking can be slightly tolerated, then an alarm is issued after smoking behavior is detected in 2 or 3 consecutive image frames or video frames.

[0082] In a possible implementation, the alarm processing method includes at least one of a pop-up window alarm, an audio alarm, a video alarm, a text message alarm, a WeChat alarm, and an email alarm. Of course, any other suitable alarm method can also be used.

[0083] Figure 8 FIG. 1 is a block diagram of a target behavior processing device according to an embodiment of the present disclosure. Figure 8 As shown, the processing device 800 may include: an extraction module 810, an acquisition module 820, a detection module 830 and a determination module 840. The extraction module 810 is used to extract a target image for detecting the target behavior from the image to be tested. The acquisition module 820 is connected to the extraction module 810, and is used to perform instance segmentation on the target image to obtain a first segmentation result of a target human body and a second segmentation result of a target object in the target image, wherein the first segmentation result includes shape information of the target human body, and the second segmentation result includes shape information of the target object. The detection module 830 is connected to the acquisition module 820, and is used to perform human interaction HOI detection based on the feature information generated by the instance segmentation process to determine the interaction information between the target human body and the target object. The determination module 840 is connected to the detection module 830, and is used to determine whether the target human body has performed the target behavior based on the interaction information.

[0084] In a possible implementation, the extraction module 810 is configured to: obtain a current image frame or a current video frame acquired by an image acquisition device as the image to be tested or the video frame to be tested; obtain a first target frame corresponding to a human body-related part in the image to be tested or the video frame to be tested, and the human body-related part includes the face and body of the human body; and extract the target image from the image to be tested or the video frame to be tested according to the position information of the first target frame.

[0085] In a possible implementation, the extraction module 810 is configured to: determine whether the first target frame contains a face and a body; if it is determined that the first target frame contains a face and a body, extract the target image from the image to be tested or the video frame to be tested according to the position information of the first target frame; if it is determined that the first target frame does not contain a face and a body, obtain the next image frame or the next video frame captured by the image acquisition device as the image to be tested.

[0086] In a possible implementation, the extraction module 810 is configured to: expand the first target frame outward by a predetermined ratio to obtain a second target frame, crop the image to be tested along the second target frame, and the cropped image is the target image.

[0087] In one possible implementation, the extraction module 810 is configured to: expand the first target frame vertically by a first value and horizontally by a second value to obtain the second target frame, wherein the width of the second target frame is twice as long as the width of the first target frame by the first value, and the height of the second target frame is twice as long as the height of the first target frame by the second value.

[0088] In a possible implementation, the acquisition module 820 is configured to: input the target image into a target model, perform instance segmentation via an instance segmentation branch of the target model, and output the first segmentation result and the second segmentation result.

[0089] In one possible implementation, the detection module 830 is configured to: perform HOI detection using the feature information via the prediction branch of the target model to obtain interaction information, wherein the interaction information includes position information of the interaction point between the target human body and the target object, the interaction category between the target human body and the target object, and the position offset from the interaction point to the target human body and the target object. Accordingly, the determination module 840 is configured to: determine whether the target human body has performed the target behavior based on the position information of the interaction point, the interaction category and the position offset.

[0090] In a possible implementation, the detection module 830 is configured to: multiply the feature information with the corresponding weight, and perform feature fusion on the product through a feature pyramid to obtain the position information of the interaction point, the interaction category and the position offset.

[0091] In a possible implementation, the processing device 800 also includes: a filtering module (not shown), which is used to filter out segmentation results whose similarity between the target object and the standard object is lower than a threshold from the second segmentation results based on the shape information of the target object. Accordingly, the detection module 830 is configured to: perform HOI detection based on the feature information to determine the interaction information between the target human body and the target object corresponding to the shape information included in the filtered second segmentation result.

[0092] In one possible implementation, the processing device also includes an alarm module (not shown), wherein the processing device sequentially processes M consecutive image frames or M consecutive video frames including the current image frame or the current video frame; detects that the target human body has performed the target behavior from N image frames or N video frames among the M image frames or M video frames, wherein if M is greater than or equal to N, the alarm module performs alarm processing for the target behavior; if M is less than N, the alarm module does not perform alarm processing for the target behavior.

[0093] In one possible implementation, the processing device 800 also includes: an alarm module (not shown), which is used to perform alarm processing in response to the determination module determining that the target human body has performed the target behavior if the tolerance for the target behavior is lower than a threshold; if the tolerance for the target behavior is not lower than the threshold, then perform alarm processing in response to the determination module determining that the target human body has performed the target behavior for a predetermined number of consecutive times.

[0094] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0095] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above processing method is implemented. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0096] The disclosed embodiment also proposes a target behavior processing device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above-mentioned processing method when executing the instructions stored in the memory.

[0097] The embodiment of the present disclosure also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned processing method.

[0098] Fig. 9 1 is a block diagram of a target behavior processing device 1900 according to an exemplary embodiment. For example, the device 1900 may be provided as a terminal device, such as a vehicle-mounted terminal device. Fig. 9, the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above-mentioned processing method.

[0099] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0100] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above-mentioned processing method.

[0101] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0102] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0103] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0104] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0105] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0106] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0107] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0108] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0109] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing a target behavior, characterized in that: include: An extraction step, extracting a target image for detecting the target behavior from the image to be tested or the video frame to be tested; an acquisition step of performing instance segmentation on the target image to obtain a first segmentation result of a target human body and a second segmentation result of a target object in the target image, wherein the first segmentation result includes shape information of the target human body, and the second segmentation result includes shape information of the target object; A detection step, based on the feature information generated by the instance segmentation process, performs human interaction HOI detection to determine the interaction information between the target human body and the target object; A determination step is performed to determine whether the target human body has performed the target behavior based on the interaction information.

2. The processing method according to claim 1, characterized in that: The extraction step comprises: Acquire a current image frame or a current video frame acquired by an image acquisition device as the image to be tested or the video frame to be tested; Acquire a first target frame corresponding to a relevant part of a human body in the image to be tested or the video frame to be tested, wherein the relevant part of the human body includes a face and a body of the human body; The target image is extracted from the image to be tested or the video frame to be tested according to the position information of the first target frame.

3. The processing method according to claim 2, characterized in that: After acquiring the first target frame, the extracting step further includes: Determining whether the first target frame contains a face and a body; If it is determined that the first target frame contains a face and a body, extracting the target image from the image to be tested or the video frame to be tested according to the position information of the first target frame; If it is determined that the first target frame does not include a face or a body, the next image frame or the next video frame acquired by the image acquisition device is acquired as the image to be tested or the video frame to be tested.

4. The processing method according to claim 2, characterized in that: Extracting the target image from the image to be tested or the video frame to be tested according to the position information of the first target frame includes: The first target frame is expanded outward by a predetermined ratio to obtain a second target frame, and the image to be measured or the video frame to be measured is cropped along the second target frame, and the cropped image is the target image.

5. The processing method according to claim 4, characterized in that: The predetermined ratio can be dynamically adjusted according to the computing power of the device executing the processing method.

6. The processing method according to claim 4, characterized in that: Expanding the first target frame outward by a predetermined ratio to obtain a second target frame comprises: The first target frame is expanded vertically by a first value and horizontally by a second value to obtain the second target frame, wherein the width of the second target frame is twice as long as the width of the first target frame by the first value, and the height of the second target frame is twice as long as the height of the first target frame by the second value.

7. The processing method according to claim 6, characterized in that: The first value and the second value are both 0.

1.

8. The processing method according to claim 1, characterized in that: The obtaining step comprises: The target image is input into a target model, instance segmentation is performed through an instance segmentation branch of the target model, and the first segmentation result and the second segmentation result are output.

9. The processing method according to claim 8, characterized in that: The first segmentation result also includes first target frame position information of the target human body, and the second segmentation result also includes object frame position information of the target object and classification information of the target object.

10. The processing method according to claim 8, characterized in that: The detection step includes: performing HOI detection using the feature information via the prediction branch of the target model to obtain interaction information, wherein the interaction information includes position information of an interaction point between the target human body and the target object, an interaction category between the target human body and the target object, and a position offset from the interaction point to the target human body and the target object. Accordingly, The determining step includes: determining whether the target human body performs the target behavior according to the position information of the interaction point, the interaction category and the position offset.

11. The processing method according to claim 10, characterized in that: Using the feature information to perform HOI detection via the prediction branch of the target model to obtain interaction information includes: The feature information is multiplied by the corresponding weight, and the product is subjected to feature fusion in a feature pyramid manner to obtain the position information of the interaction point, the interaction category, and the position offset.

12. The processing method according to claim 10, characterized in that: The position information of the interaction point is the coordinates of the center position of the line connecting the center point of the target human body and the center point of the target object for each character.

13. The processing method according to claim 1, characterized in that: Before the detection step, the processing method further includes: filtering out segmentation results whose similarity between the target object and the standard object is lower than a threshold from the second segmentation results according to the shape information of the target object, and accordingly, The detection step includes: performing HOI detection based on the feature information to determine the interaction information between the target human body and the target object corresponding to the shape information included in the filtered second segmentation result.

14. The processing method according to claim 13, characterized in that: After the determining step, the processing method further includes: After executing the processing method on the current image frame or the current video frame, continue to sequentially execute the processing method on M consecutive image frames or M consecutive video frames including the current image frame or the current video frame; Detecting that the target human body performs the target behavior from N image frames or N video frames in the M image frames or M video frames, If M is greater than or equal to N, an alarm is performed for the target behavior; If M is smaller than N, no alarm processing is performed for the target behavior.

15. The processing method according to claim 14, characterized in that: N is determined according to the number of transmission frames per second of the device executing the processing method.

16. The processing method according to claim 1, characterized in that: After the determining step, the processing method further includes: If the tolerance for the target behavior is lower than a threshold, in response to determining in the determining step that the target human body has performed the target behavior, an alarm process is performed; If the tolerance for the target behavior is not lower than a threshold, an alarm process is performed in response to determining that the target human body has performed the target behavior for a predetermined number of consecutive times.

17. The processing method according to any one of claims 14 to 16, characterized in that: The alarm processing includes at least one of a pop-up alarm, an audio alarm, a video alarm, a text message alarm, a WeChat alarm, and an email alarm.

18. A target behavior processing device, characterized in that: include: An extraction module, used for extracting a target image for detecting the target behavior from the image to be tested or the video frame to be tested; an acquisition module, configured to perform instance segmentation on the target image to obtain a first segmentation result of a target human body and a second segmentation result of a target object in the target image, wherein the first segmentation result includes shape information of the target human body, and the second segmentation result includes shape information of the target object; A detection module, configured to perform human interaction HOI detection based on feature information generated by an instance segmentation process to determine interaction information between the target human body and the target object; A determination module is used to determine whether the target human body has performed the target behavior based on the interaction information.

19. A target behavior processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the processing method described in any one of claims 1 to 17 when executing the instructions stored in the memory.

20. A vehicle equipped with a driver monitoring system DMS, characterized in that: The DMS is equipped with the processing device according to claim 18 or 19.

21. The vehicle according to claim 20, characterized in that The target behavior includes at least one of the driver's smoking behavior, the driver's drinking behavior, the driver's phone call behavior, and the driver's eating behavior.

22. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the processing method according to any one of claims 1 to 17 is implemented.

Images