Behavior recognition method and device and vehicle

Abstract

The invention provides a behavior recognition method and device and a vehicle, and the method comprises the steps: carrying out the face detection of a to-be-recognized image, and extracting a face image of a target person; based on the full-image features of the to-be-recognized image and the face features of the face image, determining a sight drop point area of the target person; under the condition that the sight line drop point area is in the range of the to-be-recognized image, performing target detection on the to-be-recognized image in the sight line drop point area; and determining a behavior recognition result of the target person based on the target detection result. According to the behavior recognition method and device and the vehicle provided by the invention, a complex space mapping relation does not need to be processed any more, the system architecture and algorithm complexity are simplified, and the accuracy and real-time performance of behavior recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a behavior recognition method, device and vehicle. Background Art

[0002] Behavior recognition refers to the analysis and recognition of human movements, postures and behaviors to determine and identify a person's identity, behavioral intentions and psychological state. Behavior recognition can be applied to a variety of scenarios, such as driving safety monitoring (determining whether the driver is distracted), human-computer interaction (understanding the passenger's focus), advertising effect evaluation (analyzing the audience's attention to the advertising content), etc.

[0003] Taking the behavior recognition of people in the car as an example, the spatial position of each member in the car at continuous moments is currently obtained through the human body region detection model, that is, the corresponding detection frame is obtained. Then, according to the detection results at each moment, these detection frames are spliced ​​together using identity or position information to form a continuous spatiotemporal image sequence. Finally, the above-generated spatiotemporal image sequence is recognized using a pre-trained behavior recognition model to obtain the corresponding behavior recognition result. However, before the above-mentioned spatiotemporal image sequence is input into the behavior recognition model, it is necessary to correctly associate the target frames that are continuously monitored at different times for the same identity or position, but the correct association involves a complex binding process and is prone to errors, which in turn affects the accuracy of behavior recognition. Summary of the invention

[0004] The present invention provides a behavior recognition method, device and vehicle to solve the defects in the prior art.

[0005] The present invention provides a behavior recognition method, comprising the following steps: Perform face detection on the image to be identified and extract the face image of the target person; Based on the full-image features of the image to be identified and the facial features of the facial image, determining the sight point area of ​​the target person; When the sight line landing area is within the range of the image to be identified, target detection is performed on the image to be identified within the sight line landing area; Based on the target detection result, a behavior recognition result of the target person is determined.

[0006] According to a behavior recognition method provided by the present invention, target detection is performed on the image to be recognized in the sight point area, including: Determine the candidate frame coordinates of the sight point area; Based on the candidate frame coordinates, cropping the target image from the image to be identified; Performing target detection on the target image.

[0007] According to a behavior recognition method provided by the present invention, determining the candidate frame coordinates of the sight point area includes: In the image to be recognized, rendering the sight point area; The coordinates of the rendering area are used as the candidate frame coordinates.

[0008] According to a behavior recognition method provided by the present invention, the determining of the sight point area of ​​the target person based on the full-image features of the image to be recognized and the facial features of the facial image includes: Fusing the full image feature and the face feature to obtain a fused feature; Based on the fusion features, the sight point area of ​​the target person is determined.

[0009] According to a behavior recognition method provided by the present invention, the method further includes: When the sight point area is outside the range of the image to be identified, it is determined that there is no gaze interaction between the target person and the object in the image to be identified.

[0010] According to a behavior recognition method provided by the present invention, the full-image features are extracted from the image to be recognized based on a first feature extraction module, and the facial features are extracted from the facial image based on a second feature extraction module, and the first feature extraction module and the second feature extraction module share weights.

[0011] The present invention also provides a behavior recognition device, comprising the following modules: An extraction unit, used for performing face detection on the image to be identified, and extracting a face image of a target person; A determination unit, configured to determine the sight point area of ​​the target person based on the full-image features of the image to be identified and the facial features of the facial image; A detection unit, configured to perform target detection on the image to be identified within the sight point area when the sight point area is within the range of the image to be identified; The recognition unit is used to determine the behavior recognition result of the target person based on the target detection result.

[0012] The present invention also provides a vehicle, comprising: The behavior recognition device as described above and the camera arranged in the vehicle, wherein the camera is used to collect images in the vehicle as the images to be recognized.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned behavior recognition methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the behavior recognition method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the behavior recognition method described above is implemented.

[0016] The behavior recognition method, device and vehicle provided by the present invention predict the sight point area of ​​the target person based on the full-image features and facial features. After the sight point area is determined, it can be bound based on the "person-sight-object" relationship. In other words, the present invention no longer needs to accurately determine the specific position of the person in space, nor does it need to construct a complex spatial mapping relationship to determine which objects the person interacts with. The present invention focuses on which objects or areas overlap or are close to the sight point area, that is, the person's behavioral intentions and interaction objects can be inferred. It can be seen that the present invention no longer needs to deal with complex spatial mapping relationships, and simplifies the system architecture and algorithm complexity. At the same time, compared with determining the specific position of a person in space, predicting the sight point area does not involve complex algorithm steps, and thus the prediction efficiency is higher. In other words, the present invention improves the accuracy and real-time performance of behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the behavior recognition method provided by the present invention.

[0019] Figure 2 It is a flow chart of another behavior recognition method provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the execution flow of the sight tracking module provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the behavior recognition device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Taking the behavior recognition of people in the car as an example, the spatial position of each member in the car at continuous moments is obtained through the human region detection model, that is, the corresponding detection frame is obtained. Then, according to the detection results at each moment, these detection frames are spliced ​​together using identity or position information to form a continuous spatiotemporal image sequence. Finally, the pre-trained behavior recognition model is used to recognize the generated spatiotemporal image sequence to obtain the corresponding behavior recognition result. However, before the above spatiotemporal image sequence is input into the behavior recognition model, it is necessary to correctly associate the target frames that are continuously monitored at different times for the same identity or position, but the correct association involves a complex binding process and is prone to errors. As the number of people in the car increases, the complexity of recognition will also increase. In addition, since the position of the people in the car may change over time, the detection frames at different times cannot be accurately aligned with the spatial position, which increases the difficulty of alignment when splicing the image sequence. Finally, the behavior recognition model is often difficult to adapt to the diversity of different scenes and behaviors. In complex environments such as smart cockpits, the gaze behavior of people in the car may vary due to personal habits, vehicle configuration and other factors, which further increases the difficulty of behavior recognition.

[0025] In addition, there are also behavior recognition based on face detection models and object detection models. Through these two models, the personnel area and the object area can be located respectively, and the relationship modeling network (such as the convolutional network in the figure) is used to model the movement relationship between the personnel area and the object area to obtain the behavior recognition result. Although this solution avoids the complexity of spatiotemporal behavior detection by focusing on the detection results of a single time point, when constructing the relationship between people and objects, it often relies on implicit feature representation in the deep learning model. Although the model can learn powerful feature extraction and classification capabilities, the decision-making process of the model is often a "black box" operation and lacks intuitive explanation. In other words, even if the model can predict certain interactive behaviors, it cannot accurately reflect the specific "interactive relationship" between people and objects in the car. For example, this solution cannot reflect the interaction details such as "eye contact" between people and objects in the car.

[0026] In this regard, the present invention provides a behavior recognition method. The method can be applied to behavior recognition in different scenarios, such as behavior recognition of people in a car, or behavior recognition of people indoors. In order to facilitate understanding of the technical solution of the present invention, the following embodiments are all described by taking the application of behavior recognition of people in a car as an example.

[0027] in, Figure 1 is a flow chart of the behavior recognition method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0028] Step 110: Perform face detection on the image to be recognized and extract the face image of the target person.

[0029] Specifically, the image to be identified is the image for which behavior recognition is required, and the image to be identified usually contains information about the target person and the surrounding environment in the vehicle. The image to be identified can be obtained by photographing a camera device or can be pre-stored, which is not specifically limited in the embodiment of the present invention.

[0030] In addition, the target person refers to a person of interest in the image to be identified, and can also be understood as an object that needs to perform behavior recognition. The target person can be any person in the image to be identified, or can be a specified person in the image to be identified. The embodiment of the present invention does not make specific limitations on this.

[0031] Face detection refers to the process of identifying and locating the face of a target person from an image to be identified. The face position of the target person is usually represented by a rectangular frame. Exemplarily, before performing face detection on the image to be identified, the image to be identified can be preprocessed, including grayscale, denoising, histogram equalization and other operations to improve the accuracy and robustness of face detection. Then, the face detection algorithm can be used to extract features used to describe information such as face shape and texture from the image to be identified, and the face position and size of the target person in the image to be identified can be determined based on the extracted features, and then the corresponding face image of the target person is output. Among them, the face image can be understood as the image corresponding to the rectangular frame area in the image to be identified.

[0032] Step 120: Determine the sight point area of ​​the target person based on the full-image features of the image to be identified and the facial features of the facial image.

[0033] Specifically, the full-image feature refers to the global feature of the entire scene in the image to be identified. The full-image feature contains information about all people and objects in the image to be identified, and is used to describe the global information in the image to be identified. The face feature refers to the feature used to describe the face shape and structure of the target person, and is used to characterize the face details of the target person.

[0034] Since the full-image features are used to characterize the global information in the image to be identified, the relationship between the target person and the surrounding objects in the image to be identified can be determined based on the full-image features, and the objects or areas that the target person may be looking at can be determined. Since the facial features are used to characterize the facial details of the target person, the precise information of the target person's line of sight direction and angle can be determined based on the facial features, helping us to estimate the line of sight landing area more accurately. On this basis, the possible focus area of ​​the target person can be roughly determined based on the global features, and then combined with the precise line of sight direction and angle information carried by the facial features, the line of sight landing area of ​​the target person can be accurately determined. Among them, the line of sight landing area refers to the image area where the line of sight is directed when the target person is looking.

[0035] If the line of sight is determined only by the full-image features, it may be affected by the complex background and interfering objects in the image to be identified, resulting in inaccurate estimation of the line of sight. In addition, the full-image features usually cannot provide accurate information about the line of sight direction and angle of the target person, so it is difficult to accurately estimate the line of sight area. Although facial features can provide accurate information about the line of sight direction and angle of the target person, relying solely on facial features to determine the line of sight area may be affected by factors such as facial posture, expression, and occlusions. In addition, if the face area in the image to be identified is small or of poor quality, the extraction and recognition of facial features may be limited, thus affecting the accuracy of the estimation of the line of sight area.

[0036] Therefore, the embodiments of the present invention combine the advantages of full-image features and facial features to improve the accuracy and robustness of the line of sight point area estimation.

[0037] Step 130: When the sight point area is within the range of the image to be identified, target detection is performed on the image to be identified within the sight point area.

[0038] Specifically, if the sight point area is outside the range of the image to be identified, it indicates that the sight point area exceeds the boundary of the image to be identified. For example, the target person may be looking outside the car, and at this time, it is impossible to determine the object the target person is looking at based on the image to be identified inside the car. If the sight point area is within the range of the image to be identified, it indicates that the target person is looking at the object in the image to be identified, and the object is located in the sight point area, and then the target detection can be performed on the image to be identified in the sight point area to identify the category of the object.

[0039] Step 140: Determine the behavior recognition result of the target person based on the target detection result.

[0040] Specifically, the target detection result is used to indicate whether an object is detected in the sight point area. If so, the detected object is identified to determine the type of the object. For example, if the sight point area is identified as a book, it can be determined that the target person is reading a book, which is the gaze recognition result of the target person. For another example, if the sight point area is identified as a mobile phone, it can be determined that the target person is playing with the mobile phone.

[0041] It should be noted that before traditional behavior recognition models can perform behavior recognition, they need to construct a complex "person-object-behavior" spatial mapping relationship. This spatial mapping relationship aims to capture the position of a person in space, which objects in space they interact with, and how these interactions constitute a specific behavior model. Constructing this spatial mapping relationship requires a lot of data collection, annotation, and preprocessing work, as well as complex algorithms to parse and interpret the spatial relationship. In addition, due to the diversity and dynamics of people, objects, and behaviors in images, it is difficult to accurately construct the resulting spatial mapping relationship.

[0042] In contrast, the embodiment of the present invention adopts a more concise and intuitive method. The embodiment of the present invention no longer relies on complex spatial mapping relationships, but directly understands the gaze behavior of the target person by predicting the line of sight landing area. Specifically, the embodiment of the present invention first predicts the line of sight landing area of ​​the target person based on the full-image features and facial features. After the line of sight landing area is determined, it can be bound based on the "person-line of sight-object" relationship. In other words, the embodiment of the present invention no longer needs to accurately determine the specific position of the person in space, nor does it need to construct a complex spatial mapping relationship to determine which objects the person interacts with. On the contrary, the embodiment of the present invention focuses on which objects or areas the line of sight landing area overlaps or is close to, that is, the person's behavioral intentions and interaction objects can be inferred.

[0043] It can be seen that the embodiment of the present invention no longer needs to process complex spatial mapping relationships, which simplifies the system architecture and algorithm complexity. At the same time, compared with determining the specific position of a person in space, predicting the sight point area does not involve complex algorithm steps, and thus the prediction efficiency is higher. In other words, the embodiment of the present invention improves the accuracy and real-time performance of behavior recognition.

[0044] Based on the above embodiment, target detection is performed on the image to be recognized within the sight point area, including: Determine the candidate frame coordinates of the sight point area; Based on the coordinates of the candidate box, the target image is cropped from the image to be identified; Perform object detection on the target image.

[0045] Specifically, considering that the image to be identified includes not only the target person and the object the target person is looking at, but also other objects in the environment, in order to avoid interference from other objects in target detection leading to false detection or missed detection, the embodiment of the present invention focuses the target detection area on the sight point area of ​​the image to be identified. The target image is the image corresponding to the sight point area in the image to be identified.

[0046] To this end, an embodiment of the present invention first determines the candidate frame coordinates of the sight point area, and then crops the target image from the image to be identified based on the candidate frame coordinates. The candidate frame coordinates refer to the rectangular frame coordinates corresponding to the sight point area in the image to be identified, and the candidate frame coordinates are used to characterize the specific position of the sight point area in the image to be identified. Optionally, the candidate frame coordinates generally include the coordinates of the upper left corner and the lower right corner of the rectangular frame (x1, y1, x2, y2), or the center point coordinates plus the width and height parameters (cx, cy, w, h).

[0047] It can be seen that the embodiment of the present invention obtains the target image by cropping the target image from the image to be identified based on the candidate frame coordinates, so that the target image is focused on the line of sight area, and then the target person's gaze object can be accurately determined based on the target image, avoiding interference from other objects in the image to be identified.

[0048] Based on any of the above embodiments, determining the candidate frame coordinates of the sight point area includes: In the image to be recognized, the area where the sight line falls is rendered; The coordinates of the rendering area are used as the candidate box coordinates.

[0049] Specifically, rendering refers to the process of marking or highlighting the sight point area in a visual way on the image to be recognized, such as heat map rendering, color filling or other rendering methods. Among them, heat map rendering represents the intensity and distribution of the sight point area through color changes. The brighter the color, the higher the sight attention of the area.

[0050] After rendering the sight point area, the sight point area is highlighted with a visual mark in the image to be identified, and then when the user needs to adjust the sight point area, the adjustment can be made intuitively through the visual mark part. For example, after rendering the sight point area, the user can fine-tune the range of the rendering area so that the boundary of the rendering area can accurately match the boundary of the target object, that is, the boundary of the target image can accurately match the boundary of the target object, and then the object in the sight point area can be accurately identified based on the target image.

[0051] Based on any of the above embodiments, based on the full-image features of the image to be identified and the facial features of the facial image, determining the sight point area of ​​the target person includes: Fusion of full-image features and face features to obtain fusion features; Based on the fusion features, the target person’s sight point area is determined.

[0052] Specifically, the fused feature is a feature obtained by merging the full image feature and the face feature. Optionally, the full image feature and the face feature may be spliced ​​to obtain the fused feature.

[0053] Since the full-image feature is used to represent the global information in the image to be identified, the relationship between the target person and the surrounding objects in the image to be identified can be determined based on the full-image feature, and then the object or area that the target person may be looking at can be determined. Since the facial feature is used to represent the facial details of the target person, the precise information of the target person's line of sight direction and angle can be determined based on the facial feature, helping us to more accurately estimate the line of sight area. On this basis, the fusion feature contains both the global information in the image to be identified and the facial details of the target person. Based on the fusion feature, the possible focus area of ​​the target person can be roughly determined, and combined with the precise line of sight direction and angle information carried by the facial feature, the line of sight area of ​​the target person can be accurately determined.

[0054] It can be seen that the embodiment of the present invention determines the sight point area of ​​the target person based on the fusion feature, which can make full use of the advantages of global features and facial features, thereby improving the accuracy of determining the sight point area. In addition, a single feature may be affected by external factors such as lighting changes, occlusion, and angle changes, resulting in unstable sight estimation. The fusion feature can reduce the interference of these external factors through the complementarity of multiple features, thereby improving the robustness of the system.

[0055] Based on any of the above embodiments, the method further includes: When the sight point area is outside the range of the image to be identified, it is determined that there is no gaze interaction between the target person and the object in the image to be identified.

[0056] Specifically, after obtaining the gaze landing area, the boundary of the gaze landing area is compared with the boundary of the image to be identified. If the gaze landing area is completely outside the range of the image to be identified, that is, there is no overlap between the gaze landing area and any object or area in the image to be identified, it can be considered that there is no gaze interaction between the target person and the object in the image to be identified.

[0057] Based on the above judgment results, if the line of sight is outside the image, a clear indication can be output to indicate that the target person is not currently paying attention to or looking at any object in the image to be identified, which will help to achieve more accurate and intelligent interactive control in applications such as human-computer interaction and video surveillance.

[0058] For example, in the current situation, navigation is being turned on, and the navigation prompts the driver to preview the driving route in advance. At this time, any of the above-mentioned embodiments can be used to detect whether the driver is looking at the screen display in the car. If not, it indicates that the driver is not looking at the display. At this time, a prompt message can be given to prompt the driver to pay attention to the content displayed on the display.

[0059] Based on any of the above embodiments, the full image features are extracted from the image to be identified based on the first feature extraction module, the face features are extracted from the face image based on the second feature extraction module, and the first feature extraction module and the second feature extraction module share weights.

[0060] Specifically, the first feature extraction module is used to extract full-image features from the image to be identified. The first feature extraction module can be a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM) or a Transformer, etc.

[0061] The second feature extraction module is used to extract facial features from the facial image. Similarly, the second feature extraction module can also be a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM) or a Transformer, etc.

[0062] In addition, the first feature extraction module shares weights with the second feature extraction module, that is, the first feature extraction module and the second feature extraction module use the same parameters (i.e., weights) for updating during the training process, thereby significantly reducing the number of parameters that need to be trained during the training process of the first feature extraction module and the second feature extraction module, thereby reducing the complexity of training and the risk of overfitting. In addition, sharing weights can reduce parameter redundancy and promote information sharing and feature extraction between the two feature extraction modules.

[0063] It should be noted that the first feature extraction module and the second feature extraction module can extract corresponding features in parallel to improve feature extraction efficiency.

[0064] Based on any of the above embodiments, Figure 2 is a flow chart of another behavior recognition method provided by the present invention, such as Figure 2 As shown, the method is applied to passenger behavior recognition in the cabin, specifically including: The camera installed inside the vehicle collects the in-vehicle image in real time, and inputs the in-vehicle image as the image to be identified into the face detection model. The face detection model performs face detection on the image to be identified, obtains the face area information of the person in the vehicle (i.e. the target person), and crops and regularizes the image to be identified based on the face area information to obtain the face image of the target person. Among them, the image to be identified is the complete scene image inside the vehicle.

[0065] The image to be identified and the face image are input into the gaze tracking module, which determines the gaze area of ​​the target person based on the full-image features of the image to be identified and the face features of the face image, determines whether the gaze area is within the range of the image to be identified, and renders the gaze area in the image to be identified.

[0066] in, Figure 3 is a schematic diagram of the execution flow of the sight tracking module provided by the present invention, such as Figure 3 As shown, the gaze tracking module includes a first feature extraction module, a second feature extraction module, a feature fusion module, a landing area binary classification model and a landing point heat map prediction model, and the first feature extraction module and the second feature extraction module share weights.

[0067] The first feature extraction module is used to extract global features from the image to be identified, and the second feature extraction module is used to extract facial features from the facial image. The first feature extraction module and the second feature extraction module extract features in parallel.

[0068] The feature fusion module is used to interact and fuse the full image features and face features to obtain fused features, and determine the sight point area based on the fused features. The feature fusion module can be designed based on technologies such as correlation convolution, feature splicing and attention mechanism.

[0069] Among them, when there are multiple people in the car, the facial images corresponding to each person can be input into the second feature extraction module, and the second feature extraction module extracts the corresponding facial features. The feature fusion module obtains the fusion features corresponding to each person and the sight point area corresponding to each person based on feature interaction and fusion of the whole image features and each facial feature. In other words, when there are multiple people in the car, the fusion features corresponding to each person and the sight point area corresponding to each person can be determined respectively according to the above process.

[0070] The two-class model of the landing area is used to determine whether the sight landing area is within the range of the image to be identified. If so, it is marked as 1, otherwise it is marked as 0. For example, when the target person's sight landing area is on a specific object in the car, such as a mobile phone, book, seat, door handle, etc., the output result is 1. On the contrary, when the sight landing area is on an object outside the car, the road ahead, etc. that is not presented in the image to be identified, the output result is 0.

[0071] The landing point heat map prediction model is used to render the sight point area in the image to be recognized, obtain the landing point area response heat map, use the high response area as the rendering area, and use the coordinates of the rendering area [x1', y1', x2', y2'] as the candidate box coordinates. Among them, the rendering area is the sight point area of ​​the target person, that is, the target person's focus area, and then the coordinates of the rendering area [x1', y1', x2', y2'] are bound to the coordinates of the face area [x1, y1, x2, y2].

[0072] Among them, when the output result of the binary classification model of the landing area is 1, the image to be identified is cropped according to the candidate box coordinates to obtain the target image, and the target image is detected based on the target detection model to obtain the target detection result, which is used to characterize the object category in the line of sight landing area.

[0073] Based on the big model, the target detection result is applied to determine the behavior recognition result. For example, if the target detection result is that the sight point area is a mobile phone, then the behavior recognition result is "the target person is playing with a mobile phone."

[0074] It can be seen that the embodiments of the present invention are based on the concept of "[someone] is looking at a certain [object], the category of the object is [category], so [someone] is performing a certain [behavior]", and analyze and interpret the behavior of the target person without the need to construct a complex spatial mapping relationship, thereby simplifying the system architecture and algorithm complexity.

[0075] The behavior recognition device provided by the present invention is described below. The behavior recognition device described below and the behavior recognition method described above can be referenced to each other.

[0076] Based on any of the above embodiments, Figure 4 is a schematic diagram of the structure of the behavior recognition device provided by the present invention, such as Figure 4 As shown, the device comprises: An extraction unit 410 is used to perform face detection on the image to be identified and extract a face image of a target person; The determination unit 420 is used to determine the sight point area of ​​the target person based on the full-image features of the image to be identified and the facial features of the facial image; A detection unit 430 is used to perform target detection on the image to be identified in the sight point area when the sight point area is within the range of the image to be identified; The recognition unit 440 is used to determine the behavior recognition result of the target person based on the target detection result.

[0077] It can be seen that the embodiment of the present invention adopts a more concise and intuitive method. The embodiment of the present invention no longer relies on complex spatial mapping relationships, but directly understands the gaze behavior of the target person by predicting the sight point area. Specifically, the embodiment of the present invention first predicts the sight point area of ​​the target person based on the full-image features and facial features. After the sight point area is determined, it can be bound based on the "person-sight-object" relationship. In other words, the embodiment of the present invention no longer needs to accurately determine the specific position of the person in space, nor does it need to construct a complex spatial mapping relationship to determine which objects the person interacts with. On the contrary, the embodiment of the present invention focuses on which objects or areas the sight point area overlaps or is close to, that is, the person's behavioral intentions and interaction objects can be inferred.

[0078] It can be seen that the embodiment of the present invention no longer needs to process complex spatial mapping relationships, which simplifies the system architecture and algorithm complexity. At the same time, compared with determining the specific position of a person in space, predicting the sight point area does not involve complex algorithm steps, and thus the prediction efficiency is higher. In other words, the embodiment of the present invention improves the accuracy and real-time performance of behavior recognition.

[0079] Based on any of the above embodiments, target detection is performed on the image to be recognized within the sight point area, including: Determine the candidate frame coordinates of the sight point area; Based on the coordinates of the candidate box, the target image is cropped from the image to be identified; Perform object detection on the target image.

[0080] Based on any of the above embodiments, determining the candidate frame coordinates of the sight point area includes: In the image to be recognized, the area where the sight line falls is rendered; The coordinates of the rendering area are used as the candidate box coordinates.

[0081] Based on any of the above embodiments, based on the full-image features of the image to be identified and the facial features of the facial image, determining the sight point area of ​​the target person includes: Fusion of full-image features and face features to obtain fusion features; Based on the fusion features, the target person’s sight point area is determined.

[0082] Based on any of the above embodiments, it also includes: When the sight point area is outside the range of the image to be identified, it is determined that there is no gaze interaction between the target person and the object in the image to be identified.

[0083] Based on any of the above embodiments, the full image features are extracted from the image to be identified based on the first feature extraction module, the face features are extracted from the face image based on the second feature extraction module, and the first feature extraction module and the second feature extraction module share weights.

[0084] Based on any of the above embodiments, the present invention further provides a vehicle, comprising: The behavior recognition device as described in any of the above embodiments and the camera arranged in the vehicle, the camera is used to collect images inside the vehicle as images to be recognized.

[0085] Specifically, after the camera captures the image inside the car, it inputs the image inside the car as the image to be recognized into the behavior recognition device, and the behavior recognition device performs face detection on the image to be recognized and extracts the face image of the target person; based on the full-image features of the image to be recognized and the face features of the face image, the sight point area of ​​the target person is determined; when the sight point area is within the range of the image to be recognized, target detection is performed on the image to be recognized within the sight point area; based on the target detection result, the behavior recognition result of the target person is determined.

[0086] It can be seen that the embodiment of the present invention no longer relies on complex spatial mapping relationships, but directly understands the gaze behavior of the target person by predicting the sight point area. Specifically, the embodiment of the present invention first predicts the sight point area of ​​the target person based on the full-image features and facial features. After the sight point area is determined, it can be bound based on the "person-sight-object" relationship. In other words, the embodiment of the present invention no longer needs to accurately determine the specific position of the person in space, nor does it need to construct a complex spatial mapping relationship to determine which objects the person interacts with. On the contrary, the embodiment of the present invention focuses on which objects or areas the sight point area overlaps or is close to, that is, the person's behavioral intentions and interaction objects can be inferred.

[0087] It can be seen that the embodiment of the present invention no longer needs to process complex spatial mapping relationships, which simplifies the system architecture and algorithm complexity. At the same time, compared with determining the specific position of a person in space, predicting the sight point area does not involve complex algorithm steps, and thus the prediction efficiency is higher. In other words, the embodiment of the present invention improves the accuracy and real-time performance of behavior recognition.

[0088] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the behavior recognition method, which includes: performing face detection on the image to be recognized and extracting the face image of the target person; determining the sight point area of ​​the target person based on the full image features of the image to be recognized and the face features of the face image; when the sight point area is within the range of the image to be recognized, performing target detection on the image to be recognized within the sight point area; and determining the behavior recognition result of the target person based on the target detection result.

[0089] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the behavior recognition method provided by the above methods, which includes: performing face detection on the image to be identified, and extracting the face image of the target person; determining the sight point area of ​​the target person based on the full-image features of the image to be identified and the face features of the face image; when the sight point area is within the range of the image to be identified, performing target detection on the image to be identified within the sight point area; and determining the behavior recognition result of the target person based on the target detection result.

[0091] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the behavior recognition method provided by the above-mentioned methods, the method comprising: performing face detection on the image to be recognized, and extracting a face image of a target person; determining a sight point area of ​​the target person based on full-image features of the image to be recognized and face features of the face image; when the sight point area is within the range of the image to be recognized, performing target detection on the image to be recognized within the sight point area; and determining a behavior recognition result of the target person based on the target detection result.

[0092] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A behavior recognition method, characterized in that: include: Perform face detection on the image to be identified and extract the face image of the target person; Based on the full-image features of the image to be identified and the facial features of the facial image, determining the sight point area of ​​the target person; When the sight line landing area is within the range of the image to be identified, target detection is performed on the image to be identified within the sight line landing area; Based on the target detection result, a behavior recognition result of the target person is determined.

2. The behavior recognition method according to claim 1, characterized in that: The performing target detection on the image to be identified in the sight point area includes: Determine the candidate frame coordinates of the sight point area; Based on the candidate frame coordinates, cropping the target image from the image to be identified; Performing target detection on the target image.

3. The behavior recognition method according to claim 2, characterized in that: The determining of the candidate frame coordinates of the sight point area includes: In the image to be recognized, rendering the sight point area; The coordinates of the rendering area are used as the candidate frame coordinates.

4. The behavior recognition method according to any one of claims 1 to 3, characterized in that: The determining the sight point area of ​​the target person based on the full-image features of the image to be identified and the facial features of the facial image includes: Fusing the full image feature and the face feature to obtain a fused feature; Based on the fusion features, the sight point area of ​​the target person is determined.

5. The behavior recognition method according to any one of claims 1 to 3, characterized in that: The method further comprises: When the sight point area is outside the range of the image to be identified, it is determined that there is no gaze interaction between the target person and the object in the image to be identified.

6. The behavior recognition method according to any one of claims 1 to 3, characterized in that: The full-image feature is extracted from the image to be recognized based on a first feature extraction module, and the face feature is extracted from the face image based on a second feature extraction module. The first feature extraction module and the second feature extraction module share a weight.

7. A behavior recognition device, characterized in that: include: An extraction unit, used for performing face detection on the image to be identified, and extracting a face image of a target person; A determination unit, configured to determine the sight point area of ​​the target person based on the full-image features of the image to be identified and the facial features of the facial image; A detection unit, configured to perform target detection on the image to be identified within the sight point area when the sight point area is within the range of the image to be identified; The recognition unit is used to determine the behavior recognition result of the target person based on the target detection result.

8. A vehicle, characterized in that: include: The behavior recognition device as described in claim 7 and the camera arranged in the vehicle, wherein the camera is used to capture the image inside the vehicle as the image to be recognized.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the behavior recognition method according to any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the behavior recognition method according to any one of claims 1 to 6 is implemented.

Images