Behavior recognition method and device, and vehicle

Abstract

The application provides a behavior recognition method, device and vehicle, and the method comprises the following steps: performing face detection on a to-be-recognized image to extract a face image of a target person; determining a line-of-sight landing point area of the target person based on a full-image feature of the to-be-recognized image and a face feature of the face image; when the line-of-sight landing point area is within the range of the to-be-recognized image, performing target detection on the to-be-recognized image in the line-of-sight landing point area; and determining a behavior recognition result of the target person based on the target detection result. The behavior recognition method, device and vehicle provided by the application do not need to process a complex spatial mapping relationship, the system architecture and algorithm complexity are simplified, and the accuracy and real-time performance of behavior recognition are improved.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a behavior recognition method, device, and vehicle. Background Technology

[0002] Behavior recognition refers to the analysis and identification of human movements, postures, and behaviors to determine and recognize a person's identity, behavioral intentions, and psychological state. Behavior recognition can be applied to various scenarios, such as driving safety monitoring (determining if the driver is distracted), human-computer interaction (understanding passenger attention focus), and advertising effectiveness evaluation (analyzing audience attention to advertising content).

[0003] Taking the behavior recognition of occupants inside a vehicle as an example, current methods often involve using human region detection models to obtain the spatial positions of each occupant at consecutive moments, thus generating corresponding bounding boxes. Then, based on the detection results at each moment, these bounding boxes are stitched together using identity or location information to form a continuous spatiotemporal image sequence. Finally, a pre-trained behavior recognition model is used to recognize the generated spatiotemporal image sequence to obtain the corresponding behavior recognition result. However, before inputting the spatiotemporal image sequence into the behavior recognition model, it is necessary to correctly associate the bounding boxes of the same identity or location that are continuously monitored at different moments. However, correct association involves a complex binding process and is prone to errors, thus affecting the accuracy of behavior recognition. Summary of the Invention

[0004] This invention provides a behavior recognition method, device, and vehicle to address the deficiencies in the prior art.

[0005] This invention provides a behavior recognition method, comprising the following steps:

[0006] Face detection is performed on the image to be identified to extract the face image of the target person;

[0007] Based on the full-image features of the image to be identified and the facial features of the face image, the gaze point area of ​​the target person is determined.

[0008] When the gaze point area is within the range of the image to be identified, target detection is performed on the image to be identified within the gaze point area;

[0009] Based on the target detection results, the behavior recognition results of the target person are determined.

[0010] According to a behavior recognition method provided by the present invention, the step of performing target detection on the image to be recognized within the area of ​​the gaze point includes:

[0011] Determine the coordinates of the candidate bounding box in the region where the line of sight falls;

[0012] Based on the candidate box coordinates, the target image is cropped from the image to be identified;

[0013] Target detection is performed on the target image.

[0014] According to a behavior recognition method provided by the present invention, determining the coordinates of the candidate bounding box of the gaze landing point region includes:

[0015] In the image to be identified, the region where the gaze falls is rendered;

[0016] The coordinates of the rendered area are used as the coordinates of the candidate bounding box.

[0017] According to a behavior recognition method provided by the present invention, determining the gaze location region of the target person based on the full-image features of the image to be recognized and the facial features of the face image includes:

[0018] The full-image features and the face features are fused to obtain the fused features;

[0019] Based on the fusion features, the area where the target person's gaze falls is determined.

[0020] According to a behavior recognition method provided by the present invention, the method further includes:

[0021] If the gaze point is outside the range of the image to be identified, it is determined that there is no gaze interaction between the target person and the object in the image to be identified.

[0022] According to a behavior recognition method provided by the present invention, the full-image features are extracted from the image to be recognized based on a first feature extraction module, and the face features are extracted from the face image based on a second feature extraction module, wherein the first feature extraction module and the second feature extraction module share weights.

[0023] The present invention also provides a behavior recognition device, comprising the following modules:

[0024] The extraction unit is used to perform face detection on the image to be recognized and extract the face image of the target person;

[0025] The determining unit is used to determine the gaze location area of ​​the target person based on the full-image features of the image to be identified and the facial features of the face image.

[0026] The detection unit is configured to perform target detection on the image to be identified within the area of ​​the line of sight when the area of ​​the line of sight is within the range of the image to be identified.

[0027] The identification unit is used to determine the behavior identification result of the target person based on the target detection result.

[0028] The present invention also provides a vehicle, comprising:

[0029] The behavior recognition device described above and the camera installed inside the vehicle are used to capture images inside the vehicle as the images to be recognized.

[0030] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the behavior recognition method as described above.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the behavior recognition method as described above.

[0032] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the behavior recognition method as described above.

[0033] The behavior recognition method, device, and vehicle provided by this invention predict the gaze location area of ​​a target person based on full-image features and facial features. After determining the gaze location area, it can be bound based on the relationship of "person-gaze-object". That is, this invention no longer needs to precisely determine the person's specific location in space, nor does it need to construct complex spatial mapping relationships to determine which objects the person interacts with. This invention focuses on which objects or areas overlap or are close to the gaze location area, thus inferring the person's behavioral intention and the objects they interact with. Therefore, this invention eliminates the need to handle complex spatial mapping relationships, simplifying the system architecture and algorithm complexity. Furthermore, predicting the gaze location area, compared to determining the person's specific location in space, does not involve complex algorithmic steps, resulting in higher prediction efficiency. In other words, this invention improves the accuracy and real-time performance of behavior recognition. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating the behavior recognition method provided by the present invention.

[0036] Figure 2This is a flowchart illustrating another behavior recognition method provided by the present invention.

[0037] Figure 3 This is a schematic diagram of the execution flow of the gaze tracking module provided by the present invention.

[0038] Figure 4 This is a schematic diagram of the behavior recognition device provided by the present invention.

[0039] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0041] Taking in-vehicle occupant behavior recognition as an example, current methods primarily rely on human region detection models to obtain the spatial positions of each occupant at consecutive moments, resulting in corresponding bounding boxes. Then, based on the detection results at each moment, these bounding boxes are stitched together using identity or location information to form a continuous spatiotemporal image sequence. Finally, a pre-trained behavior recognition model is used to recognize the generated spatiotemporal image sequence to obtain the corresponding behavior recognition result. However, before inputting the spatiotemporal image sequence into the behavior recognition model, it is necessary to correctly associate the target boxes continuously monitored at different moments for the same identity or location. However, correct association involves a complex binding process and is prone to errors; the complexity of recognition increases with the number of occupants. Furthermore, since the positions of occupants may change over time, the bounding boxes at different moments may not be precisely aligned with their spatial positions, increasing the difficulty of alignment during image sequence stitching. Finally, behavior recognition models often struggle to adapt to the diversity of different scenarios and behaviors. In complex environments such as smart cockpits, the gaze behavior of occupants may vary due to personal habits, vehicle configuration, and other factors, further increasing the difficulty of behavior recognition.

[0042] In addition, there are also approaches to behavior recognition based on face detection models and object detection models, respectively. These two models can locate the areas containing people and objects, and then use a relationship modeling network (as shown in the convolutional network) to model the movement relationships between these areas to obtain the behavior recognition results. While this approach avoids the complexity of spatiotemporal behavior detection by focusing on detection results at a single time point, it often relies on implicit feature representations in deep learning models when constructing the relationship between people and objects. Although these models can learn powerful feature extraction and classification capabilities, their decision-making process is often a "black box," lacking intuitive interpretability. In other words, even if the model can predict certain interactive behaviors, it cannot accurately reflect the specific "interaction relationship" between people and objects inside the vehicle; for example, this approach cannot reflect interaction details such as "eye contact" between people and objects.

[0043] To address this issue, the present invention provides a behavior recognition method. This method can be applied to behavior recognition in different scenarios, such as recognizing the behavior of occupants inside a vehicle or indoors. For ease of understanding, the following embodiments are all illustrated using the application to recognize the behavior of occupants inside a vehicle as an example.

[0044] in, Figure 1 This is a flowchart illustrating the behavior recognition method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0045] Step 110: Perform face detection on the image to be recognized and extract the face image of the target person.

[0046] Specifically, the image to be identified is the image for which behavior recognition needs to be performed. This image typically contains information about the target person and their surrounding environment inside the vehicle. The image to be identified can be captured by a camera or pre-stored; this embodiment of the invention does not specifically limit its acquisition.

[0047] Furthermore, the target person refers to the person of interest in the image to be identified, or it can be understood as the object for which behavior recognition needs to be performed. The target person can be any person in the image to be identified, or it can be a specific person in the image to be identified. This embodiment of the invention does not make specific limitations in this regard.

[0048] Face detection refers to the process of identifying and locating the face of a target person in an image to be recognized. The location of the target person's face is usually represented by a rectangular bounding box. For example, before performing face detection on the image to be recognized, preprocessing can be performed, including grayscale conversion, noise reduction, and histogram equalization, to improve the accuracy and robustness of face detection. Next, face detection algorithms can be used to extract features from the image to be recognized to describe information such as the shape and texture of the face. Based on the extracted features, the location and size of the target person's face in the image to be recognized are determined, and the corresponding face image of the target person is output. This face image can be understood as the image corresponding to the rectangular bounding box region in the image to be recognized.

[0049] Step 120: Based on the full-image features of the image to be identified and the facial features of the face image, determine the gaze point area of ​​the target person.

[0050] Specifically, full-image features refer to the global features of the entire scene in the image to be identified. Full-image features contain information about all people and objects in the image to be identified, and are used to describe the global information in the image to be identified. Facial features refer to the features used to describe the shape and structure of the target person's face, and are used to characterize the detailed information of the target person's face.

[0051] Since global image features are used to represent global information in the image to be identified, the relationship between the target person and surrounding objects in the image can be determined based on these features, thus identifying the object or region that the target person might be looking at. Since facial features are used to represent the detailed information of the target person's face, precise information about the direction and angle of the target person's gaze can be determined based on these features, helping us to more accurately estimate the gaze's focal point area. Based on this, global features can be used to roughly determine the target person's likely area of ​​focus, and combined with the precise gaze direction and angle information carried by facial features, the gaze's focal point area can be precisely determined. Here, the gaze's focal point area refers to the image region that the target person's gaze points to when they are looking at it.

[0052] Relying solely on full-image features to determine the gaze location region can be affected by complex backgrounds and interfering objects in the image, leading to inaccurate estimation. Furthermore, full-image features typically lack precise information about the direction and angle of the target person's gaze, making accurate estimation difficult. While facial features can provide precise information about the direction and angle of the target person's gaze, relying solely on facial features for gaze location determination can be affected by factors such as facial pose, expression, and occlusions. Additionally, if the face region in the image is small or of poor quality, facial feature extraction and recognition may be limited, impacting the accuracy of gaze location estimation.

[0053] Therefore, the embodiments of the present invention combine the advantages of full-image features and facial features to improve the accuracy and robustness of gaze point estimation.

[0054] Step 130: If the area where the line of sight falls is within the range of the image to be recognized, perform target detection on the image to be recognized within the area where the line of sight falls.

[0055] Specifically, if the gaze point is outside the image to be identified, it indicates that the gaze point exceeds the boundary of the image. For example, the target person may be looking out of a car, and in this case, it is impossible to determine the object being looked at based on the image inside the car. If the gaze point is within the image to be identified, it indicates that the target person is looking at an object in the image, and that the object is located within the gaze point. Therefore, target detection can be performed on the image to be identified within the gaze point to determine the category of the object.

[0056] Step 140: Based on the target detection results, determine the behavior recognition results of the target person.

[0057] Specifically, the object detection result is used to characterize whether an object was detected within the gaze-focused area. If so, the detected object is identified, and its type is determined. For example, if the gaze-focused area is identified as a book, it can be determined that the target person is reading a book, i.e., the gaze recognition result of the target person. Conversely, if the gaze-focused area is identified as a mobile phone, it can be determined that the target person is using a mobile phone.

[0058] It's important to note that traditional behavior recognition models require constructing a complex spatial mapping of "people-objects-behaviors" before performing behavior recognition. This mapping aims to capture the location of people in space, their interactions with objects within that space, and how these interactions constitute a specific behavior model. Constructing this spatial mapping requires extensive data collection, annotation, and preprocessing, as well as sophisticated algorithms to analyze and interpret the spatial relationships. Furthermore, the diversity and dynamism of people, objects, and behaviors in images make it difficult to accurately construct such spatial mappings.

[0059] In contrast, this invention employs a simpler and more intuitive method. Instead of relying on complex spatial mapping relationships, it directly understands the gaze behavior of a target by predicting the gaze placement area. Specifically, this invention first predicts the gaze placement area of ​​the target based on full-image features and facial features. After determining the gaze placement area, it can be bound based on the "person-gaze-object" relationship. That is, this invention no longer needs to precisely determine the person's specific location in space, nor does it need to construct complex spatial mapping relationships to determine which objects the person interacts with. Instead, this invention focuses on which objects or areas overlap or are close to the gaze placement area, thus inferring the person's behavioral intentions and the objects they interact with.

[0060] Therefore, the embodiments of the present invention no longer require handling complex spatial mapping relationships, simplifying the system architecture and algorithm complexity. Furthermore, predicting the gaze point area, compared to determining the specific location of a person in space, does not involve complex algorithmic steps, resulting in higher prediction efficiency. In other words, the embodiments of the present invention improve the accuracy and real-time performance of behavior recognition.

[0061] Based on the above embodiments, target detection is performed on the image to be recognized within the area of ​​the line of sight, including:

[0062] Determine the coordinates of the candidate bounding boxes in the area where the line of sight falls;

[0063] The target image is obtained by cropping from the image to be identified based on the candidate box coordinates;

[0064] Perform target detection on the target image.

[0065] Specifically, considering that the image to be identified includes not only the target person and the object being looked at, but also other objects in the environment, in order to avoid interference from other objects in target detection leading to false detections or missed detections, this embodiment of the invention focuses the target detection region on the gaze-on area of ​​the image to be identified. The target image mentioned above is the image corresponding to the gaze-on area in the image to be identified.

[0066] To address this, this embodiment of the invention first determines the coordinates of candidate bounding boxes for the gaze-point region, and then crops the target image from the image to be recognized based on these candidate bounding box coordinates. The candidate bounding box coordinates refer to the coordinates of the rectangular bounding box corresponding to the gaze-point region in the image to be recognized, and these coordinates characterize the specific location of the gaze-point region within the image. Optionally, the candidate bounding box coordinates typically include the coordinates of the upper left and lower right corners of the rectangle (x1, y1, x2, y2), or the center point coordinates plus width and height parameters (cx, cy, w, h).

[0067] Therefore, the embodiments of the present invention obtain the target image from the image to be identified based on the candidate box coordinates, thereby focusing the target image on the area where the gaze falls, and thus accurately determining the object being gazed at by the target person based on the target image, avoiding interference from other objects in the image to be identified.

[0068] Based on any of the above embodiments, determining the coordinates of the candidate bounding box of the viewing point region includes:

[0069] In the image to be identified, the area where the gaze falls is rendered;

[0070] Use the coordinates of the rendered area as the coordinates of the candidate bounding box.

[0071] Specifically, rendering refers to the process of visually marking or highlighting the gaze-attention area on an image to be identified. This can be achieved through methods such as heatmap rendering, color filling, or other rendering techniques. Heatmap rendering uses color variations to represent the intensity and distribution of the gaze-attention area; the more vibrant the color, the higher the gaze attention level in that area.

[0072] After rendering the gaze-focused area, it is highlighted with visual markers in the image to be recognized. This allows users to intuitively adjust the gaze-focused area using these markers. For example, after rendering, users can fine-tune the rendered area to ensure its boundaries precisely match the boundaries of the target object. This allows for accurate identification of the object within the gaze-focused area based on the target image.

[0073] Based on any of the above embodiments, determining the gaze location area of ​​the target person based on the full-image features of the image to be identified and the facial features of the face image includes:

[0074] By fusing features from the entire image and facial features, a fused feature is obtained.

[0075] Based on the fusion features, the area where the target person's gaze falls is determined.

[0076] Specifically, the fused feature is the feature obtained by merging the full image features and the face features. Optionally, the full image features and the face features can be spliced ​​together to obtain the fused feature.

[0077] Since full-image features are used to represent global information in the image to be identified, the relationship between the target person and surrounding objects in the image can be determined based on full-image features, thus identifying the object or region that the target person may be looking at. Since facial features are used to represent facial details of the target person, precise information about the direction and angle of the target person's gaze can be determined based on facial features, helping us to more accurately estimate the area where the gaze falls. On this basis, the fused features contain both global information in the image to be identified and facial details of the target person. Therefore, based on the fused features, the possible area of ​​interest of the target person can be roughly determined, and combined with the precise gaze direction and angle information carried by the facial features, the area where the target person's gaze falls can be accurately determined.

[0078] Therefore, the embodiments of the present invention, which determine the gaze location area of ​​a target person based on fused features, can fully utilize the advantages of both global features and facial features, thereby improving the accuracy of determining the gaze location area. Furthermore, a single feature may be affected by external factors such as changes in lighting, occlusion, and angle changes, leading to instability in gaze estimation. Fused features can reduce the interference of these external factors through the complementarity of multiple features, improving the robustness of the system.

[0079] Based on any of the above embodiments, the method further includes:

[0080] If the gaze point is outside the range of the image to be identified, it is determined that there is no gaze interaction between the target person and the object in the image to be identified.

[0081] Specifically, after obtaining the gaze point area, the boundary of the gaze point area is compared with the boundary of the image to be identified. If the gaze point area is completely outside the range of the image to be identified, that is, the gaze point area does not overlap with any object or area in the image to be identified, it can be considered that there is no gaze interaction between the target person and the object in the image to be identified.

[0082] Based on the above judgment results, if the gaze point is outside the image, a clear indication can be output, indicating that the target person is not currently paying attention to or looking at any object in the image to be identified, which helps to achieve more accurate and intelligent interactive control in applications such as human-computer interaction and video surveillance.

[0083] For example, in the current situation, navigation is being activated and the navigation prompts the driver to preview the driving route in advance. At this time, the method described in any of the above embodiments can be used to detect whether the driver is looking at the screen display in the vehicle. If not, it indicates that the driver is not looking at the display, and a prompt message can be given to remind the driver to pay attention to the content displayed on the display.

[0084] Based on any of the above embodiments, the full-image features are extracted from the image to be identified by the first feature extraction module, and the face features are extracted from the face image by the second feature extraction module. The first feature extraction module and the second feature extraction module share weights.

[0085] Specifically, the first feature extraction module is used to extract full-image features from the image to be identified. The first feature extraction module can be a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), or a Transformer, etc.

[0086] The second feature extraction module is used to extract facial features from the face image. Similarly, the second feature extraction module can also be a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), or a Transformer, etc.

[0087] Furthermore, the first and second feature extraction modules share weights, meaning they use the same parameters (i.e., weights) for updates during training. This significantly reduces the number of parameters that need to be trained in both modules, thereby lowering training complexity and the risk of overfitting. Additionally, sharing weights reduces parameter redundancy and promotes information sharing and feature extraction between the two modules.

[0088] It should be noted that the first feature extraction module and the second feature extraction module can extract corresponding features in parallel to improve feature extraction efficiency.

[0089] Based on any of the above embodiments Figure 2 This is a flowchart illustrating another behavior recognition method provided by the present invention, as shown below. Figure 2 As shown, this method is applied to passenger behavior recognition within the cabin, specifically including:

[0090] The system uses cameras installed inside the vehicle to capture real-time images of the vehicle's interior. These images are then input into a face detection model, which performs face detection to obtain the facial region information of the person inside the vehicle (i.e., the target person). Based on this facial region information, the image is cropped and reshaped to obtain the target person's face image. The image to be recognized is a complete scene image of the vehicle's interior.

[0091] The image to be recognized and the face image are input into the gaze tracking module. The gaze tracking module determines the gaze point area of ​​the target person based on the full-image features of the image to be recognized and the face features of the face image, determines whether the gaze point area is within the range of the image to be recognized, and renders the gaze point area in the image to be recognized.

[0092] in, Figure 3 This is a schematic diagram of the execution flow of the gaze tracking module provided by the present invention, as shown below. Figure 3 As shown, the gaze tracking module includes a first feature extraction module, a second feature extraction module, a feature fusion module, a landing point region binary classification model, and a landing point heatmap prediction model. The first feature extraction module and the second feature extraction module share weights.

[0093] The first feature extraction module is used to extract global features from the image to be identified, and the second feature extraction module is used to extract facial features from the face image. The first and second feature extraction modules extract features in parallel.

[0094] The feature fusion module is used to perform feature interaction and fusion between the full-image features and facial features to obtain fused features, and to determine the gaze location region based on the fused features. The feature fusion module can be designed based on techniques such as relevance convolution, feature concatenation, and attention mechanisms.

[0095] When multiple people are present in the vehicle, their facial images can be input into the second feature extraction module, which then extracts their respective facial features. The feature fusion module performs feature interaction and fusion on the overall image features and the features of each face separately to obtain the fused features for each person and the gaze point region for each person. In other words, when multiple people are present in the vehicle, the fused features for each person and the gaze point region for each person can be determined separately according to the above process.

[0096] The gaze location binary classification model is used to determine whether the gaze location area is within the range of the image to be recognized. If it is, it is marked as 1; otherwise, it is marked as 0. For example, when the gaze location area of ​​the target person is on a specific object inside the car, such as a mobile phone, book, seat, or door handle, the output result is 1. Conversely, when the gaze location area is on an object outside the car, such as the road ahead, which is not present in the image to be recognized, the output result is 0.

[0097] The gaze-attention heatmap prediction model is used to render the gaze-attention area in the image to be identified, obtaining a gaze-attention area response heatmap. High-response areas are used as the rendering area, and the coordinates [x1', y1', x2', y2'] of the rendering area are used as candidate bounding box coordinates. The rendering area is the gaze-attention area of ​​the target person, i.e., the target person's area of ​​focus, and therefore the coordinates [x1', y1', x2', y2'] of the rendering area are bound to the face region coordinates [x1, y1, x2, y2].

[0098] Specifically, when the binary classification model outputs a result of 1 in the landing area, the image to be identified is cropped according to the candidate box coordinates to obtain the target image. The target image is then detected based on the target detection model to obtain the target detection result. This target detection result is used to characterize the object category in the line-of-sight landing area.

[0099] Based on the large model, the object detection results are applied to determine the behavior recognition result. For example, if the object detection result indicates that the gaze point is a mobile phone, then the behavior recognition result is "the target person is playing with a mobile phone".

[0100] Therefore, the embodiments of the present invention are based on the concept that "[someone] is looking at a certain [object], the object is a [category], therefore [someone] is performing a certain [behavior]", and analyze and interpret the behavior of the target person without having to construct a complex spatial mapping relationship, thus simplifying the system architecture and algorithm complexity.

[0101] The behavior recognition device provided by the present invention is described below. The behavior recognition device described below and the behavior recognition method described above can be referred to in correspondence.

[0102] Based on any of the above embodiments Figure 4 This is a schematic diagram of the behavior recognition device provided by the present invention, as shown below. Figure 4 As shown, the device includes:

[0103] Extraction unit 410 is used to perform face detection on the image to be recognized and extract the face image of the target person;

[0104] The determining unit 420 is used to determine the gaze location area of ​​the target person based on the full-image features of the image to be identified and the facial features of the face image.

[0105] The detection unit 430 is used to perform target detection on the image to be recognized within the area of ​​the line of sight when the area of ​​the line of sight is within the range of the image to be recognized.

[0106] The recognition unit 440 is used to determine the behavior recognition result of the target person based on the target detection result.

[0107] Therefore, this invention employs a simpler and more intuitive method. Instead of relying on complex spatial mapping relationships, it directly understands the gaze behavior of a target by predicting the gaze placement area. Specifically, this invention first predicts the gaze placement area of ​​the target based on full-image features and facial features. After determining the gaze placement area, it can be bound based on the "person-gaze-object" relationship. In other words, this invention no longer needs to precisely determine the person's specific location in space, nor does it need to construct complex spatial mapping relationships to determine which objects the person interacts with. Instead, this invention focuses on which objects or areas overlap or are close to the gaze placement area, thus inferring the person's behavioral intentions and the objects they interact with.

[0108] Therefore, the embodiments of the present invention no longer require handling complex spatial mapping relationships, simplifying the system architecture and algorithm complexity. Furthermore, predicting the gaze point area, compared to determining the specific location of a person in space, does not involve complex algorithmic steps, resulting in higher prediction efficiency. In other words, the embodiments of the present invention improve the accuracy and real-time performance of behavior recognition.

[0109] Based on any of the above embodiments, target detection is performed on the image to be recognized within the area of ​​the line of sight, including:

[0110] Determine the coordinates of the candidate bounding boxes in the area where the line of sight falls;

[0111] The target image is obtained by cropping from the image to be identified based on the candidate box coordinates;

[0112] Perform target detection on the target image.

[0113] Based on any of the above embodiments, determining the coordinates of the candidate bounding box of the viewing point region includes:

[0114] In the image to be identified, the area where the gaze falls is rendered;

[0115] Use the coordinates of the rendered area as the coordinates of the candidate bounding box.

[0116] Based on any of the above embodiments, determining the gaze location area of ​​the target person based on the full-image features of the image to be identified and the facial features of the face image includes:

[0117] By fusing features from the entire image and facial features, a fused feature is obtained.

[0118] Based on the fusion features, the area where the target person's gaze falls is determined.

[0119] Based on any of the above embodiments, it further includes:

[0120] If the gaze point is outside the range of the image to be identified, it is determined that there is no gaze interaction between the target person and the object in the image to be identified.

[0121] Based on any of the above embodiments, the full-image features are extracted from the image to be identified by the first feature extraction module, and the face features are extracted from the face image by the second feature extraction module. The first feature extraction module and the second feature extraction module share weights.

[0122] Based on any of the above embodiments, the present invention also provides a vehicle, comprising:

[0123] The behavior recognition device and the camera installed in the vehicle as described in any of the above embodiments, the camera being used to capture images inside the vehicle as images to be recognized.

[0124] Specifically, after the camera captures images inside the vehicle, it inputs these images as images to be recognized into the behavior recognition device. The behavior recognition device then performs face detection on the images to be recognized, extracting the face image of the target person. Based on the full-image features of the images to be recognized and the facial features of the face image, the gaze point area of ​​the target person is determined. If the gaze point area is within the range of the images to be recognized, target detection is performed on the images to be recognized within the gaze point area. Based on the target detection results, the behavior recognition result of the target person is determined.

[0125] Therefore, this invention no longer relies on complex spatial mapping relationships, but directly understands the gaze behavior of a target person by predicting the gaze placement area. Specifically, this invention first predicts the gaze placement area of ​​the target person based on full-image features and facial features. After determining the gaze placement area, it can be bound based on the "person-gaze-object" relationship. That is to say, this invention no longer needs to accurately determine the specific location of the person in space, nor does it need to construct complex spatial mapping relationships to determine which objects the person interacts with. Instead, this invention focuses on which objects or areas overlap or are close to the gaze placement area, thus inferring the person's behavioral intentions and the objects they interact with.

[0126] Therefore, the embodiments of the present invention no longer require handling complex spatial mapping relationships, simplifying the system architecture and algorithm complexity. Furthermore, predicting the gaze point area, compared to determining the specific location of a person in space, does not involve complex algorithmic steps, resulting in higher prediction efficiency. In other words, the embodiments of the present invention improve the accuracy and real-time performance of behavior recognition.

[0127] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a behavior recognition method, which includes: performing face detection on the image to be recognized to extract a face image of the target person; determining the gaze point region of the target person based on the full-image features of the image to be recognized and the face features of the face image; if the gaze point region is within the range of the image to be recognized, performing target detection on the image to be recognized within the gaze point region; and determining the behavior recognition result of the target person based on the target detection result.

[0128] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the behavior recognition method provided by the above methods. The method includes: performing face detection on an image to be recognized to extract a face image of a target person; determining the gaze point region of the target person based on the full-image features of the image to be recognized and the face features of the face image; when the gaze point region is within the range of the image to be recognized, performing target detection on the image to be recognized within the gaze point region; and determining the behavior recognition result of the target person based on the target detection result.

[0130] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the behavior recognition method provided by the above methods. The method includes: performing face detection on an image to be recognized to extract a face image of a target person; determining the gaze point region of the target person based on the full-image features of the image to be recognized and the face features of the face image; when the gaze point region is within the range of the image to be recognized, performing target detection on the image to be recognized within the gaze point region; and determining the behavior recognition result of the target person based on the target detection result.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A behavior recognition method, characterized in that, include: Face detection is performed on the image to be identified to extract the face image of the target person; Based on the full-image features of the image to be identified and the facial features of the face image, the gaze point area of ​​the target person is determined; the full-image features contain information about all people and all objects in the image to be identified, and the full-image features are used to describe the global information in the image to be identified; the facial features refer to the features that describe the shape and structure of the target person's face, and the facial features are used to characterize the facial detail information of the target person; When the gaze point area is within the range of the image to be identified, target detection is performed on the image to be identified within the gaze point area; Based on the target detection results, the behavior recognition results of the target person are determined, and the target detection results are used to characterize the object category in the area where the gaze falls.

2. The behavior recognition method according to claim 1, characterized in that, The step of performing target detection on the image to be identified within the area of ​​the line of sight includes: Determine the coordinates of the candidate bounding box in the region where the line of sight falls; Based on the candidate box coordinates, the target image is cropped from the image to be identified; Target detection is performed on the target image.

3. The behavior recognition method according to claim 2, characterized in that, Determining the coordinates of the candidate bounding box for the region where the line of sight falls includes: In the image to be identified, the region where the gaze falls is rendered; The coordinates of the rendered area are used as the coordinates of the candidate bounding box.

4. The behavior recognition method according to any one of claims 1 to 3, characterized in that, The step of determining the gaze point region of the target person based on the full-image features of the image to be identified and the facial features of the face image includes: The full-image features and the face features are fused to obtain the fused features; Based on the fusion features, the area where the target person's gaze falls is determined.

5. The behavior recognition method according to any one of claims 1 to 3, characterized in that, The method further includes: If the gaze point is outside the range of the image to be identified, it is determined that there is no gaze interaction between the target person and the object in the image to be identified.

6. The behavior recognition method according to any one of claims 1 to 3, characterized in that, The full-image features are extracted from the image to be identified based on the first feature extraction module, and the face features are extracted from the face image based on the second feature extraction module. The first feature extraction module and the second feature extraction module share weights.

7. A behavior recognition device, characterized in that, include: The extraction unit is used to perform face detection on the image to be recognized and extract the face image of the target person; The determining unit is used to determine the gaze point region of the target person based on the full-image features of the image to be identified and the facial features of the face image; the full-image features include information on all people and all objects in the image to be identified, and the full-image features are used to describe global information in the image to be identified; the facial features refer to features that describe the shape and structure of the target person's face, and the facial features are used to characterize the facial detail information of the target person; The detection unit is configured to perform target detection on the image to be identified within the area of ​​the line of sight when the area of ​​the line of sight is within the range of the image to be identified. The recognition unit is used to determine the behavior recognition result of the target person based on the target detection result, wherein the target detection result is used to characterize the object category in the area where the gaze falls.

8. A vehicle, characterized in that, include: The behavior recognition device as described in claim 7 and the camera installed in the vehicle, wherein the camera is used to capture images inside the vehicle as the images to be recognized.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the behavior recognition method as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the behavior recognition method as described in any one of claims 1 to 6.

Images