Object behavior recognition method in vehicle cabin, vehicle and storage medium

Abstract

The invention discloses an object behavior recognition method in a vehicle cabin, a vehicle and a storage medium, and relates to the technical field of vehicles. The method comprises the following steps: acquiring a first image of a target object in a cabin, and determining a second image based on the first image; the second image is a local image corresponding to a target area in the first image, and the target area is an area related to a to-be-recognized behavior of the target object; inputting the first image and the second image into a twin network to obtain a similarity probability between a first image feature and a second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image; and based on the similarity probability, determining an identification result of the to-be-identified behavior of the target object. By adopting the method, the behavior recognition accuracy of the object in the vehicle cabin can be improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a method for identifying object behavior in a vehicle cabin, a vehicle, and a storage medium. Background Art

[0002] At present, the object behavior (or user behavior) recognition method in the vehicle cabin is mainly realized by the following methods: face detection is performed on the main driver's seat image frame sequence. If a face is detected, the face key point regression is performed in the face region of interest (ROI); the image of the ROI area is intercepted according to the key points. For example, if it is necessary to determine whether the main driver's seat user smokes, the image of the ROI area near the mouth is intercepted, and the intercepted image of the ROI area is input into the target image classification model to judge the main driver's seat object behavior. However, for the judgment of certain object behaviors, such as judging whether the main driver's seat user smokes, since smoking requires the behavioral feature of arm movement, judging the object behavior only based on the intercepted image of the ROI area near the mouth will result in low accuracy of the recognition results for certain object behaviors.

[0003] Therefore, how to improve the accuracy of object behavior recognition in the vehicle cabin has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a method for identifying object behavior in a vehicle cabin, a vehicle, and a storage medium, which can improve the accuracy of object behavior identification in the vehicle cabin.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying object behavior in a vehicle cabin, the method comprising:

[0006] Acquire a first image of a target object in the cockpit, and determine a second image based on the first image; the second image is a partial image corresponding to a target area in the first image, and the target area is an area related to the behavior to be identified of the target object;

[0007] Input the first image and the second image into the twin network to obtain the similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image;

[0008] Based on the similarity probability, determining the recognition result of the behavior to be recognized for the target object;

[0009] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0010] In one embodiment, a first image and a second image are input into a twin network to obtain a similarity probability between a feature of the first image and a feature of the second image, including: inputting the first image into a first feature extraction network included in the twin network to obtain a first image feature, and inputting the second image into a second feature extraction network included in the twin network to obtain a second image feature; determining the similarity between the first image feature and the second image feature; converting the similarity into a target value within a preset value range, and using the target value as the similarity probability between the first image feature and the second image feature.

[0011] In one embodiment, based on the similarity probability, the recognition result of the behavior to be recognized for the target object is determined, including: when it is determined that the similarity probability is greater than a preset threshold, the recognition result of the behavior to be recognized for the target object is determined as the occurrence of the behavior to be recognized; when it is determined that the similarity probability is less than or equal to the preset threshold, the recognition result of the behavior to be recognized for the target object is determined as the non-occurrence of the behavior to be recognized.

[0012] In one of the embodiments, when it is determined that the similarity probability is greater than a preset threshold, after determining that the recognition result of the behavior to be identified for the target object is the occurrence of the behavior to be identified, the method also includes: outputting prompt information, wherein the prompt information is used to indicate that the recognition result of the behavior to be identified for the target object is the occurrence of the behavior to be identified.

[0013] In one embodiment, the twin network is trained in the following manner: for each sample pair, the first sample image in the sample pair is input into the first feature extraction network included in the twin network to be trained to obtain the first sample image feature, and the second sample image in the sample pair is input into the second feature extraction network included in the twin network to be trained to obtain the second sample image feature; the total loss value of the sample pair is determined; the total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the sample pair The loss value between the first predicted behavior of the image and the actual behavior of the sample object, the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior of the sample object and the actual behavior, the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when the total loss values ​​of multiple sample pairs do not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample pairs meet the conditions for stopping training to obtain the twin network.

[0014] In one embodiment, the first loss value is determined by: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining the first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0015] In one embodiment, the second loss value and the third loss value are determined in the following manner: inputting the first sample image feature into the first classification network included in the twin network to be trained to obtain the first predicted behavior of the sample object; based on the first loss function, determining the second loss value between the first predicted behavior and the actual behavior of the sample object; and, inputting the second sample image feature into the second classification network included in the twin network to be trained to obtain the second predicted behavior of the sample object; based on the second loss function, determining the third loss value between the second predicted behavior and the actual behavior.

[0016] In one embodiment, determining a second image based on a first image includes: inputting the first image into a face detection model, and outputting face information corresponding to the target object when detecting that the first image contains a target object; based on the face information, capturing a face image corresponding to the target object from the first image; inputting the face image into a face key point detection model to obtain the face key point coordinates of the target object; determining a target area based on the face key point coordinates, and obtaining a second image corresponding to the target area from the face image.

[0017] In a second aspect, an embodiment of the present application provides a device for identifying object behavior in a vehicle cabin, the device comprising:

[0018] An acquisition and determination module, used to acquire a first image of a target object in the cabin, and determine a second image based on the first image; the second image is a local image corresponding to a target area in the first image, and the target area is an area related to the behavior to be identified of the target object;

[0019] A processing module, used for inputting the first image and the second image into the twin network to obtain a similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image;

[0020] A determination module, used to determine the recognition result of the to-be-recognized behavior of the target object based on the similarity probability;

[0021] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0022] In a third aspect, an embodiment of the present application provides a vehicle control device, including a memory and a processor, wherein the memory stores a computer program; when the processor executes the computer program, the following steps are implemented:

[0023] Acquire a first image of a target object in the cockpit, and determine a second image based on the first image; the second image is a partial image corresponding to a target area in the first image, and the target area is an area related to the behavior to be identified of the target object;

[0024] Input the first image and the second image into the twin network to obtain the similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image;

[0025] Based on the similarity probability, determining the recognition result of the behavior to be recognized for the target object;

[0026] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0027] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0028] Acquire a first image of a target object in the cockpit, and determine a second image based on the first image; the second image is a partial image corresponding to a target area in the first image, and the target area is an area related to the behavior to be identified of the target object;

[0029] Input the first image and the second image into the twin network to obtain the similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image;

[0030] Based on the similarity probability, determining the recognition result of the behavior to be recognized for the target object;

[0031] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0032] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0033] Acquire a first image of a target object in the cockpit, and determine a second image based on the first image; the second image is a partial image corresponding to a target area in the first image, and the target area is an area related to the behavior to be identified of the target object;

[0034] Input the first image and the second image into the twin network to obtain the similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image;

[0035] Based on the similarity probability, determining the recognition result of the behavior to be recognized for the target object;

[0036] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0037] In the sixth aspect, the present application also provides a vehicle, including a vehicle processor and an image acquisition device; the image acquisition device is used to acquire images including a target object in a cockpit; the vehicle processor is used to process the image to obtain a first image of the target object, and execute the steps in the method designed in the first aspect to determine the recognition result of the behavior to be recognized for the target object.

[0038] The above-mentioned method for identifying object behavior in the vehicle cabin, vehicle and storage medium, the vehicle can obtain a first image of the target object in the cabin, and determine a second image based on the first image; the second image is a local image corresponding to the target area in the first image, and the target area is an area related to the behavior to be identified of the target object; the first image and the second image are input into the twin network to obtain the similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image; based on the similarity probability, the recognition result of the behavior to be identified for the target object is determined; wherein, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image. Compared with the method of inputting a local image of the target object (i.e., the image of the face ROI area) into the target image classification model for object behavior recognition, the vehicle can use the twin network to determine the similarity probability between the first image (i.e., the full image) features and the second image (i.e., the local image) features corresponding to the target object, and then determine the recognition result of the behavior to be recognized for the target object based on the similarity probability, thereby improving the accuracy of object behavior recognition in the vehicle cabin. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 This is a schematic diagram of an application scenario of a method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application;

[0041] Figure 2 It is a flowchart of a method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application;

[0042] Figure 3 is a flow chart of another method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application;

[0043] Figure 4 It is a flowchart of another method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application;

[0044] Figure 5 It is a structural diagram of a twin network provided in an embodiment of the present application;

[0045] Figure 6 is a structural schematic diagram of an object behavior recognition device in a vehicle cabin provided by an embodiment of the present application;

[0046] Figure 7 It is a structural schematic diagram of a vehicle control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] First, combine Figure 1 The application scenario of the object behavior recognition method in the vehicle cabin provided by the embodiment of the present application is introduced. Figure 1 , Figure 1 Schematic diagram of an application scenario of a method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application. Figure 1 As shown, the application scenario includes a vehicle 100 , the vehicle 100 includes a vehicle controller 101 , and an image acquisition device 102 is included in the cockpit.

[0049] The vehicle controller 101 may first collect an image of the target object in the cockpit through the image acquisition device 102, and then process the collected image to obtain a first image of the target object, and store the first image. In the process of object behavior recognition, the vehicle controller 101 may obtain the first image of the target object, and determine the second image based on the first image, that is, obtain the local image of the target object from the global image of the target object; further, the vehicle controller 101 may input the first image and the second image into the twin network to obtain the image feature similarity probability between the two images; finally, based on the image feature similarity probability, determine the recognition result of the behavior to be recognized for the target object; wherein, the twin network is obtained by training with multiple sample pairs; each sample pair is composed of a global image and a local image corresponding to the sample object. Compared with the method of inputting a local image of the target object (i.e., the image of the face ROI area) into the target image classification model for object behavior recognition, the vehicle can use the twin network to determine the similarity probability between the first image (i.e., the full image) features and the second image (i.e., the local image) features corresponding to the target object, and then determine the recognition result of the behavior to be recognized for the target object based on the similarity probability, thereby improving the accuracy of object behavior recognition in the vehicle cabin.

[0050] The following is a detailed description of the object behavior recognition method in the vehicle cabin provided in an embodiment of the present application.

[0051] See also Figure 2 , Figure 2 is a flow chart of a method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application. The method can be performed by a vehicle (e.g. Figure 1 The vehicle controller 101 in FIG. Figure 2 As shown, the object behavior recognition method in the vehicle cabin may include but is not limited to the following steps:

[0052] S201, acquiring a first image of a target object in a cockpit, and determining a second image based on the first image; the second image is a local image corresponding to a target area in the first image, and the target area is an area related to a behavior to be identified of the target object.

[0053] The target object may be a driver, and correspondingly, the first image of the target object is the first image of the driver. Assuming that the behavior to be identified is whether the driver smokes, the target area is the area related to the smoking behavior, that is, the driver's mouth area. In this case, the second image is a partial image corresponding to the mouth area in the first image of the driver.

[0054] In an optional implementation, before step S201, the vehicle controller may also control an image acquisition device in the cockpit to acquire an image of the target object, and then process the acquired image to obtain a first image of the target object.

[0055] The image acquisition device is, for example, an infrared sensor. The image of the target object may be a single-channel infrared image of the target object in the cabin acquired by the infrared sensor. In this case, the vehicle controller may convert the single-channel infrared image into a three-channel infrared image, and use the three-channel infrared image as the first image of the target object.

[0056] S202. Input the first image and the second image into the twin network to obtain the similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image.

[0057] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0058] In an optional embodiment, the twin network is trained based on the total loss value of each sample pair in multiple sample pairs; wherein the total loss value of each sample pair is obtained based on the image features of each sample image in the sample pair.

[0059] S203: Determine the recognition result of the to-be-recognized behavior of the target object based on the similarity probability.

[0060] In the embodiment of the present application, the vehicle controller can obtain a first image of the target object in the cockpit, and determine a second image based on the first image; the second image is a local image corresponding to the target area in the first image, and the target area is an area related to the behavior to be identified of the target object; the first image and the second image are input into the twin network to obtain the similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image; based on the similarity probability, the recognition result of the behavior to be identified for the target object is determined; wherein the twin network is obtained by training with multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image. Using this method, compared with the method of inputting the local image of the target object (i.e., the image of the face ROI area) into the target image classification model for object behavior recognition, the vehicle controller can use the twin network to determine the similarity probability between the first image (i.e., the whole image) feature and the second image (i.e., the local image) feature corresponding to the target object, and then, based on the similarity probability, determine the recognition result of the behavior to be identified for the target object, thereby improving the accuracy of object behavior recognition in the vehicle cockpit.

[0061] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of another method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application. Figure 2 The object behavior recognition method shown in the vehicle cabin is different in that Figure 3 The method shown in FIG. 1 specifically describes how the vehicle controller inputs the first image and the second image into the twin network to obtain the similarity probability between the first image feature and the second image feature. Figure 3 As shown, the object behavior recognition method in the vehicle cabin may include but is not limited to the following steps:

[0062] S301, acquiring a first image of a target object in a cockpit, and determining a second image based on the first image; the second image is a local image corresponding to a target area in the first image, and the target area is an area related to a behavior to be identified of the target object.

[0063] In an optional implementation, the relevant description of step S301 can refer to the description of the aforementioned step S201, which will not be repeated here.

[0064] S302: input the first image into a first feature extraction network included in the twin network to obtain first image features, and input the second image into a second feature extraction network included in the twin network to obtain second image features.

[0065] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0066] Optionally, the first feature extraction network and the second feature extraction network included in the twin network can both be convolutional neural networks, wherein the first feature extraction network can be the target detection network YOLO-V8, and the second feature extraction network can be the lightweight neural network MobileNet-V3.

[0067] S303: Determine the similarity between the first image feature and the second image feature.

[0068] In an optional embodiment, the vehicle controller may determine the similarity between the first image feature and the second image feature by measuring the difference vector similarity, such as the following formula (1).

[0069]

[0070] In formula (1), G w (D1) is the first eigenvector, which represents the first image feature; G w (D2) is the second eigenvector, which represents the second image feature; e It means G w The number of elements in (D1) that are not equal to 0; z f It means G w The number of elements in (D2) that are not equal to 0; ef It means G w (D1) and G w The number of elements in the difference of (D2) that are not equal to 0.

[0071] In another optional embodiment, the vehicle controller may determine the similarity between the first image feature and the second image feature by a measurement method that combines distance similarity and cosine similarity, such as the following formula (2).

[0072] S(G w (D1), G w (D2))=dis(G w (D1), G w (D2))·cos(G w (D1), G w (D2)) (2)

[0073] In formula (2), G w (D1) is the first eigenvector, which represents the first image feature; G w(D2) is the second eigenvector, which represents the second image feature; dis(G w (D1),G w (D2)) represents G w (D1) and G w (D2) distance similarity between; cos(G w (D1),G w (D2)) represents G w (D1) and G w (D2) The cosine similarity between them.

[0074] S304: Convert the similarity into a target value within a preset value range, and use the target value as the similarity probability between the first image feature and the second image feature.

[0075] Optionally, the preset value interval may be [0,1].

[0076] That is to say, after determining the similarity between the first image feature and the second image feature, the vehicle controller may convert the determined similarity into a numerical value greater than or equal to 0 and less than or equal to 1, and use the converted numerical value as the similarity probability between the first image feature and the second image feature.

[0077] In an optional embodiment, the vehicle controller converts the similarity into a target value within a preset value range, and the similarity can be converted into a value within the range of [0,1] through a fully connected layer and a sigmoid layer included in the twin network.

[0078] S305: Determine the recognition result of the to-be-recognized behavior of the target object based on the similarity probability.

[0079] In an optional embodiment, the vehicle controller determines the recognition result of the behavior to be recognized for the target object based on the similarity probability, which may include: when it is determined that the similarity probability is greater than a preset threshold, determining that the recognition result of the behavior to be recognized for the target object is the occurrence of the behavior to be recognized; when it is determined that the similarity probability is less than or equal to the preset threshold, determining that the recognition result of the behavior to be recognized for the target object is that the behavior to be recognized did not occur.

[0080] For example, assuming that the preset threshold is 0.5, and the behavior to be identified is whether the target object is smoking, if the vehicle controller determines that the similarity probability between the first image feature and the second image feature is 0.86, the vehicle controller can further determine that the similarity probability 0.86 is greater than the preset threshold 0.5. In this case, the vehicle controller can determine that the target object is smoking. If the vehicle controller determines that the similarity probability between the first image feature and the second image feature is 0.48, the vehicle controller can further determine that the similarity probability 0.48 is less than the preset threshold 0.5. In this case, the vehicle controller can determine that the target object is not smoking.

[0081] In an embodiment of the present application, the vehicle controller can obtain the image features (i.e., first image features) of the first image (i.e., full-image image) and the image features (i.e., second image features) of the second image (i.e., partial image) corresponding to the target object through the twin network, and determine the similarity between the first image features and the second image features, and then, based on the similarity, determine the similarity probability between the first image features and the second image features, and finally, based on the similarity probability, determine the recognition result of the behavior to be recognized for the target object. In this way, compared to the method of inputting the partial image of the target object (i.e., the image of the face ROI area) into the target image classification model for object behavior recognition, the accuracy of object behavior recognition in the vehicle cabin can be improved by combining the full-image image features and partial image features corresponding to the target object to recognize the behavior of the target object.

[0082] See also Figure 4 , Figure 4 FIG. 1 is a flow chart of another method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application. Figure 3 Compared with the method shown, Figure 4 The method shown specifically describes how the vehicle controller determines the second image based on the first image, and how the vehicle controller determines the recognition result of the behavior to be recognized for the target object based on the similarity probability. Figure 4 As shown, the object behavior recognition method in the vehicle cabin may include but is not limited to the following steps:

[0083] S401: Acquire a first image of a target object in a cockpit.

[0084] In an optional implementation, before step S201, the vehicle controller may also control an image acquisition device in the cockpit to acquire an image of the target object, and then process the acquired image to obtain a first image of the target object.

[0085] The image acquisition device is, for example, an infrared sensor. The image of the target object is, for example, a single-channel infrared image of the target object in the cabin acquired by the infrared sensor. In this case, the vehicle controller may convert the single-channel infrared image into a three-channel infrared image, and use the three-channel infrared image as the first image of the target object.

[0086] S402: Input the first image into a face detection model, and when it is detected that the first image contains a target object, output face information corresponding to the target object.

[0087] In an optional implementation, the face detection model may be a deep learning model or a machine learning model, which is not limited here.

[0088] S403: Based on the facial information, capture a facial image corresponding to the target object from the first image.

[0089] In an optional implementation, the facial information may include but is not limited to facial coordinates, facial height, and facial width.

[0090] In this embodiment, the vehicle controller captures the facial image corresponding to the target object from the first image based on the facial information, and may capture the facial image corresponding to the target object from the first image based on the facial coordinates, the height of the face, and the width of the face.

[0091] S404: Input the face image into a face key point detection model to obtain the coordinates of the face key points of the target object.

[0092] S405: Determine a target area based on the coordinates of the facial key points, and obtain a second image corresponding to the target area from the facial image.

[0093] S406. Input the first image into a first feature extraction network included in the twin network to obtain first image features, and input the second image into a second feature extraction network included in the twin network to obtain second image features.

[0094] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0095] S407: Determine the similarity between the first image feature and the second image feature.

[0096] S408: Convert the similarity into a target value within a preset value range, and use the target value as the similarity probability between the first image feature and the second image feature.

[0097] In an optional implementation, the relevant descriptions of steps S406 to S408 can refer to the descriptions of the aforementioned steps S302 to S304, and will not be repeated here.

[0098] S409, determine whether the similarity probability is greater than a preset threshold, if so, execute step S410; if not, execute step S411.

[0099] S410: Determine that the recognition result of the behavior to be recognized for the target object is that the behavior to be recognized occurs.

[0100] In an optional implementation, after step S410, the vehicle controller may further output prompt information, which is used to indicate that the recognition result of the behavior to be recognized for the target object is the occurrence of the behavior to be recognized. This is helpful to remind passengers in the vehicle cabin to stop the behavior to be recognized in time.

[0101] For example, assuming that the target object is a driver, the behavior to be identified is whether the driver smokes, and assuming that the identification result of whether the driver smokes is that the driver has a smoking behavior, the vehicle controller can output a prompt message, and the prompt message is used to indicate that the driver has a smoking behavior. This is helpful to remind the driver to stop smoking in time, thereby helping to improve the safety of driving the vehicle.

[0102] S411: Determine that the recognition result of the behavior to be recognized for the target object is that the behavior to be recognized has not occurred.

[0103] In an embodiment of the present application, the vehicle controller may input a first image (i.e., a full-image image) into a face detection model, and when it is detected that the target object is contained in the first image, output face information corresponding to the target object; based on the face information, capture a face image corresponding to the target object from the first image; input the face image into a face key point detection model to obtain the face key point coordinates of the target object; determine the target area based on the face key point coordinates, and obtain a second image (i.e., a local image) corresponding to the target area from the face image; then, obtain image features of the full-image image (i.e., first image features) and image features of the local image (i.e., second image features) corresponding to the target object through a twin network, and determine the similarity between the first image features and the second image features, and then, based on the similarity, determine the similarity probability between the first image features and the second image features, and finally, based on the similarity probability, determine the recognition result of the behavior to be recognized for the target object. In this way, compared with the method of inputting a local image of the target object (i.e., the image of the face ROI area) into the target image classification model for object behavior recognition, the accuracy of object behavior recognition in the vehicle cabin can be improved by identifying the behavior of the target object by combining the full-image image features and local image features corresponding to the target object.

[0104] In an optional embodiment, Figure 2 , Figure 3 and Figure 4 The structure of the twin network mentioned in the method shown can be as follows Figure 5 As shown, Figure 5 Schematic diagram of a twin network structure provided in the embodiment of the present application. Figure 5 As shown, the twin network may include a first feature extraction network ( Figure 5 The first feature extraction network is a convolutional neural network 1, and the second feature extraction network ( Figure 5 In the figure, the second feature extraction network is drawn by convolutional neural network 2), the first classification network and the second classification network.

[0105] Optionally, the twin network can be trained by the vehicle controller through the following steps:

[0106] Step 1: Obtain multiple sample pairs and determine the actual behavior of the sample object in the sample image in each sample pair.

[0107] Step 2: For each sample pair, the first sample image in the sample pair is input into the first feature extraction network included in the twin network to be trained to obtain the first sample image feature, and the second sample image in the sample pair is input into the second feature extraction network included in the twin network to be trained to obtain the second sample image feature.

[0108] Step three, determine the total loss value of the sample pair; the total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained.

[0109] In an optional implementation, the vehicle controller may use the following formula (3) to determine the total loss value of the sample pair.

[0110] L = λ×(L cls (G w (D1′)+L cls (G w (D2′)))+(1-2λ)×L rps (3)

[0111] In formula (3), L represents the total loss value of the sample pair; λ represents the loss weight; G w (D1′) is the first feature vector, which represents the first sample image feature; G w (D2′) is the second feature vector, which represents the second sample image feature; L rps It represents the first loss value, that is, the loss value between the first sample image and the second sample image; L cls (G w (D1′) represents the second loss value, that is, the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object; L cls (G w (D2′) represents the third loss value, that is, the loss value between the second predicted behavior and the actual behavior of the sample object.

[0112] In an optional embodiment, the first loss value (ie, L rps ) can be determined by the vehicle controller in the following manner: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining a first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0113] In this embodiment, the sample image feature similarity between the first sample image feature and the second sample image feature can be determined by the vehicle controller through a difference vector similarity measurement method, or it can be determined by the vehicle controller through a measurement method combining distance similarity and cosine similarity, which is not limited here.

[0114] When the vehicle controller determines the sample image feature similarity between the first sample image feature and the second sample image feature by measuring the difference vector similarity, the following formula (4) may be used.

[0115]

[0116] In formula (4), G w (D1′) is the first feature vector, which represents the first sample image feature; G w (D2′) is the second feature vector, which represents the second sample image feature; e It means G w The number of elements in (D1′) that are not equal to 0; f It means G w The number of elements in (D2′) that are not equal to 0; ef It means G w (D1′) and Gw The number of elements in the difference of (D2′) that are not equal to 0.

[0117] When the vehicle controller determines the sample image feature similarity between the first sample image feature and the second sample image feature by combining distance similarity and cosine similarity, the following formula (5) may be used.

[0118] S(G w (D1′), G w (D2′))=dis(G w (D1′), G w (D2′))·cos(G w (D1′), G w (D2′)) (5)

[0119] In formula (5), G w (D1′) is the first feature vector, which represents the first sample image feature; G w (D2′) is the second feature vector, which represents the second sample image feature; dis() represents the distance similarity; and cos() represents the cosine similarity.

[0120] In this embodiment, the preset value interval may be [0, 1]. Optionally, the vehicle controller converts the sample image feature similarity into the sample image feature similarity probability within the preset value interval, which may be achieved by using Figure 5 The fully connected layer and sigmoid layer in the twin network shown convert the sample image feature similarity into a numerical value in the interval [0,1], namely RPS, and use RPS as the sample image feature similarity probability.

[0121] It can be understood that if the probability of similarity of sample image features is higher, it means that the recognition result of the first classification network in the twin network for the target object's to-be-identified behavior in the first sample image (full image) is more consistent with the recognition result of the second classification network for the target object's to-be-identified behavior in the second sample image (partial image).

[0122] In this implementation, the vehicle controller may use the following formula (6) to determine the first loss value between the first sample image and the second sample image based on the similarity probability of the sample image features.

[0123]

[0124] In formula (6), L rpsIt represents the first loss value, that is, the loss value between the first sample image and the second sample image; RPS represents the probability of similarity of sample image features; λ represents the loss weight; l represents the label of the sample pair, if the sample pair is a positive sample pair, l is 1, if the sample pair is a negative sample pair, l is 0; It represents the regularization function.

[0125] In an optional embodiment, the second loss value and the third loss value can be determined by the vehicle controller in the following manner: inputting the first sample image feature into the first classification network included in the twin network to be trained to obtain the first predicted behavior of the sample object; based on the first loss function, determining the second loss value between the first predicted behavior and the actual behavior of the sample object; and, inputting the second sample image feature into the second classification network included in the twin network to be trained to obtain the second predicted behavior of the sample object; based on the second loss function, determining the third loss value between the second predicted behavior and the actual behavior.

[0126] Optionally, the loss function may include but is not limited to a mean square error loss function, a mean absolute error loss function, or a cross entropy loss function. The first loss function and the second loss function may be the same or different. For example, the first loss function and the second loss function are both mean square error loss functions or both are cross entropy loss functions; for another example, the first loss function is a mean square error loss function, and the second loss function is a cross entropy loss function; for another example, the first loss function is a cross entropy loss function, and the second loss function is a mean square error loss function.

[0127] Step 4: When the total loss values ​​of multiple sample pairs do not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample pairs meet the conditions for stopping training to obtain the twin network.

[0128] With this implementation, the vehicle controller can train the twin network based on the total loss value of each sample pair in a plurality of sample pairs, wherein the total loss value of each sample pair is composed of the first loss value between the first sample image and the second sample image in the sample pair, the second loss value between the first predicted behavior of the sample object and the actual behavior of the sample object, and the third loss value between the second predicted behavior of the sample object and the actual behavior; thereafter, when the total loss values ​​of the plurality of sample pairs do not meet the stop training condition, the network parameters of the twin network are updated based on the total loss value of each sample pair, and the twin network is trained again until the total loss values ​​of the plurality of sample pairs meet the stop training condition. In this way, when the object behavior in the vehicle cabin is subsequently identified based on the trained twin network, the accuracy of the object behavior identification in the vehicle cabin can be improved.

[0129] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0130] Based on the same inventive concept, the embodiment of the present application also provides a vehicle cabin object behavior recognition device for implementing the vehicle cabin object behavior recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more vehicle cabin object behavior recognition device embodiments provided below can refer to the limitations of the vehicle cabin object behavior recognition method above, and will not be repeated here.

[0131] See also Figure 6 , Figure 6 Schematic diagram of the structure of an object behavior recognition device in a vehicle cabin provided by an embodiment of the present application. Figure 6 As shown, the object behavior recognition device in the vehicle cabin may include but is not limited to:

[0132] The acquisition and determination module 601 is used to acquire a first image of a target object in the cabin, and determine a second image based on the first image; the second image is a local image corresponding to a target area in the first image, and the target area is an area related to the behavior to be identified of the target object;

[0133] A processing module 602 is used to input the first image and the second image into the twin network to obtain a similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image;

[0134] A determination module 603 is used to determine the recognition result of the behavior to be recognized for the target object based on the similarity probability;

[0135] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0136] In one embodiment, when the processing module 602 is used to input the first image and the second image into the twin network to obtain the similarity probability between the first image feature and the second image feature, it is specifically used to: input the first image into the first feature extraction network included in the twin network to obtain the first image feature, and, input the second image into the second feature extraction network included in the twin network to obtain the second image feature; determine the similarity between the first image feature and the second image feature; convert the similarity into a target value within a preset value range, and use the target value as the similarity probability between the first image feature and the second image feature.

[0137] In one embodiment, when the determination module 603 is used to determine the recognition result of the behavior to be recognized for the target object based on the similarity probability, it is specifically used to: when it is determined that the similarity probability is greater than a preset threshold, determine that the recognition result of the behavior to be recognized for the target object is the occurrence of the behavior to be recognized; when it is determined that the similarity probability is less than or equal to the preset threshold, determine that the recognition result of the behavior to be recognized for the target object is that the behavior to be recognized does not occur.

[0138] In one embodiment, the device also includes an output module, which is used to output prompt information after determining that the recognition result of the behavior to be identified for the target object is the occurrence of the behavior to be identified when it is determined that the similarity probability is greater than a preset threshold. The prompt information is used to indicate that the recognition result of the behavior to be identified for the target object is the occurrence of the behavior to be identified.

[0139] In one embodiment, the device also includes a training module; the training module is used to input the first sample image in the sample pair into the first feature extraction network included in the twin network to be trained for each sample pair to obtain the first sample image feature, and input the second sample image in the sample pair into the second feature extraction network included in the twin network to be trained to obtain the second sample image feature; determine the total loss value of the sample pair; the total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the loss value between the first sample image and the second sample image The loss value between the first predicted behavior of the object and the actual behavior of the sample object, the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when the total loss values ​​of multiple sample pairs do not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample pairs meet the conditions for stopping training to obtain the twin network.

[0140] In one embodiment, the determination module 603 is also used to: determine the sample image feature similarity between the first sample image feature and the second sample image feature; convert the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determine a first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0141] In one embodiment, the processing module 602 is also used to input the first sample image feature into the first classification network included in the twin network to be trained, so as to obtain the first predicted behavior of the sample object; the determination module 603 is also used to determine the second loss value between the first predicted behavior and the actual behavior of the sample object based on the first loss function; and the processing module 602 is also used to input the second sample image feature into the second classification network included in the twin network to be trained, so as to obtain the second predicted behavior of the sample object; the determination module 603 is also used to determine the third loss value between the second predicted behavior and the actual behavior based on the second loss function.

[0142] In one embodiment, when the acquisition and determination module 601 is used to determine the second image based on the first image, it is specifically used to: input the first image into a face detection model, and when it is detected that the target object is contained in the first image, output the face information corresponding to the target object; based on the face information, capture the face image corresponding to the target object from the first image; input the face image into a face key point detection model to obtain the face key point coordinates of the target object; determine the target area based on the face key point coordinates, and obtain the second image corresponding to the target area from the face image.

[0143] Each module in the above-mentioned object behavior recognition device in the vehicle cabin can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a vehicle control device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0144] In an exemplary embodiment, a vehicle control device is provided, which can be applied to a vehicle. The vehicle control device can be a terminal, and its internal structure diagram can be as shown in FIG. Figure 7 As shown. The vehicle control device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the vehicle control device is used to provide computing and control capabilities. The memory of the vehicle control device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the vehicle control device is used to exchange information between the processor and the external device. The communication interface of the vehicle control device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, the steps in a method for identifying object behavior in a vehicle cabin are implemented. The display unit of the vehicle control device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the vehicle control device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the shell of the vehicle control device, or an external keyboard, touchpad, etc.

[0145] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the vehicle control device or server to which the scheme of the present application is applied. The specific vehicle control device or server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0146] In an exemplary embodiment, the present application provides a vehicle control device, including a memory and a processor, wherein the memory stores a computer program; when the processor executes the computer program, the following steps are implemented:

[0147] Acquire a first image of a target object in the cockpit, and determine a second image based on the first image; the second image is a partial image corresponding to a target area in the first image, and the target area is an area related to the behavior to be identified of the target object;

[0148] Input the first image and the second image into the twin network to obtain the similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image;

[0149] Based on the similarity probability, determining the recognition result of the behavior to be recognized for the target object;

[0150] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0151] In one embodiment, when the processor executes a computer program to input the first image and the second image into the twin network and obtain the similarity probability between the first image feature and the second image feature, the following steps are specifically implemented: input the first image into the first feature extraction network included in the twin network to obtain the first image feature, and input the second image into the second feature extraction network included in the twin network to obtain the second image feature; determine the similarity between the first image feature and the second image feature; convert the similarity into a target value within a preset value range, and use the target value as the similarity probability between the first image feature and the second image feature.

[0152] In one embodiment, when the processor executes a computer program to determine the recognition result of the behavior to be recognized for the target object based on the similarity probability, the following steps are specifically implemented: when it is determined that the similarity probability is greater than a preset threshold, the recognition result of the behavior to be recognized for the target object is determined as the occurrence of the behavior to be recognized; when it is determined that the similarity probability is less than or equal to the preset threshold, the recognition result of the behavior to be recognized for the target object is determined as the non-occurrence of the behavior to be recognized.

[0153] In one embodiment, the processor executes a computer program to implement the following steps after determining that the recognition result of the behavior to be identified for the target object is the occurrence of the behavior to be identified when it is determined that the similarity probability is greater than a preset threshold: outputting prompt information, wherein the prompt information is used to indicate that the recognition result of the behavior to be identified for the target object is the occurrence of the behavior to be identified.

[0154] In one embodiment, when the processor executes the computer program, the following steps are also implemented: for each sample pair, the first sample image in the sample pair is input into the first feature extraction network included in the twin network to be trained to obtain the first sample image feature, and the second sample image in the sample pair is input into the second feature extraction network included in the twin network to be trained to obtain the second sample image feature; the total loss value of the sample pair is determined; the total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the loss value between the first sample image and the second sample image. The loss value between the first predicted behavior of the object and the actual behavior of the sample object, the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when the total loss values ​​of multiple sample pairs do not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample pairs meet the conditions for stopping training to obtain the twin network.

[0155] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining a first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0156] In one embodiment, when the processor executes the computer program, the following steps are also implemented: inputting the first sample image feature into the first classification network included in the twin network to be trained to obtain the first predicted behavior of the sample object; based on the first loss function, determining the second loss value between the first predicted behavior and the actual behavior of the sample object; and, inputting the second sample image feature into the second classification network included in the twin network to be trained to obtain the second predicted behavior of the sample object; based on the second loss function, determining the third loss value between the second predicted behavior and the actual behavior.

[0157] In one embodiment, when a processor executes a computer program to determine a second image based on a first image, the following steps are specifically implemented: the first image is input into a face detection model, and when it is detected that the first image contains a target object, the face information corresponding to the target object is output; based on the face information, a face image corresponding to the target object is captured from the first image; the face image is input into a face key point detection model to obtain the face key point coordinates of the target object; the target area is determined based on the face key point coordinates, and a second image corresponding to the target area is obtained from the face image.

[0158] In an exemplary embodiment, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0159] Acquire a first image of a target object in the cockpit, and determine a second image based on the first image; the second image is a partial image corresponding to a target area in the first image, and the target area is an area related to the behavior to be identified of the target object;

[0160] Input the first image and the second image into the twin network to obtain the similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image;

[0161] Based on the similarity probability, determining the recognition result of the behavior to be recognized for the target object;

[0162] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0163] In one embodiment, when a computer program is executed by a processor to input a first image and a second image into a twin network to obtain a similarity probability between a first image feature and a second image feature, the following steps are specifically implemented: inputting the first image into a first feature extraction network included in the twin network to obtain a first image feature, and inputting the second image into a second feature extraction network included in the twin network to obtain a second image feature; determining the similarity between the first image feature and the second image feature; converting the similarity into a target value within a preset value range, and using the target value as the similarity probability between the first image feature and the second image feature.

[0164] In one embodiment, when a computer program is executed by a processor to determine the recognition result of the behavior to be recognized for the target object based on the similarity probability, the following steps are specifically implemented: when it is determined that the similarity probability is greater than a preset threshold, the recognition result of the behavior to be recognized for the target object is determined to be the occurrence of the behavior to be recognized; when it is determined that the similarity probability is less than or equal to the preset threshold, the recognition result of the behavior to be recognized for the target object is determined to be the non-occurrence of the behavior to be recognized.

[0165] In one embodiment, the computer program is executed by a processor to implement the following steps after determining that the similarity probability is greater than a preset threshold and that the recognition result of the behavior to be identified for the target object is the occurrence of the behavior to be identified: outputting prompt information, wherein the prompt information is used to indicate that the recognition result of the behavior to be identified for the target object is the occurrence of the behavior to be identified.

[0166] In one embodiment, the computer program is executed by the processor to implement the following steps: for each sample pair, the first sample image in the sample pair is input into the first feature extraction network included in the twin network to be trained to obtain the first sample image feature, and the second sample image in the sample pair is input into the second feature extraction network included in the twin network to be trained to obtain the second sample image feature; the total loss value of the sample pair is determined; the total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the loss value between the first sample image and the second sample image. The loss value between the first predicted behavior of the object and the actual behavior of the sample object, the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when the total loss values ​​of multiple sample pairs do not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample pairs meet the conditions for stopping training to obtain the twin network.

[0167] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining a first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0168] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the first sample image feature into the first classification network included in the twin network to be trained to obtain the first predicted behavior of the sample object; based on the first loss function, determining the second loss value between the first predicted behavior and the actual behavior of the sample object; and, inputting the second sample image feature into the second classification network included in the twin network to be trained to obtain the second predicted behavior of the sample object; based on the second loss function, determining the third loss value between the second predicted behavior and the actual behavior.

[0169] In one embodiment, when a computer program is executed by a processor to determine a second image based on a first image, the following steps are specifically implemented: the first image is input into a face detection model, and when it is detected that the first image contains a target object, the face information corresponding to the target object is output; based on the face information, a face image corresponding to the target object is captured from the first image; the face image is input into a face key point detection model to obtain the face key point coordinates of the target object; the target area is determined based on the face key point coordinates, and a second image corresponding to the target area is obtained from the face image.

[0170] In an exemplary embodiment, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0171] Acquire a first image of a target object in the cockpit, and determine a second image based on the first image; the second image is a partial image corresponding to a target area in the first image, and the target area is an area related to the behavior to be identified of the target object;

[0172] Input the first image and the second image into the twin network to obtain the similarity probability between the first image feature and the second image feature; the first image feature is the image feature of the first image, and the second image feature is the image feature of the second image;

[0173] Based on the similarity probability, determining the recognition result of the behavior to be recognized for the target object;

[0174] Among them, the twin network is trained using multiple sample pairs; each sample pair is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

[0175] In one embodiment, when a computer program is executed by a processor to input a first image and a second image into a twin network to obtain a similarity probability between a first image feature and a second image feature, the following steps are specifically implemented: inputting the first image into a first feature extraction network included in the twin network to obtain a first image feature, and inputting the second image into a second feature extraction network included in the twin network to obtain a second image feature; determining the similarity between the first image feature and the second image feature; converting the similarity into a target value within a preset value range, and using the target value as the similarity probability between the first image feature and the second image feature.

[0176] In one embodiment, when a computer program is executed by a processor to determine the recognition result of the behavior to be recognized for the target object based on the similarity probability, the following steps are specifically implemented: when it is determined that the similarity probability is greater than a preset threshold, the recognition result of the behavior to be recognized for the target object is determined to be the occurrence of the behavior to be recognized; when it is determined that the similarity probability is less than or equal to the preset threshold, the recognition result of the behavior to be recognized for the target object is determined to be the non-occurrence of the behavior to be recognized.

[0177] In one embodiment, the computer program is executed by a processor to implement the following steps after determining that the similarity probability is greater than a preset threshold and that the recognition result of the behavior to be identified for the target object is the occurrence of the behavior to be identified: outputting prompt information, wherein the prompt information is used to indicate that the recognition result of the behavior to be identified for the target object is the occurrence of the behavior to be identified.

[0178] In one embodiment, the computer program is executed by the processor to implement the following steps: for each sample pair, the first sample image in the sample pair is input into the first feature extraction network included in the twin network to be trained to obtain the first sample image feature, and the second sample image in the sample pair is input into the second feature extraction network included in the twin network to be trained to obtain the second sample image feature; the total loss value of the sample pair is determined; the total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is the loss value between the first sample image and the second sample image, and the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the loss value between the first sample image and the second sample image. The loss value between the first predicted behavior of the object and the actual behavior of the sample object, the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the twin network to be trained; the third loss value is the loss value between the second predicted behavior and the actual behavior of the sample object, the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the twin network to be trained; when the total loss values ​​of multiple sample pairs do not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample pair, and the twin network to be trained is trained again until the total loss values ​​of multiple sample pairs meet the conditions for stopping training to obtain the twin network.

[0179] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: determining the sample image feature similarity between the first sample image feature and the second sample image feature; converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; and determining a first loss value between the first sample image and the second sample image based on the sample image feature similarity probability.

[0180] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the first sample image feature into the first classification network included in the twin network to be trained to obtain the first predicted behavior of the sample object; based on the first loss function, determining the second loss value between the first predicted behavior and the actual behavior of the sample object; and, inputting the second sample image feature into the second classification network included in the twin network to be trained to obtain the second predicted behavior of the sample object; based on the second loss function, determining the third loss value between the second predicted behavior and the actual behavior.

[0181] In one embodiment, when a computer program is executed by a processor to determine a second image based on a first image, the following steps are specifically implemented: the first image is input into a face detection model, and when it is detected that the first image contains a target object, the face information corresponding to the target object is output; based on the face information, a face image corresponding to the target object is captured from the first image; the face image is input into a face key point detection model to obtain the face key point coordinates of the target object; the target area is determined based on the face key point coordinates, and a second image corresponding to the target area is obtained from the face image.

[0182] It should be noted that the data involved in this application (including but not limited to the first image, the second image, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations. In addition, the images of user objects involved in this application are also all images authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant images must comply with relevant regulations.

[0183] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0184] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0185] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying object behavior in a vehicle cabin, characterized in that: The method comprises: Acquire a first image of a target object in the cabin, and determine a second image based on the first image; the second image is a local image corresponding to a target area in the first image, and the target area is an area related to the behavior to be identified of the target object; Inputting the first image and the second image into the twin network to obtain a similarity probability between a first image feature and a second image feature; the first image feature is an image feature of the first image, and the second image feature is an image feature of the second image; Based on the similarity probability, determining a recognition result of the to-be-recognized behavior of the target object; Wherein, the twin network is trained using multiple sample pairs; each of the sample pairs is composed of a first sample image and a second sample image corresponding to the sample object; the second sample image is a local image corresponding to the target area in the first sample image.

2. The method according to claim 1, characterized in that The step of inputting the first image and the second image into the twin network to obtain a similarity probability between features of the first image and features of the second image includes: Inputting the first image into a first feature extraction network included in the twin network to obtain a first image feature, and inputting the second image into a second feature extraction network included in the twin network to obtain a second image feature; determining a similarity between the first image feature and the second image feature; The similarity is converted into a target value within a preset value range, and the target value is used as the similarity probability between the first image feature and the second image feature.

3. The method according to claim 1, characterized in that The determining, based on the similarity probability, a recognition result of the to-be-recognized behavior of the target object includes: In the case where it is determined that the similarity probability is greater than a preset threshold, determining that a recognition result of the behavior to be recognized for the target object is that the behavior to be recognized occurs; When it is determined that the similarity probability is less than or equal to a preset threshold, it is determined that the recognition result of the to-be-recognized behavior for the target object is that the to-be-recognized behavior does not occur.

4. The method according to claim 3, characterized in that In the case where it is determined that the similarity probability is greater than a preset threshold, after determining that the recognition result of the behavior to be recognized for the target object is that the behavior to be recognized occurs, the method further includes: Output prompt information, where the prompt information is used to indicate that a recognition result of the behavior to be recognized for the target object is that the behavior to be recognized occurs.

5. The method according to any one of claims 1 to 4, characterized in that: The twin network is trained in the following way: For each of the sample pairs, the first sample image in the sample pair is input into a first feature extraction network included in the twin network to be trained to obtain a first sample image feature, and the second sample image in the sample pair is input into a second feature extraction network included in the twin network to be trained to obtain a second sample image feature; Determine the total loss value of the sample pair; the total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is a loss value between the first sample image and the second sample image, and the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is a loss value between a first predicted behavior of the sample object and an actual behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into a first classification network included in the twin network to be trained; the third loss value is a loss value between a second predicted behavior of the sample object and the actual behavior, and the second predicted behavior is obtained by inputting the second sample image feature into a second classification network included in the twin network to be trained; When the total loss values ​​of the multiple sample pairs do not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated based on the total loss value of each sample pair, and the twin network to be trained is trained again until the total loss values ​​of the multiple sample pairs meet the conditions for stopping training, thereby obtaining the twin network.

6. The method according to claim 5, characterized in that The first loss value is determined in the following manner: determining a sample image feature similarity between the first sample image feature and the second sample image feature; Converting the sample image feature similarity into a sample image feature similarity probability within a preset value range; Based on the sample image feature similarity probability, a first loss value between the first sample image and the second sample image is determined.

7. The method according to claim 5, characterized in that The second loss value and the third loss value are determined in the following manner: Inputting the first sample image feature into a first classification network included in the twin network to be trained to obtain a first predicted behavior of the sample object; determining a second loss value between the first predicted behavior and an actual behavior of the sample object based on a first loss function; as well as, The second sample image feature is input into the second classification network included in the twin network to be trained to obtain a second predicted behavior of the sample object; based on a second loss function, a third loss value between the second predicted behavior and the actual behavior is determined.

8. The method according to claim 1, characterized in that The determining the second image based on the first image comprises: Inputting the first image into a face detection model, and when detecting that the target object is contained in the first image, outputting face information corresponding to the target object; Based on the facial information, intercepting a facial image corresponding to the target object from the first image; Inputting the face image into a face key point detection model to obtain the face key point coordinates of the target object; The target area is determined based on the coordinates of the facial key points, and a second image corresponding to the target area is acquired from the facial image.

9. A vehicle, characterized in that: It comprises a vehicle processor and an image acquisition device in the cabin; the image acquisition device is used to acquire an image including a target object in the cabin; the vehicle processor is used to process the image to obtain a first image of the target object, and execute the method as described in any one of claims 1 to 8 to determine the recognition result of the behavior to be recognized for the target object.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images