Object behavior recognition method in vehicle cabin, vehicle and storage medium

Abstract

The invention discloses an object behavior recognition method in a vehicle cabin, a vehicle and a storage medium, and relates to the technical field of vehicles. The method comprises the following steps: acquiring a first image of a target object in a cabin, and determining a second image and a third image based on the first image; inputting the first image and the second image into a first sub-twinning network in the twinning network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into a second sub-twinning network in the twinning network to obtain a second similarity probability between the first image feature and the second image feature, obtaining a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image; and based on the first similarity probability, determining an identification result of a first to-be-identified behavior of the target object, and based on the second similarity probability, determining an identification result of a second to-be-identified behavior of the target object. By adopting the method, the waste of computing power resources of computing equipment can be reduced.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a method for identifying object behavior in a vehicle cabin, a vehicle, and a storage medium. Background Art

[0002] At present, the multi-object behavior recognition method in the vehicle cabin is mainly implemented in the following ways: face detection is performed on the main driver's seat image frame sequence. If a face is detected, the face key point regression is performed in the face region of interest (ROI); the image of the ROI area is intercepted according to the key points. For example, if it is necessary to determine whether the main driver's seat user is smoking, the image of the ROI area near the mouth is intercepted. At the same time, if it is necessary to determine whether the main driver's seat user is making a phone call, the image of the ROI area near the eyes is intercepted; then, the intercepted image of the ROI area near the mouth is input into the smoking image classification model to judge the smoking behavior of the main driver's seat user, and the intercepted image of the ROI area near the eyes is input into the phone call image classification model to judge the phone call behavior of the main driver's seat user. Among them, the smoking image classification model and the phone call image classification model are the same classification model, but two such classification models are required for deployment, which invisibly wastes computing resources.

[0003] Therefore, in the process of identifying multi-object behaviors in the vehicle cabin, how to reduce the waste of computing resources has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a method for identifying object behavior in a vehicle cabin, a vehicle, and a storage medium, which can reduce the waste of computing resources in the process of identifying multiple object behaviors in the vehicle cabin.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying object behavior in a vehicle cabin, the method comprising:

[0006] Acquire a first image of a target object in the cockpit, and determine a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object;

[0007] Inputting the first image and the second image into a first sub-twin network in the twin network, obtaining a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into a second sub-twin network in the twin network, obtaining a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image;

[0008] Determine a recognition result of a first behavior to be recognized of the target object based on the first similarity probability, and determine a recognition result of a second behavior to be recognized of the target object based on the second similarity probability;

[0009] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0010] In one embodiment, the first sub-twin network includes a first feature extraction network and a second feature extraction network, and the second sub-twin network includes a third feature extraction network and a fourth feature extraction network; the first image and the second image are input into the first sub-twin network in the twin network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and the first image and the third image are input into the second sub-twin network in the twin network to obtain a second similarity probability between the third image feature corresponding to the first image and the fourth image feature corresponding to the third image, including: inputting the first image and the second image into the first feature extraction network and the second feature extraction network respectively to obtain the first image corresponding to the first image. An image feature and a second image feature corresponding to a second image, and, respectively, inputting the first image and the third image into a third feature extraction network and a fourth feature extraction network to obtain a third image feature corresponding to the first image and a fourth image feature corresponding to the third image; determining a first similarity between the first image feature and the second image feature, and, determining a second similarity between the third image feature and the fourth image feature; converting the first similarity into a first target value within a preset value range, and using the first target value as a first similarity probability between the first image feature and the second image feature, and, converting the second similarity into a second target value within a preset value range, and using the second target value as a second similarity probability between the third image feature and the fourth image feature.

[0011] In one embodiment, based on a first similarity probability, a recognition result of a first behavior to be recognized for a target object is determined, and based on a second similarity probability, a recognition result of a second behavior to be recognized for a target object is determined, including: when it is determined that the first similarity probability is greater than a first preset threshold, determining that the recognition result of the first behavior to be recognized for the target object is the occurrence of the first behavior to be recognized; and when it is determined that the second similarity probability is greater than a second preset threshold, determining that the recognition result of the second behavior to be recognized for the target object is the occurrence of the second behavior to be recognized; wherein the first preset threshold and the second preset threshold are the same or different.

[0012] In one of the embodiments, the method further includes: outputting prompt information, where the prompt information is used to indicate recognition results of the first behavior to be recognized and the second behavior to be recognized for the target object.

[0013] In one embodiment, the twin network is trained in the following manner: for the first sample pair, the first sample image feature corresponding to the first sample image is obtained by the first feature extraction network included in the first sub-twin network in the twin network to be trained, and the second sample image feature corresponding to the second sample image is obtained by the second feature extraction network included in the first sub-twin network in the twin network to be trained; the first total loss value corresponding to the first sample pair is determined; wherein the first total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the first loss value of the sample object. A loss value between a predicted behavior and the actual first behavior of the sample object, the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained; a third loss value is a loss value between the second predicted behavior of the sample object and the actual first behavior, the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained; the first predicted behavior and the second predicted behavior are both prediction results for the first behavior to be identified; for the second sample pair, the third feature extraction network included in the second sub-twin network in the twin network to be trained is used. A third sample image feature corresponding to the first sample image is obtained, and a fourth sample image feature corresponding to the third sample image is obtained through a fourth feature extraction network included in the second sub-twin network in the twin network to be trained; a second total loss value corresponding to the second sample pair is determined; the second total loss value is composed of a fourth loss value, a fifth loss value and a sixth loss value; the fourth loss value is determined based on the third sample image feature and the fourth sample image feature; the fifth loss value is a loss value between a third predicted behavior of the sample object and an actual second behavior of the sample object, and the third predicted behavior is obtained by inputting the third sample image feature into the fourth feature extraction network included in the second sub-twin network in the twin network to be trained. The sixth loss value is the loss value between the fourth predicted behavior of the sample object and the actual second behavior. The fourth predicted behavior is obtained by inputting the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained; the third predicted behavior and the fourth predicted behavior are both prediction results for the second behavior to be identified; based on the first total loss value and the second total loss value, the total loss value corresponding to the twin network is determined; when it is determined that the total loss value does not meet the conditions for stopping training, the network parameters of the twin network are updated based on the total loss value, and the twin network is trained again until the total loss value meets the conditions for stopping training to obtain the twin network.

[0014] In one embodiment, the first loss value and the fourth loss value are determined in the following manner: determining the first sample image feature similarity between the first sample image feature and the second sample image feature; converting the first sample image feature similarity into a first sample image feature similarity probability within a preset value range; determining the first loss value between the first sample image and the second sample image based on the first sample image feature similarity probability; and determining the second sample image feature similarity between the third sample image feature and the fourth sample image feature; converting the second sample image feature similarity into a second sample image feature similarity probability within a preset value range; and determining the fourth loss value between the first sample image and the third sample image based on the second sample image feature similarity probability.

[0015] In one embodiment, the second loss value and the third loss value are determined in the following manner: inputting the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained, and obtaining the first predicted behavior of the sample object; based on the first loss function, determining the second loss value between the first predicted behavior and the actual first behavior of the sample object; and, inputting the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained, and obtaining the second predicted behavior of the sample object; based on the second loss function, determining the third loss value between the second predicted behavior and the actual first behavior.

[0016] In one embodiment, the fifth loss value and the sixth loss value are determined in the following manner: inputting the third sample image feature into the third classification network included in the second sub-twin network in the twin network to be trained, and obtaining the third predicted behavior of the sample object; based on the third loss function, determining the fifth loss value between the third predicted behavior and the actual second behavior of the sample object; and, inputting the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained, and obtaining the fourth predicted behavior of the sample object; based on the fourth loss function, determining the sixth loss value between the fourth predicted behavior and the actual second behavior of the sample object.

[0017] In one embodiment, determining a second image and a third image based on a first image includes: inputting the first image into a face detection model, and outputting face information corresponding to the target object when detecting that the target object is contained in the first image; based on the face information, capturing a face image corresponding to the target object from the first image; inputting the face image into a face key point detection model to obtain the face key point coordinates of the target object; determining a first area and a second area based on the face key point coordinates, and obtaining a second image corresponding to the first area and a third image corresponding to the second area from the face image.

[0018] In a second aspect, an embodiment of the present application provides a device for identifying object behavior in a vehicle cabin, the device comprising:

[0019] an acquisition and determination module, for acquiring a first image of a target object in the cockpit, and determining a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, the first area being an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, the second area being an area related to a second behavior to be identified of the target object;

[0020] A processing module, used for inputting the first image and the second image into a first sub-twin network in the twin network, obtaining a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into a second sub-twin network in the twin network, obtaining a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image;

[0021] A determination module, configured to determine a recognition result of a first behavior to be identified of a target object based on a first similarity probability, and to determine a recognition result of a second behavior to be identified of the target object based on a second similarity probability;

[0022] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0023] In a third aspect, an embodiment of the present application provides a vehicle control device, including a memory and a processor, wherein the memory stores a computer program; when the processor executes the computer program, the following steps are implemented:

[0024] Acquire a first image of a target object in the cockpit, and determine a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object;

[0025] Inputting the first image and the second image into a first sub-twin network in the twin network, obtaining a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into a second sub-twin network in the twin network, obtaining a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image;

[0026] Determine a recognition result of a first behavior to be recognized of the target object based on the first similarity probability, and determine a recognition result of a second behavior to be recognized of the target object based on the second similarity probability;

[0027] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0028] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0029] Acquire a first image of a target object in the cockpit, and determine a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object;

[0030] Inputting the first image and the second image into a first sub-twin network in the twin network, obtaining a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into a second sub-twin network in the twin network, obtaining a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image;

[0031] Determine a recognition result of a first behavior to be recognized of the target object based on the first similarity probability, and determine a recognition result of a second behavior to be recognized of the target object based on the second similarity probability;

[0032] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0033] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0034] Acquire a first image of a target object in the cockpit, and determine a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object;

[0035] Inputting the first image and the second image into a first sub-twin network in the twin network, obtaining a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into a second sub-twin network in the twin network, obtaining a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image;

[0036] Determine a recognition result of a first behavior to be recognized of the target object based on the first similarity probability, and determine a recognition result of a second behavior to be recognized of the target object based on the second similarity probability;

[0037] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0038] In the sixth aspect, the present application also provides a vehicle, including a vehicle processor and an image acquisition device; the image acquisition device is used to acquire images including a target object in a cockpit; the vehicle processor is used to process the image to obtain a first image of the target object, and execute the steps in the method designed in the first aspect to determine the recognition result of the behavior to be recognized for the target object.

[0039] The above-mentioned object behavior recognition method in the vehicle cabin, vehicle and storage medium, the vehicle can obtain a first image of the target object in the cabin, and determine the second image and the third image based on the first image; the second image is a local image corresponding to the first area in the first image, and the first area is an area related to the first behavior to be recognized of the target object; the third image is a local image corresponding to the second area in the first image, and the second area is an area related to the second behavior to be recognized of the target object; the first image and the second image are input into the first sub-twin network in the twin network to obtain a first similarity probability between the first image feature corresponding to the first image and the second image feature corresponding to the second image, and the first image and the third image are input into the first sub-twin network in the twin network. In the two-child twin network, a second similarity probability is obtained between the third image feature corresponding to the first image and the fourth image feature corresponding to the third image; based on the first similarity probability, the recognition result of the first behavior to be recognized for the target object is determined, and, based on the second similarity probability, the recognition result of the second behavior to be recognized for the target object is determined; wherein, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is a local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is a local image corresponding to the second area in the first sample image. By adopting this method, compared with the method of deploying two identical classification models when performing multi-object behavior recognition, the vehicle can use multi-task learning to input the first image (full image), the second image (the local image corresponding to the first area related to the first behavior to be recognized) and the third image (the local image corresponding to the first area related to the second behavior to be recognized) into the twin network, and obtain the first similarity probability corresponding to the first behavior to be recognized (that is, the similarity probability between the first image feature corresponding to the first image and the second image feature corresponding to the second image) and the second similarity probability corresponding to the second behavior to be recognized (that is, the third image feature corresponding to the first image and the fourth image feature corresponding to the third image). Then, based on the first similarity probability, the recognition result of the first behavior to be recognized is determined, and based on the second similarity probability, the recognition result of the second behavior to be recognized is determined, thereby reducing the waste of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is a schematic diagram of an application scenario of a method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application;

[0042] Figure 2 It is a flow chart of a method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application;

[0043] Figure 3 is a flow chart of another method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application;

[0044] Figure 4 It is a flowchart of another method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application;

[0045] Figure 5 It is a structural diagram of a twin network provided in an embodiment of the present application;

[0046] Figure 6 It is a structural schematic diagram of an object behavior recognition device in a vehicle cabin provided by an embodiment of the present application;

[0047] Figure 7 It is a structural schematic diagram of a vehicle control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] First, combine Figure 1 The application scenarios of the object behavior recognition method in the vehicle cabin provided by the embodiment of the present application are introduced. Figure 1 , Figure 1 Schematic diagram of an application scenario of an object behavior recognition method in a vehicle cabin provided by an embodiment of the present application. Figure 1 As shown, the application scenario includes a vehicle 100 , the vehicle 100 includes a vehicle controller 101 (or also called a vehicle processor), and an image acquisition device 102 is included in the cockpit.

[0050] The vehicle controller 101 may first acquire an image of the target object in the cockpit through the image acquisition device 102, and then process the acquired image to obtain a first image of the target object, and store the first image. In the process of object behavior recognition, the vehicle controller 101 may obtain the first image of the target object, and determine the second image and the third image based on the first image, that is, obtain the first local image and the second local image of the target object from the local image of the target object; wherein the first local image is an image corresponding to the area in the first image related to the first behavior to be recognized of the target object, and the second local image is an image corresponding to the area in the first image related to the second behavior to be recognized of the target object; further, the vehicle controller 101 may input the first image and the second image into the first sub-twin network in the twin network, obtain the first similarity probability between the first image feature corresponding to the first image and the second image feature corresponding to the second image, and, The image and the third image are input into the second sub-twin network in the twin network to obtain the second similarity probability between the third image feature corresponding to the first image and the fourth image feature corresponding to the third image; finally, based on the first similarity probability, the recognition result of the first behavior to be recognized for the target object is determined, and, based on the second similarity probability, the recognition result of the second behavior to be recognized for the target object is determined; wherein the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, that is, the global image and the first local image of the sample object; the second sample pair is composed of the first sample image and the third sample image, that is, the global image and the second local image of the sample object. By adopting this method, compared with the method of deploying two identical classification models when performing multi-object behavior recognition, the vehicle can use multi-task learning to input the first image (full image), the second image (the local image corresponding to the first area related to the first behavior to be recognized) and the third image (the local image corresponding to the first area related to the second behavior to be recognized) into the twin network, and obtain the first similarity probability corresponding to the first behavior to be recognized (that is, the similarity probability between the first image feature corresponding to the first image and the second image feature corresponding to the second image) and the second similarity probability corresponding to the second behavior to be recognized (that is, the third image feature corresponding to the first image and the fourth image feature corresponding to the third image). Then, based on the first similarity probability, the recognition result of the first behavior to be recognized is determined, and based on the second similarity probability, the recognition result of the second behavior to be recognized is determined, thereby reducing the waste of computing resources.

[0051] The following is a detailed description of the object behavior recognition method in the vehicle cabin provided in an embodiment of the present application.

[0052] See also Figure 2 , Figure 2is a flow chart of a method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application. The method can be performed by a vehicle (e.g. Figure 1 The vehicle controller 101 in FIG. Figure 2 As shown, the object behavior recognition method in the vehicle cabin may include but is not limited to the following steps:

[0053] S201. Acquire a first image of a target object in a cockpit, and determine a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object.

[0054] The target object may be the driver or a passenger other than the driver. When the target object is the driver, the first image of the target object is the first image of the driver. Exemplarily, assuming that the first behavior to be identified is whether the driver smokes, the first area is an area related to the behavior of smoking, that is, the mouth area of ​​the target driver. In this case, the second image is a partial image corresponding to the mouth area in the first image of the driver. Assuming that the second behavior to be identified is whether the driver is making a phone call, the second area is an area related to making a phone call, such as the driver's eye area. In this case, the third image is a partial image corresponding to the eye area in the first image of the driver.

[0055] In an optional implementation, before step S201, the vehicle controller may also control an image acquisition device in the cockpit to acquire an image of the target object, and then process the acquired image to obtain a first image of the target object.

[0056] The image acquisition device is, for example, an infrared sensor. The image of the target object may be a single-channel infrared image of the target object in the cabin acquired by the infrared sensor. In this case, the vehicle controller may convert the single-channel infrared image into a three-channel infrared image, and use the three-channel infrared image as the first image of the target object.

[0057] S202. Input the first image and the second image into the first sub-twin network in the twin network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and input the first image and the third image into the second sub-twin network in the twin network to obtain a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image.

[0058] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0059] The first image feature corresponding to the first image and the third image feature corresponding to the first image are the same or different.

[0060] In an optional embodiment, the twin network is trained based on a first total loss value corresponding to the first sub-twin network and a second total loss value corresponding to the second sub-twin network; the first total loss value is obtained based on the image features of each sample image in the first sample pair; and the second total loss value is obtained based on the image features of each sample image in the second sample pair.

[0061] S203: Determine a recognition result of a first behavior to be recognized of the target object based on the first similarity probability, and determine a recognition result of a second behavior to be recognized of the target object based on the second similarity probability.

[0062] In an embodiment of the present application, a vehicle controller may obtain a first image of a target object in a cockpit, and determine a second image and a third image based on the first image; the second image is a local image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a local image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object; the first image and the second image are input into a first sub-twin network in the twin network, and a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image is obtained, and the first image and the third image are input into a second sub-twin network in the twin network. , obtain a second similarity probability between the third image feature corresponding to the first image and the fourth image feature corresponding to the third image; based on the first similarity probability, determine the recognition result of the first behavior to be recognized for the target object, and, based on the second similarity probability, determine the recognition result of the second behavior to be recognized for the target object; wherein the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is a local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is a local image corresponding to the second area in the first sample image. By adopting this method, compared with the method of inputting multiple local images of the target object (such as the image of the ROI area near the mouth and the image of the ROI area near the eyes of the face) into two target image classification models of the same type for object behavior recognition, the vehicle controller can use multi-task learning to input the first image (full image), the second image (the local image corresponding to the first area related to the first behavior to be recognized) and the third image (the local image corresponding to the first area related to the second behavior to be recognized) into the twin network to obtain the first similarity probability corresponding to the first behavior to be recognized (that is, the similarity probability between the first image feature corresponding to the first image and the second image feature corresponding to the second image) and the second similarity probability corresponding to the second behavior to be recognized (that is, the third image feature corresponding to the first image and the fourth image feature corresponding to the third image). Then, based on the first similarity probability, the recognition result of the first behavior to be recognized is determined, and based on the second similarity probability, the recognition result of the second behavior to be recognized is determined, thereby reducing the waste of computing resources.

[0063] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of another method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application. Figure 2 The object behavior recognition method shown in the vehicle cabin is different in that Figure 3The method shown specifically describes how the vehicle controller inputs the first image and the second image into the first sub-twin network in the twin network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputs the first image and the third image into the second sub-twin network in the twin network to obtain a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image. Figure 3 As shown, the object behavior recognition method in the vehicle cabin may include but is not limited to the following steps:

[0064] S301, obtaining a first image of a target object in a cockpit, and determining a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object.

[0065] In an optional implementation, the relevant description of step S301 can refer to the description of the aforementioned step S201, which will not be repeated here.

[0066] S302. Input the first image and the second image into the first feature extraction network and the second feature extraction network included in the first sub-twin network in the twin network respectively, and obtain the first image feature corresponding to the first image and the second image feature corresponding to the second image; and input the first image and the third image into the third feature extraction network and the fourth feature extraction network included in the second sub-twin network in the twin network respectively, and obtain the third image feature corresponding to the first image and the fourth image feature corresponding to the third image.

[0067] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0068] It can be understood that the first image feature corresponding to the first image and the third image feature corresponding to the first image are the same or different. Wherein, when the first feature extraction network and the third feature extraction network are the same, the first image feature corresponding to the first image and the third image feature corresponding to the first image are the same; when the first feature extraction network and the third feature extraction network are different, the first image feature corresponding to the first image and the third image feature corresponding to the first image are different.

[0069] Optionally, the first feature extraction network, the second feature extraction network, the third feature extraction network, and the fourth feature extraction network can all be convolutional neural networks. For example, the first feature extraction network and the third feature extraction network can both be target detection networks YOLO-V8, and the second feature extraction network and the fourth feature extraction network can both be lightweight neural networks MobileNet-V3. In this case, the first image feature corresponding to the first image is the same as the third image feature corresponding to the first image.

[0070] S303: Determine a first similarity between the first image feature and the second image feature, and determine a second similarity between the third image feature and the fourth image feature.

[0071] In an optional embodiment, the vehicle controller may determine the similarity between the first image feature and the second image feature by measuring the difference vector similarity (as shown in the following formula (1)), or by combining the distance similarity and the cosine similarity (as shown in the following formula (2)).

[0072]

[0073] In formula (1), G w (D1) is the first eigenvector, which represents the first image feature; G w (D2) is the second eigenvector, which represents the second image feature; e It means G w The number of elements in (D1) that are not equal to 0; f It means G w The number of elements in (D2) that are not equal to 0; ef It means G w (D1) and G w The number of elements in the difference of (D2) that are not equal to 0.

[0074] S(G w (D1), G w (D2))=dis(G w (D1), G w (D2))·cos(G w (D1), G w (D2)) (2)

[0075] In formula (2), G w (D1) is the first eigenvector, which represents the first image feature; G w (D2) is the second eigenvector, which represents the second image feature; dis(G w (D1),G w (D2)) represents Gw (D1) and G w (D2) distance similarity; cos(G w (D1),G w (D2)) represents G w (D1) and G w (D2) The cosine similarity between them.

[0076] In an optional embodiment, the vehicle controller may determine the similarity between the third image feature and the fourth image feature by measuring the difference vector similarity (as shown in the following formula (3)), or by combining the distance similarity and the cosine similarity (as shown in the following formula (4)).

[0077]

[0078] In formula (3), G w (D3) is the third eigenvector, which represents the third image feature; G w (D4) is the fourth eigenvector, which represents the fourth image feature; e It means G w The number of elements in (D3) that are not equal to 0; f It means G w The number of elements in (D4) that are not equal to 0; ef It means G w (D3) and G w The number of elements in the difference of (D4) that are not equal to 0.

[0079] S(G w (D3), G w (D4))=dis(G w (D3), G w (D4))·cos(G w (D3), G w (D4)) (4)

[0080] In formula (4), G w (D3) is the third eigenvector, which represents the third image feature; G w (D4) is the fourth eigenvector, which represents the fourth image feature; dis(G w (D3),G w (D4)) represents G w (D3) and G w (D4) distance similarity between; cos(G w (D3),G w (D4)) represents G w (D3) and G w(D4) The cosine similarity between them.

[0081] S304, converting the first similarity into a first target value within a preset value range, and using the first target value as a first similarity probability between the first image feature and the second image feature, and converting the second similarity into a second target value within a preset value range, and using the second target value as a second similarity probability between the third image feature and the fourth image feature.

[0082] Optionally, the preset value interval may be [0,1].

[0083] That is, after determining the first similarity between the first image feature and the second image feature, the vehicle controller may convert the determined similarity into a numerical value greater than or equal to 0 and less than or equal to 1, and use the converted numerical value as the first similarity probability between the first image feature and the second image feature. Similarly, after determining the second similarity between the third image feature and the fourth image feature, the vehicle controller may convert the determined similarity into a numerical value greater than or equal to 0 and less than or equal to 1, and use the converted numerical value as the second similarity probability between the first image feature and the third image feature.

[0084] In an optional embodiment, the vehicle controller converts the first similarity into a first target value within a preset value range, and converts the second similarity into a second target value within a preset value range. The first similarity can be converted into a value within the interval [0,1] through the fully connected layer and the sigmoid layer included in the first sub-twin network in the twin network, and the second similarity can be converted into a value within the interval [0,1] through the fully connected layer and the sigmoid layer included in the second sub-twin network in the twin network.

[0085] S305: Determine a recognition result of a first behavior to be recognized of the target object based on the first similarity probability, and determine a recognition result of a second behavior to be recognized of the target object based on the second similarity probability.

[0086] In an optional embodiment, the vehicle controller determines the recognition result of a first behavior to be identified for the target object based on a first similarity probability, and determines the recognition result of a second behavior to be identified for the target object based on a second similarity probability, which may include: when it is determined that the first similarity probability is greater than a first preset threshold, determining that the recognition result of the first behavior to be identified for the target object is the occurrence of the first behavior to be identified; and when it is determined that the second similarity probability is greater than a second preset threshold, determining that the recognition result of the second behavior to be identified for the target object is the occurrence of the second behavior to be identified; wherein the first preset threshold and the second preset threshold are the same or different.

[0087] In this embodiment, the vehicle controller may further output prompt information, where the prompt information is used to indicate the recognition results of the first behavior to be recognized and the second behavior to be recognized for the target object.

[0088] Exemplarily, assuming that the first preset threshold is 0.5, the first behavior to be identified is whether the target object is smoking, and assuming that the second preset threshold is 0.5, the second behavior to be identified is whether the target object is making a phone call. If the vehicle controller determines that the similarity probability between the first image feature and the second image feature is 0.86, the vehicle controller can further determine that the similarity probability 0.86 is greater than the first preset threshold 0.5. In this case, the vehicle controller can determine that the target object is smoking. At the same time, if the vehicle controller determines that the similarity probability between the third image feature and the fourth image feature is 0.64, the vehicle controller can further determine that the similarity probability 0.64 is less than the second preset threshold 0.5. In this case, the vehicle controller can determine that the target object is making a phone call.

[0089] In an embodiment of the present application, the vehicle controller can use a multi-task learning method to input the first image (full image), the second image (the local image corresponding to the first area related to the first behavior to be identified) and the third image (the local image corresponding to the first area related to the second behavior to be identified) into the twin network to obtain the first similarity probability corresponding to the first behavior to be identified (that is, the similarity probability between the first image feature corresponding to the first image and the second image feature corresponding to the second image) and the second similarity probability corresponding to the second behavior to be identified (that is, the third image feature corresponding to the first image and the fourth image feature corresponding to the third image). Then, based on the first similarity probability, the recognition result of the first behavior to be identified is determined, and based on the second similarity probability, the recognition result of the second behavior to be identified is determined. Therefore, compared with the method of inputting multiple local images of the target object (such as the image of the ROI area near the mouth and the image of the ROI area near the eyes of the human face) into two target image classification models of the same type for object behavior recognition, on the one hand, the waste of computing resources can be reduced. On the other hand, the vehicle controller inputs the full-image image, the local image corresponding to the first area related to the first behavior to be recognized, and the local image corresponding to the first area related to the second behavior to be recognized into the twin network, that is, by combining the full-image image features and multiple local image features corresponding to the target object to recognize multiple behaviors of the target object, the accuracy of object behavior recognition in the vehicle cabin can be improved.

[0090] See also Figure 4 , Figure 4 FIG. 1 is a flow chart of another method for identifying object behavior in a vehicle cabin provided by an embodiment of the present application. Figure 3 Compared with the method shown, Figure 4The method shown in FIG. 1 specifically describes how the vehicle controller determines the second image and the third image based on the first image. Figure 4 As shown, the object behavior recognition method in the vehicle cabin may include but is not limited to the following steps:

[0091] S401: Acquire a first image of a target object in a cockpit.

[0092] In an optional implementation, before step S201, the vehicle controller may also control an image acquisition device in the cockpit to acquire an image of the target object, and then process the acquired image to obtain a first image of the target object.

[0093] The image acquisition device is, for example, an infrared sensor. The image of the target object is, for example, a single-channel infrared image of the target object in the cabin acquired by the infrared sensor. In this case, the vehicle controller may convert the single-channel infrared image into a three-channel infrared image, and use the three-channel infrared image as the first image of the target object.

[0094] S402: Input the first image into a face detection model, and when it is detected that the first image contains a target object, output face information corresponding to the target object.

[0095] In an optional implementation, the face detection model may be a deep learning model or a machine learning model, which is not limited here.

[0096] S403: Based on the facial information, capture a facial image corresponding to the target object from the first image.

[0097] In an optional implementation, the facial information may include but is not limited to facial coordinates, facial height, and facial width.

[0098] In this embodiment, the vehicle controller captures the facial image corresponding to the target object from the first image based on the facial information, and may capture the facial image corresponding to the target object from the first image based on the facial coordinates, the height of the face, and the width of the face.

[0099] S404: Input the face image into a face key point detection model to obtain the coordinates of the face key points of the target object.

[0100] S405: Determine a first region and a second region based on the coordinates of the facial key points, and obtain a second image corresponding to the first region and a third image corresponding to the second region from the facial image.

[0101] The first area is an area related to a first behavior to be identified of the target object, and the second area is an area related to a second behavior to be identified of the target object.

[0102] Exemplarily, assuming that the first behavior to be identified is whether the target object smokes, the first area is an area related to the behavior of smoking, that is, the mouth area of ​​the target object. In this case, the second image is a partial image corresponding to the mouth area in the first image. Assuming that the second behavior to be identified is whether the target object makes a phone call, the second area is an area related to making a phone call, such as the eye area of ​​the target object. In this case, the third image is a partial image corresponding to the eye area in the first image.

[0103] S406. Input the first image and the second image into the first feature extraction network and the second feature extraction network included in the first sub-twin network in the twin network respectively, and obtain the first image feature corresponding to the first image and the second image feature corresponding to the second image; and input the first image and the third image into the third feature extraction network and the fourth feature extraction network included in the second sub-twin network in the twin network respectively, and obtain the third image feature corresponding to the first image and the fourth image feature corresponding to the third image.

[0104] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0105] S407: Determine a first similarity between the first image feature and the second image feature, and determine a second similarity between the third image feature and the fourth image feature.

[0106] S408, converting the first similarity into a first target value within a preset value range, and using the first target value as a first similarity probability between the first image feature and the second image feature, and converting the second similarity into a second target value within a preset value range, and using the second target value as a second similarity probability between the third image feature and the fourth image feature.

[0107] In an optional implementation, the relevant descriptions of steps S406 to S408 can refer to the descriptions of the aforementioned steps S302 to S304, and will not be repeated here.

[0108] S409: Determine a recognition result of a first behavior to be recognized of the target object based on the first similarity probability, and determine a recognition result of a second behavior to be recognized of the target object based on the second similarity probability.

[0109] In an optional embodiment, when the vehicle controller determines the recognition result of the first behavior to be identified for the target object based on the first similarity probability, and determines the recognition result of the second behavior to be identified for the target object based on the second similarity probability, the recognition result of the first behavior to be identified for the target object can be determined by comparing the first similarity probability with the first preset threshold, and the recognition result of the second behavior to be identified for the target object can be determined by comparing the second similarity probability with the second preset threshold. The first preset threshold and the second preset threshold are the same or different. The process of determining the recognition result of the first behavior to be identified for the target object and determining the recognition result of the second behavior to be identified for the target object is described below in the manner of embodiments 1.1 to 1.4.

[0110] Implementation 1.1: The first similarity probability is greater than a first preset threshold, and the second similarity probability is greater than a second preset threshold.

[0111] In this embodiment, the vehicle controller may determine that the recognition result of the first behavior to be identified for the target object is the occurrence of the first behavior to be identified when it is determined that the first similarity probability is greater than the first preset threshold; and determine that the recognition result of the second behavior to be identified for the target object is the occurrence of the second behavior to be identified when it is determined that the second similarity probability is greater than the second preset threshold. In this case, the vehicle controller may also output prompt information, which is used to indicate that the target object has the first behavior to be identified and the second behavior to be identified.

[0112] Implementation 1.2: The first similarity probability is greater than a first preset threshold, and the second similarity probability is less than or equal to a second preset threshold.

[0113] In this embodiment, the vehicle controller may determine that the recognition result of the first behavior to be identified for the target object is that the first behavior to be identified occurs when it is determined that the first similarity probability is greater than the first preset threshold value; and, when it is determined that the second similarity probability is less than or equal to the second preset threshold value, determine that the recognition result of the second behavior to be identified for the target object is that the second behavior to be identified does not occur. In this case, the vehicle controller may also output prompt information, which is used to indicate that the target object has the first behavior to be identified.

[0114] Implementation 1.3: The first similarity probability is less than or equal to a first preset threshold, and the second similarity probability is greater than a second preset threshold.

[0115] In this embodiment, the vehicle controller may determine that the recognition result of the first behavior to be identified for the target object is that the first behavior to be identified did not occur if the first similarity probability is less than or equal to the first preset threshold; and determine that the recognition result of the second behavior to be identified for the target object is that the second behavior to be identified occurred if the second similarity probability is greater than the second preset threshold. In this case, the vehicle controller may also output prompt information, which is used to indicate that the target object has the second behavior to be identified.

[0116] Implementation 1.4: The first similarity probability is less than or equal to a first preset threshold, and the second similarity probability is less than or equal to a second preset threshold.

[0117] In this embodiment, the vehicle controller may determine that the recognition result of the first behavior to be recognized for the target object is that the first behavior to be recognized has not occurred when the first similarity probability is determined to be less than or equal to the first preset threshold; and determine that the recognition result of the second behavior to be recognized for the target object is that the second behavior to be recognized has not occurred when the second similarity probability is determined to be less than or equal to the second preset threshold. In this case, the vehicle controller may perform steps S401 to S409 again without outputting prompt information.

[0118] In an embodiment of the present application, the vehicle controller may input a first image (full image) into a face detection model, and when it is detected that the target object is contained in the first image, output facial information corresponding to the target object; based on the facial information, capture a facial image corresponding to the target object from the first image; input the facial image into a facial key point detection model to obtain the facial key point coordinates of the target object; determine the target area based on the facial key point coordinates, and obtain from the facial image a second image (partial image) corresponding to the first area related to the first behavior to be identified and a third image (partial image) corresponding to the second area related to the second area to be identified; thereafter, obtain the full image corresponding to the target object through the twin network. The image features (i.e., the first image features and the third image features) and the image features of multiple local images (i.e., the second image features and the fourth image features) are determined, and the first similarity between the first image features and the second image features, and the second similarity between the third image features and the fourth image features are determined; then, based on the first similarity, the first similarity probability between the first image features and the second image features is determined, and based on the second similarity, the second similarity probability between the third image features and the fourth image features is determined; finally, based on the first similarity probability, the recognition result of the first behavior to be recognized for the target object is determined, and, based on the second similarity probability, the recognition result of the second behavior to be recognized for the target object is determined. Compared with the method of inputting multiple local images of the target object (such as the image of the ROI area near the mouth of the face and the image of the ROI area near the eyes) into two target image classification models of the same type for object behavior recognition, on the one hand, the multi-object behavior recognition task is combined into the twin network by using the multi-task learning method, which can reduce the waste of computing resources, and on the other hand, by combining the full-image image features and multiple local image features corresponding to the target object to recognize multiple behaviors of the target object, the accuracy of object behavior recognition in the vehicle cabin can be improved.

[0119] In an optional embodiment, Figure 2 , Figure 3 and Figure 4 The structure of the twin network mentioned in the method shown can be as follows Figure 5 As shown, Figure 5 Schematic diagram of a twin network structure provided in the embodiment of the present application. Figure 5 As shown, the twin network may include a first sub-twin network and a second sub-twin network, and the first sub-twin network includes a first feature extraction network ( Figure 5 The first feature extraction network is a convolutional neural network 1, and the second feature extraction network ( Figure 5 The second feature extraction network is drawn as a convolutional neural network 2), the first classification network and the second classification network; the second sub-twin network includes the third feature extraction network ( Figure 5The third feature extraction network is a convolutional neural network 3, and the fourth feature extraction network ( Figure 5 In the figure, the fourth feature extraction network is a convolutional neural network 4), the third classification network and the fourth classification network. Optionally, the first feature extraction network and the third feature extraction network can both be the target detection network YOLO-V8, and the second feature extraction network and the fourth feature extraction network can both be the lightweight neural network MobileNet-V3. Optionally, the twin network can be trained by the vehicle controller in the following way:

[0120] Step 1: Obtain a first sample pair and a second sample pair, and determine an actual first behavior of a sample object in each sample image in the first sample pair, and an actual second behavior of a sample object in each sample image in the second sample pair.

[0121] Step 2: For the first sample pair, the first sample image feature corresponding to the first sample image is obtained through the first feature extraction network included in the first sub-twin network in the twin network to be trained, and the second sample image feature corresponding to the second sample image is obtained through the second feature extraction network included in the first sub-twin network in the twin network to be trained.

[0122] Step three, determine the first total loss value corresponding to the first sample pair; wherein the first total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the loss value between the first predicted behavior of the sample object and the actual first behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained; the third loss value is the loss value between the second predicted behavior of the sample object and the actual first behavior, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained; the first predicted behavior and the second predicted behavior are both prediction results for the first behavior to be identified.

[0123] In an optional implementation, the vehicle controller may use the following formula (5) to determine the first total loss value corresponding to the first sample pair.

[0124]

[0125] In formula (5), Loss1 represents the first total loss value corresponding to the first sample pair; λ represents the loss weight; G w (D1′) is the first feature vector, which represents the first sample image feature; G w (D2′) is the second feature vector, which represents the second sample image feature; Lrps1 It represents the first loss value, that is, the loss value between the first sample image and the second sample image; L cls (G w (D1′) represents the second loss value, that is, the loss value between the first predicted behavior of the sample object and the actual first behavior of the sample object; L cls (G w (D2′) represents the third loss value, that is, the loss value between the second predicted behavior and the actual first behavior of the sample object.

[0126] In an optional embodiment, the first loss value (i.e., Lrps1) can be determined by the vehicle controller in the following manner: determining the first sample image feature similarity between the first sample image feature and the second sample image feature; converting the first sample image feature similarity into a first sample image feature similarity probability within a preset value range; and determining the first loss value between the first sample image and the second sample image based on the first sample image feature similarity probability.

[0127] In this embodiment, the similarity of the first sample image feature between the first sample image feature and the second sample image feature can be determined by the vehicle controller through a difference vector similarity measurement method, or it can be determined by the vehicle controller through a measurement method combining distance similarity and cosine similarity, which is not limited here.

[0128] The vehicle controller determines the first sample image feature similarity S(G) between the first sample image feature and the second sample image feature by measuring the difference vector similarity. w (D1′), G w (D2′)), the following formula (6) can be used.

[0129]

[0130] In formula (6), G w (D1′) is the first feature vector, which represents the first sample image feature; G w (D2′) is the second feature vector, which represents the second sample image feature; e It means G w The number of elements in (D1′) that are not equal to 0; f It means G w The number of elements in (D2′) that are not equal to 0; z ef It means G w (D1′) and G w The number of elements in the difference of (D2′) that are not equal to 0.

[0131] The vehicle controller determines the first sample image feature similarity S(G) between the first sample image feature and the second sample image feature by combining distance similarity and cosine similarity. w (D1′), G w (D2′)), the following formula (7) can be used.

[0132] S(G w (D1′), G w (D2′))=dis(G w (D1′), G w (D2′))·cos(G w (D1′), G w (D2′)) (7)

[0133] In formula (7), G w (D1′) is the first feature vector, which represents the first sample image feature; G w (D2′) is the second feature vector, which represents the second sample image feature; dis() represents the distance similarity; and cos() represents the cosine similarity.

[0134] In this implementation, the preset value interval may be [0, 1]. Optionally, the vehicle controller converts the first sample image feature similarity into the first sample image feature similarity probability within the preset value interval, which may be achieved by using Figure 5 The fully connected layer and the sigmoid layer included in the first sub-twin network in the shown twin network convert the feature similarity of the first sample image into a numerical value in the interval [0,1], namely RPS1, and use RPS1 as the feature similarity probability of the first sample image.

[0135] It can be understood that if the probability of similarity of the features of the first sample image is higher, it indicates that the recognition result of the first classification network in the twin network for the first behavior to be identified of the target object in the first sample image (full image) is more consistent with the recognition result of the second classification network for the first behavior to be identified of the target object in the second sample image (partial image).

[0136] In this implementation, the vehicle controller may use the following formula (8) to determine the first loss value between the first sample image and the second sample image based on the probability of feature similarity of the first sample image.

[0137]

[0138] In formula (8), L rps1It represents the first loss value, that is, the loss value between the first sample image and the second sample image; RPS1 represents the similarity probability of the first sample image feature; λ represents the loss weight; l represents the label of the sample pair, if the sample pair is a positive sample pair, l is 1, if the sample pair is a negative sample pair, l is 0; It represents the regularization function.

[0139] In an optional embodiment, the second loss value and the third loss value can be determined by the vehicle controller in the following manner: inputting the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained, and obtaining the first predicted behavior of the sample object; based on the first loss function, determining the second loss value between the first predicted behavior and the actual first behavior of the sample object; and, inputting the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained, and obtaining the second predicted behavior of the sample object; based on the second loss function, determining the third loss value between the second predicted behavior and the actual first behavior.

[0140] Optionally, the loss function may include but is not limited to a mean square error loss function, a mean absolute error loss function, or a cross entropy loss function. The first loss function and the second loss function may be the same or different. For example, the first loss function and the second loss function are both mean square error loss functions or both are cross entropy loss functions; for another example, the first loss function is a mean square error loss function, and the second loss function is a cross entropy loss function; for another example, the first loss function is a cross entropy loss function, and the second loss function is a mean square error loss function.

[0141] Step 4: For the second sample pair, the third sample image feature corresponding to the first sample image is obtained through the third feature extraction network included in the second sub-twin network in the twin network to be trained, and the fourth sample image feature corresponding to the third sample image is obtained through the fourth feature extraction network included in the second sub-twin network in the twin network to be trained.

[0142] Step 5. Determine the second total loss value corresponding to the second sample pair; the second total loss value is composed of the fourth loss value, the fifth loss value and the sixth loss value; the fourth loss value is determined based on the third sample image feature and the fourth sample image feature; the fifth loss value is the loss value between the third predicted behavior of the sample object and the actual second behavior of the sample object, and the third predicted behavior is obtained by inputting the third sample image feature into the third classification network included in the second sub-twin network in the twin network to be trained; the sixth loss value is the loss value between the fourth predicted behavior of the sample object and the actual second behavior, and the fourth predicted behavior is obtained by inputting the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained; the third predicted behavior and the fourth predicted behavior are both prediction results for the second behavior to be identified.

[0143] In an optional implementation, the vehicle controller may use the following formula (9) to determine the second total loss value Loss2 corresponding to the second sample pair.

[0144] Loss2=λ×(L cls (G w (D3′)+L cls (G w (D4′)))+(1-2λ)×L rps2 (9)

[0145] In formula (9), Loss2 represents the second total loss value corresponding to the second sample pair; λ represents the loss weight; G w (D3′) is the third eigenvector, which represents the third sample image feature; G w (D4′) is the fourth eigenvector, which represents the fourth sample image feature; L rps2 It represents the fourth loss value, that is, the loss value between the first sample image and the third sample image; L cls (G w (D3′) represents the fifth loss value, that is, the loss value between the third predicted behavior of the sample object and the actual second behavior of the sample object; L cls (G w (D4′) represents the sixth loss value, that is, the loss value between the fourth predicted behavior and the actual second behavior of the sample object.

[0146] In an optional embodiment, the fourth loss value (ie, L rps2) can be determined by the vehicle controller in the following manner: determining the second sample image feature similarity between the third sample image feature and the fourth sample image feature; converting the second sample image feature similarity into a second sample image feature similarity probability within a preset value range; and determining a fourth loss value between the first sample image and the third sample image based on the second sample image feature similarity probability.

[0147] In this embodiment, the similarity of the second sample image feature between the third sample image feature and the fourth sample image feature can be determined by the vehicle controller through a difference vector similarity measurement method, or it can be determined by the vehicle controller through a measurement method combining distance similarity and cosine similarity, which is not limited here.

[0148] When the vehicle controller determines the sample image feature similarity between the first sample image feature and the second sample image feature by measuring the difference vector similarity, the aforementioned formula (6) may be referred to, which is not described in detail here. When the vehicle controller determines the sample image feature similarity between the first sample image feature and the second sample image feature by measuring the distance similarity and the cosine similarity, the aforementioned formula (7) may be referred to, which is not described in detail here.

[0149] In this embodiment, the preset value interval may be [0, 1]. Optionally, the vehicle controller converts the second sample image feature similarity into the second sample image feature similarity probability within the preset value interval, which may be achieved by using Figure 5 The fully connected layer and the sigmoid layer included in the second sub-twin network in the shown twin network convert the feature similarity of the first sample image into a numerical value in the interval [0,1], namely RPS2, and use RPS2 as the feature similarity probability of the second sample image.

[0150] It can be understood that if the probability of similarity of the features of the second sample image is higher, it indicates that the recognition result of the third classification network in the twin network for the second behavior to be identified of the target object in the first sample image (full image) is more consistent with the recognition result of the fourth classification network for the second behavior to be identified of the target object in the third sample image (partial image).

[0151] In this implementation, the vehicle controller may use the following formula (10) to determine the fourth loss value between the first sample image and the third sample image based on the probability of feature similarity of the second sample image.

[0152]

[0153] In formula (10), L rps2represents the fourth loss value, that is, the loss value between the first sample image and the third sample image; RPS2 represents the similarity probability of the second sample image feature; λ represents the loss weight; l represents the label of the sample pair, if the sample pair is a positive sample pair, l is 1, if the sample pair is a negative sample pair, l is 0; It represents the regularization function.

[0154] In an optional embodiment, the fifth loss value and the sixth loss value can be determined by the vehicle controller in the following manner: input the third sample image feature into the third classification network included in the second sub-twin network in the twin network to be trained, and obtain the third predicted behavior of the sample object; based on the third loss function, determine the fifth loss value between the third predicted behavior and the actual second behavior of the sample object; and, input the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained, and obtain the fourth predicted behavior of the sample object; based on the fourth loss function, determine the sixth loss value between the fourth predicted behavior and the actual second behavior of the sample object.

[0155] Optionally, the loss function may include but is not limited to a mean square error loss function, a mean absolute error loss function, or a cross entropy loss function. The first loss function and the second loss function may be the same or different. For example, the third loss function and the fourth loss function are both mean square error loss functions or both are cross entropy loss functions; for another example, the third loss function is a mean square error loss function, and the fourth loss function is a cross entropy loss function; for another example, the third loss function is a cross entropy loss function, and the fourth loss function is a mean square error loss function.

[0156] Step 6: Based on the first total loss value and the second total loss value, determine the total loss value corresponding to the twin network.

[0157] In an optional embodiment, when the vehicle controller determines the total loss value L corresponding to the twin network based on the first total loss value and the second total loss value, the following formula (11) may be used.

[0158] L=Loss1+Loss2 (11)

[0159] In formula (11), Loss1 represents the first total loss value corresponding to the first sample pair, which can be determined by the aforementioned formula (5); Loss2 represents the second total loss value corresponding to the second sample pair, which can be determined by the aforementioned formula (9).

[0160] Step 7. When it is determined that the total loss value does not meet the conditions for stopping training, update the network parameters of the twin network based on the total loss value, and train the twin network again until the total loss value meets the conditions for stopping training to obtain the twin network.

[0161] With this implementation, the vehicle controller can train the twin network based on the first total loss value corresponding to the first sub-twin network and the second total loss value corresponding to the second sub-twin network in the twin network; wherein the first total loss value is obtained based on the image features of each sample image in the first sample pair; the second total loss value is obtained based on the image features of each sample image in the second sample pair; thereafter, when it is determined that the total loss value does not meet the conditions for stopping training, the network parameters of the twin network are updated based on the total loss value, and the twin network is trained again until the total loss value meets the conditions for stopping training. In this way, when the multi-object behavior in the vehicle cabin is subsequently identified based on the trained twin network, the accuracy of the multi-object behavior identification in the vehicle cabin can be improved.

[0162] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0163] Based on the same inventive concept, the embodiment of the present application also provides a vehicle cabin object behavior recognition device for implementing the vehicle cabin object behavior recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more vehicle cabin object behavior recognition device embodiments provided below can refer to the limitations of the vehicle cabin object behavior recognition method above, and will not be repeated here.

[0164] See also Figure 6 , Figure 6 Schematic diagram of the structure of an object behavior recognition device in a vehicle cabin provided by an embodiment of the present application. Figure 6 As shown, the object behavior recognition device in the vehicle cabin may include but is not limited to:

[0165] The acquisition and determination module 601 is used to acquire a first image of the target object in the cockpit, and determine a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object;

[0166] Processing module 602 is used to input the first image and the second image into a first sub-twin network in the twin network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and to input the first image and the third image into a second sub-twin network in the twin network to obtain a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image;

[0167] A determination module 603 is used to determine a recognition result of a first behavior to be recognized of the target object based on the first similarity probability, and to determine a recognition result of a second behavior to be recognized of the target object based on the second similarity probability;

[0168] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0169] In one embodiment, the first sub-twin network includes a first feature extraction network and a second feature extraction network, and the second sub-twin network includes a third feature extraction network and a fourth feature extraction network; the processing module 602 is used to input the first image and the second image into the first sub-twin network in the twin network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and to input the first image and the third image into the second sub-twin network in the twin network to obtain a second similarity probability between the third image feature corresponding to the first image and the fourth image feature corresponding to the third image. The processing module 602 is specifically used to: input the first image and the second image into the first feature extraction network and the second feature extraction network respectively to obtain the first image; The first image feature corresponding to the first image and the second image feature corresponding to the second image are obtained, and the first image and the third image are respectively input into the third feature extraction network and the fourth feature extraction network to obtain the third image feature corresponding to the first image and the fourth image feature corresponding to the third image; determine the first similarity between the first image feature and the second image feature, and determine the second similarity between the third image feature and the fourth image feature; convert the first similarity into a first target value within a preset value range, and use the first target value as the first similarity probability between the first image feature and the second image feature, and convert the second similarity into a second target value within a preset value range, and use the second target value as the second similarity probability between the third image feature and the fourth image feature.

[0170] In one embodiment, when the determination module 603 is used to determine the recognition result of a first behavior to be identified for a target object based on a first similarity probability, and to determine the recognition result of a second behavior to be identified for the target object based on a second similarity probability, it is specifically used to: when it is determined that the first similarity probability is greater than a first preset threshold, determine that the recognition result of the first behavior to be identified for the target object is the occurrence of the first behavior to be identified; and when it is determined that the second similarity probability is greater than a second preset threshold, determine that the recognition result of the second behavior to be identified for the target object is the occurrence of the second behavior to be identified; wherein the first preset threshold and the second preset threshold are the same or different.

[0171] In one embodiment, the device further includes an output module, wherein the output module is used to output prompt information, wherein the prompt information is used to indicate recognition results of the first behavior to be recognized and the second behavior to be recognized for the target object.

[0172] In one embodiment, the device also includes a training module; the training module is used to obtain, for the first sample pair, a first sample image feature corresponding to the first sample image through a first feature extraction network included in the first sub-twin network in the to-be-trained twin network, and obtain a second sample image feature corresponding to the second sample image through a second feature extraction network included in the first sub-twin network in the to-be-trained twin network; determine a first total loss value corresponding to the first sample pair; wherein the first total loss value is composed of a first loss value, a second loss value, and a third loss value; the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is a loss value of the sample object. The loss value between the first predicted behavior and the actual first behavior of the sample object, the first predicted behavior is obtained by inputting the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained; the third loss value is the loss value between the second predicted behavior of the sample object and the actual first behavior, the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained; the first predicted behavior and the second predicted behavior are both prediction results for the first behavior to be identified; for the second sample pair, the third feature extraction network included in the second sub-twin network in the twin network to be trained is used. A third sample image feature corresponding to the first sample image is obtained, and a fourth sample image feature corresponding to the third sample image is obtained through a fourth feature extraction network included in the second sub-twin network in the twin network to be trained; a second total loss value corresponding to the second sample pair is determined; the second total loss value is composed of a fourth loss value, a fifth loss value and a sixth loss value; the fourth loss value is determined based on the third sample image feature and the fourth sample image feature; the fifth loss value is a loss value between a third predicted behavior of the sample object and an actual second behavior of the sample object, and the third predicted behavior is obtained by inputting the third sample image feature into the fourth feature extraction network included in the second sub-twin network in the twin network to be trained. The sixth loss value is the loss value between the fourth predicted behavior of the sample object and the actual second behavior. The fourth predicted behavior is obtained by inputting the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained; the third predicted behavior and the fourth predicted behavior are both prediction results for the second behavior to be identified; based on the first total loss value and the second total loss value, the total loss value corresponding to the twin network is determined; when it is determined that the total loss value does not meet the conditions for stopping training, the network parameters of the twin network are updated based on the total loss value, and the twin network is trained again until the total loss value meets the conditions for stopping training to obtain the twin network.

[0173] In one embodiment, the determination module 603 is also used to: determine the first sample image feature similarity between the first sample image feature and the second sample image feature; convert the first sample image feature similarity into a first sample image feature similarity probability within a preset value range; determine the first loss value between the first sample image and the second sample image based on the first sample image feature similarity probability; and determine the second sample image feature similarity between the third sample image feature and the fourth sample image feature; convert the second sample image feature similarity into a second sample image feature similarity probability within a preset value range; and determine the fourth loss value between the first sample image and the third sample image based on the second sample image feature similarity probability.

[0174] In one embodiment, the processing module 602 is also used to input the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained, and obtain the first predicted behavior of the sample object; the determination module 603 is also used to determine the second loss value between the first predicted behavior and the actual first behavior of the sample object based on the first loss function; and the processing module 602 is also used to input the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained, and obtain the second predicted behavior of the sample object; the determination module 603 is also used to determine the third loss value between the second predicted behavior and the actual first behavior based on the second loss function.

[0175] In one embodiment, the processing module 602 is also used to input the third sample image feature into the third classification network included in the second sub-twin network in the twin network to be trained, and obtain the third predicted behavior of the sample object; the determination module 603 is also used to determine the fifth loss value between the third predicted behavior and the actual second behavior of the sample object based on the third loss function; and the processing module 602 is also used to input the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained, and obtain the fourth predicted behavior of the sample object; the determination module 603 is also used to determine the sixth loss value between the fourth predicted behavior and the actual second behavior of the sample object based on the fourth loss function.

[0176] In one embodiment, when the acquisition and determination module 601 is used to determine the second image and the third image based on the first image, it is specifically used to: input the first image into a face detection model, and when it is detected that the target object is contained in the first image, output the face information corresponding to the target object; based on the face information, capture the face image corresponding to the target object from the first image; input the face image into a face key point detection model to obtain the face key point coordinates of the target object; determine the first area and the second area based on the face key point coordinates, and obtain the second image corresponding to the first area and the third image corresponding to the second area from the face image.

[0177] Each module in the above-mentioned object behavior recognition device in the vehicle cabin can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a vehicle control device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0178] In an exemplary embodiment, a vehicle control device is provided, which can be applied to a vehicle. The vehicle control device can be a terminal, and its internal structure diagram can be as shown in FIG. Figure 7 As shown. The vehicle control device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the vehicle control device is used to provide computing and control capabilities. The memory of the vehicle control device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the vehicle control device is used to exchange information between the processor and the external device. The communication interface of the vehicle control device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, the steps in a method for identifying object behavior in a vehicle cabin are implemented. The display unit of the vehicle control device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the vehicle control device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the shell of the vehicle control device, or an external keyboard, touchpad, etc.

[0179] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the vehicle control device or server to which the scheme of the present application is applied. The specific vehicle control device or server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] In an exemplary embodiment, the present application provides a vehicle control device, including a memory and a processor, wherein the memory stores a computer program; when the processor executes the computer program, the following steps are implemented:

[0181] Acquire a first image of a target object in the cockpit, and determine a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object;

[0182] Inputting the first image and the second image into a first sub-twin network in the twin network, obtaining a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into a second sub-twin network in the twin network, obtaining a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image;

[0183] Determine a recognition result of a first behavior to be recognized of the target object based on the first similarity probability, and determine a recognition result of a second behavior to be recognized of the target object based on the second similarity probability;

[0184] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0185] In one embodiment, the first sub-twin network includes a first feature extraction network and a second feature extraction network, and the second sub-twin network includes a third feature extraction network and a fourth feature extraction network; the processor executes a computer program to input the first image and the second image into the first sub-twin network in the twin network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and input the first image and the third image into the second sub-twin network in the twin network to obtain a second similarity probability between the third image feature corresponding to the first image and the fourth image feature corresponding to the third image. The following steps are specifically implemented: the first image and the second image are respectively input into the first feature extraction network and the second feature extraction network to obtain A first image feature corresponding to the first image and a second image feature corresponding to the second image, and the first image and the third image are respectively input into a third feature extraction network and a fourth feature extraction network to obtain a third image feature corresponding to the first image and a fourth image feature corresponding to the third image; a first similarity between the first image feature and the second image feature is determined, and a second similarity between the third image feature and the fourth image feature is determined; the first similarity is converted into a first target value within a preset value range, and the first target value is used as a first similarity probability between the first image feature and the second image feature, and the second similarity is converted into a second target value within a preset value range, and the second target value is used as a second similarity probability between the third image feature and the fourth image feature.

[0186] In one embodiment, when a processor executes a computer program to determine a recognition result of a first behavior to be identified for a target object based on a first similarity probability, and to determine a recognition result of a second behavior to be identified for the target object based on a second similarity probability, the following steps are specifically implemented: when it is determined that the first similarity probability is greater than a first preset threshold, the recognition result of the first behavior to be identified for the target object is determined to be the occurrence of the first behavior to be identified; and when it is determined that the second similarity probability is greater than a second preset threshold, the recognition result of the second behavior to be identified for the target object is determined to be the occurrence of the second behavior to be identified; wherein the first preset threshold and the second preset threshold are the same or different.

[0187] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: outputting prompt information, where the prompt information is used to indicate the recognition results of the first behavior to be recognized and the second behavior to be recognized for the target object.

[0188] In one embodiment, when the processor executes the computer program, the following steps are also implemented: for the first sample pair, the first sample image feature corresponding to the first sample image is obtained through the first feature extraction network included in the first sub-twin network in the twin network to be trained, and the second sample image feature corresponding to the second sample image is obtained through the second feature extraction network included in the first sub-twin network in the twin network to be trained; determine the first total loss value corresponding to the first sample pair; wherein the first total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the sample object The first predicted behavior and the actual first behavior of the sample object are obtained by inputting the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained; the third loss value is the loss value between the second predicted behavior of the sample object and the actual first behavior, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained; the first predicted behavior and the second predicted behavior are both prediction results for the first behavior to be identified; for the second sample pair, the third feature extraction network included in the second sub-twin network in the twin network to be trained is used to extract the first prediction behavior of the sample object and the actual first behavior. The method comprises the following steps: obtaining a third sample image feature corresponding to the first sample image through a fourth feature extraction network included in the second sub-twin network in the twin network to be trained, and obtaining a fourth sample image feature corresponding to the third sample image through a fourth feature extraction network included in the second sub-twin network in the twin network to be trained; determining a second total loss value corresponding to the second sample pair; the second total loss value is composed of a fourth loss value, a fifth loss value and a sixth loss value; the fourth loss value is determined based on the third sample image feature and the fourth sample image feature; the fifth loss value is a loss value between a third predicted behavior of the sample object and an actual second behavior of the sample object, and the third predicted behavior is obtained by inputting the third sample image feature into the first sub-twin network included in the second sub-twin network in the twin network to be trained. The sixth loss value is the loss value between the fourth predicted behavior of the sample object and the actual second behavior. The fourth predicted behavior is obtained by inputting the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained; the third predicted behavior and the fourth predicted behavior are both prediction results for the second behavior to be identified; based on the first total loss value and the second total loss value, the total loss value corresponding to the twin network is determined; when it is determined that the total loss value does not meet the conditions for stopping training, the network parameters of the twin network are updated based on the total loss value, and the twin network is trained again until the total loss value meets the conditions for stopping training to obtain the twin network.

[0189] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining the first sample image feature similarity between the first sample image feature and the second sample image feature; converting the first sample image feature similarity into a first sample image feature similarity probability within a preset value range; determining a first loss value between the first sample image and the second sample image based on the first sample image feature similarity probability; and determining the second sample image feature similarity between the third sample image feature and the fourth sample image feature; converting the second sample image feature similarity into a second sample image feature similarity probability within a preset value range; and determining a fourth loss value between the first sample image and the third sample image based on the second sample image feature similarity probability.

[0190] In one embodiment, when the processor executes the computer program, the following steps are also implemented: inputting the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained, and obtaining the first predicted behavior of the sample object; based on the first loss function, determining the second loss value between the first predicted behavior and the actual first behavior of the sample object; and, inputting the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained, and obtaining the second predicted behavior of the sample object; based on the second loss function, determining the third loss value between the second predicted behavior and the actual first behavior.

[0191] In one embodiment, when the processor executes the computer program, the following steps are also implemented: inputting the third sample image feature into the third classification network included in the second sub-twin network in the twin network to be trained, and obtaining the third predicted behavior of the sample object; based on the third loss function, determining the fifth loss value between the third predicted behavior and the actual second behavior of the sample object; and, inputting the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained, and obtaining the fourth predicted behavior of the sample object; based on the fourth loss function, determining the sixth loss value between the fourth predicted behavior and the actual second behavior of the sample object.

[0192] In one embodiment, when a processor executes a computer program to determine a second image and a third image based on a first image, the following steps are specifically implemented: inputting the first image into a face detection model, and when it is detected that the target object is contained in the first image, outputting face information corresponding to the target object; based on the face information, capturing a face image corresponding to the target object from the first image; inputting the face image into a face key point detection model to obtain the face key point coordinates of the target object; determining a first area and a second area based on the face key point coordinates, and obtaining a second image corresponding to the first area and a third image corresponding to the second area from the face image.

[0193] In an exemplary embodiment, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0194] Acquire a first image of a target object in the cockpit, and determine a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object;

[0195] Inputting the first image and the second image into a first sub-twin network in the twin network, obtaining a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into a second sub-twin network in the twin network, obtaining a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image;

[0196] Determine a recognition result of a first behavior to be recognized of the target object based on the first similarity probability, and determine a recognition result of a second behavior to be recognized of the target object based on the second similarity probability;

[0197] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0198] In one embodiment, the first sub-twin network includes a first feature extraction network and a second feature extraction network, and the second sub-twin network includes a third feature extraction network and a fourth feature extraction network; the computer program is executed by the processor to input the first image and the second image into the first sub-twin network in the twin network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and input the first image and the third image into the second sub-twin network in the twin network to obtain a second similarity probability between the third image feature corresponding to the first image and the fourth image feature corresponding to the third image. The following steps are specifically implemented: the first image and the second image are respectively input into the first feature extraction network and the second feature extraction network to obtain A first image feature corresponding to the first image and a second image feature corresponding to the second image, and the first image and the third image are respectively input into a third feature extraction network and a fourth feature extraction network to obtain a third image feature corresponding to the first image and a fourth image feature corresponding to the third image; a first similarity between the first image feature and the second image feature is determined, and a second similarity between the third image feature and the fourth image feature is determined; the first similarity is converted into a first target value within a preset value range, and the first target value is used as a first similarity probability between the first image feature and the second image feature, and the second similarity is converted into a second target value within a preset value range, and the second target value is used as a second similarity probability between the third image feature and the fourth image feature.

[0199] In one embodiment, when a computer program is executed by a processor to determine a recognition result of a first behavior to be identified for a target object based on a first similarity probability, and to determine a recognition result of a second behavior to be identified for the target object based on a second similarity probability, the following steps are specifically implemented: when it is determined that the first similarity probability is greater than a first preset threshold, the recognition result of the first behavior to be identified for the target object is determined to be the occurrence of the first behavior to be identified; and when it is determined that the second similarity probability is greater than a second preset threshold, the recognition result of the second behavior to be identified for the target object is determined to be the occurrence of the second behavior to be identified; wherein the first preset threshold and the second preset threshold are the same or different.

[0200] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: outputting prompt information, where the prompt information is used to indicate the recognition results of the first behavior to be recognized and the second behavior to be recognized for the target object.

[0201] In one embodiment, the computer program is executed by the processor to implement the following steps: for the first sample pair, the first sample image feature corresponding to the first sample image is obtained through the first feature extraction network included in the first sub-twin network in the twin network to be trained, and the second sample image feature corresponding to the second sample image is obtained through the second feature extraction network included in the first sub-twin network in the twin network to be trained; determine the first total loss value corresponding to the first sample pair; wherein the first total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the sample object The first predicted behavior and the actual first behavior of the sample object are obtained by inputting the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained; the third loss value is the loss value between the second predicted behavior of the sample object and the actual first behavior, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained; the first predicted behavior and the second predicted behavior are both prediction results for the first behavior to be identified; for the second sample pair, the third feature extraction network included in the second sub-twin network in the twin network to be trained is used to extract the first prediction behavior of the sample object and the actual first behavior. The method comprises the following steps: obtaining a third sample image feature corresponding to the first sample image through a fourth feature extraction network included in the second sub-twin network in the twin network to be trained, and obtaining a fourth sample image feature corresponding to the third sample image through a fourth feature extraction network included in the second sub-twin network in the twin network to be trained; determining a second total loss value corresponding to the second sample pair; the second total loss value is composed of a fourth loss value, a fifth loss value and a sixth loss value; the fourth loss value is determined based on the third sample image feature and the fourth sample image feature; the fifth loss value is a loss value between a third predicted behavior of the sample object and an actual second behavior of the sample object, and the third predicted behavior is obtained by inputting the third sample image feature into the first sub-twin network included in the second sub-twin network in the twin network to be trained. The sixth loss value is the loss value between the fourth predicted behavior of the sample object and the actual second behavior. The fourth predicted behavior is obtained by inputting the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained; the third predicted behavior and the fourth predicted behavior are both prediction results for the second behavior to be identified; based on the first total loss value and the second total loss value, the total loss value corresponding to the twin network is determined; when it is determined that the total loss value does not meet the conditions for stopping training, the network parameters of the twin network are updated based on the total loss value, and the twin network is trained again until the total loss value meets the conditions for stopping training to obtain the twin network.

[0202] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: determining the first sample image feature similarity between the first sample image feature and the second sample image feature; converting the first sample image feature similarity into a first sample image feature similarity probability within a preset value range; determining a first loss value between the first sample image and the second sample image based on the first sample image feature similarity probability; and determining the second sample image feature similarity between the third sample image feature and the fourth sample image feature; converting the second sample image feature similarity into a second sample image feature similarity probability within a preset value range; and determining a fourth loss value between the first sample image and the third sample image based on the second sample image feature similarity probability.

[0203] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained, and obtaining the first predicted behavior of the sample object; based on the first loss function, determining the second loss value between the first predicted behavior and the actual first behavior of the sample object; and, inputting the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained, and obtaining the second predicted behavior of the sample object; based on the second loss function, determining the third loss value between the second predicted behavior and the actual first behavior.

[0204] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the third sample image feature into the third classification network included in the second sub-twin network in the twin network to be trained, and obtaining the third predicted behavior of the sample object; based on the third loss function, determining the fifth loss value between the third predicted behavior and the actual second behavior of the sample object; and, inputting the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained, and obtaining the fourth predicted behavior of the sample object; based on the fourth loss function, determining the sixth loss value between the fourth predicted behavior and the actual second behavior of the sample object.

[0205] In one embodiment, when a computer program is executed by a processor to determine a second image and a third image based on a first image, the following steps are specifically implemented: the first image is input into a face detection model, and when it is detected that the target object is contained in the first image, the face information corresponding to the target object is output; based on the face information, a face image corresponding to the target object is captured from the first image; the face image is input into a face key point detection model to obtain the face key point coordinates of the target object; the first area and the second area are determined based on the face key point coordinates, and the second image corresponding to the first area and the third image corresponding to the second area are obtained from the face image.

[0206] In an exemplary embodiment, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0207] Acquire a first image of a target object in the cockpit, and determine a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object;

[0208] Inputting the first image and the second image into a first sub-twin network in the twin network, obtaining a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into a second sub-twin network in the twin network, obtaining a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image;

[0209] Determine a recognition result of a first behavior to be recognized of the target object based on the first similarity probability, and determine a recognition result of a second behavior to be recognized of the target object based on the second similarity probability;

[0210] Among them, the twin network is trained using the first sample pair and the second sample pair; the first sample pair is composed of the first sample image and the second sample image of the sample object, and the second sample image is the local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and the third sample image, and the third sample image is the local image corresponding to the second area in the first sample image.

[0211] In one embodiment, the first sub-twin network includes a first feature extraction network and a second feature extraction network, and the second sub-twin network includes a third feature extraction network and a fourth feature extraction network; the computer program is executed by the processor to input the first image and the second image into the first sub-twin network in the twin network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and input the first image and the third image into the second sub-twin network in the twin network to obtain a second similarity probability between the third image feature corresponding to the first image and the fourth image feature corresponding to the third image. The following steps are specifically implemented: the first image and the second image are respectively input into the first feature extraction network and the second feature extraction network to obtain A first image feature corresponding to the first image and a second image feature corresponding to the second image, and the first image and the third image are respectively input into a third feature extraction network and a fourth feature extraction network to obtain a third image feature corresponding to the first image and a fourth image feature corresponding to the third image; a first similarity between the first image feature and the second image feature is determined, and a second similarity between the third image feature and the fourth image feature is determined; the first similarity is converted into a first target value within a preset value range, and the first target value is used as a first similarity probability between the first image feature and the second image feature, and the second similarity is converted into a second target value within a preset value range, and the second target value is used as a second similarity probability between the third image feature and the fourth image feature.

[0212] In one embodiment, when a computer program is executed by a processor to determine a recognition result of a first behavior to be identified for a target object based on a first similarity probability, and to determine a recognition result of a second behavior to be identified for the target object based on a second similarity probability, the following steps are specifically implemented: when it is determined that the first similarity probability is greater than a first preset threshold, the recognition result of the first behavior to be identified for the target object is determined to be the occurrence of the first behavior to be identified; and when it is determined that the second similarity probability is greater than a second preset threshold, the recognition result of the second behavior to be identified for the target object is determined to be the occurrence of the second behavior to be identified; wherein the first preset threshold and the second preset threshold are the same or different.

[0213] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: outputting prompt information, where the prompt information is used to indicate the recognition results of the first behavior to be recognized and the second behavior to be recognized for the target object.

[0214] In one embodiment, the computer program is executed by the processor to implement the following steps: for the first sample pair, the first sample image feature corresponding to the first sample image is obtained through the first feature extraction network included in the first sub-twin network in the twin network to be trained, and the second sample image feature corresponding to the second sample image is obtained through the second feature extraction network included in the first sub-twin network in the twin network to be trained; determine the first total loss value corresponding to the first sample pair; wherein the first total loss value is composed of the first loss value, the second loss value and the third loss value; the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is the sample object The first predicted behavior and the actual first behavior of the sample object are obtained by inputting the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained; the third loss value is the loss value between the second predicted behavior of the sample object and the actual first behavior, and the second predicted behavior is obtained by inputting the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained; the first predicted behavior and the second predicted behavior are both prediction results for the first behavior to be identified; for the second sample pair, the third feature extraction network included in the second sub-twin network in the twin network to be trained is used to extract the first prediction behavior of the sample object and the actual first behavior. The method comprises the following steps: obtaining a third sample image feature corresponding to the first sample image through a fourth feature extraction network included in the second sub-twin network in the twin network to be trained, and obtaining a fourth sample image feature corresponding to the third sample image through a fourth feature extraction network included in the second sub-twin network in the twin network to be trained; determining a second total loss value corresponding to the second sample pair; the second total loss value is composed of a fourth loss value, a fifth loss value and a sixth loss value; the fourth loss value is determined based on the third sample image feature and the fourth sample image feature; the fifth loss value is a loss value between a third predicted behavior of the sample object and an actual second behavior of the sample object, and the third predicted behavior is obtained by inputting the third sample image feature into the first sub-twin network included in the second sub-twin network in the twin network to be trained. The sixth loss value is the loss value between the fourth predicted behavior of the sample object and the actual second behavior. The fourth predicted behavior is obtained by inputting the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained; the third predicted behavior and the fourth predicted behavior are both prediction results for the second behavior to be identified; based on the first total loss value and the second total loss value, the total loss value corresponding to the twin network is determined; when it is determined that the total loss value does not meet the conditions for stopping training, the network parameters of the twin network are updated based on the total loss value, and the twin network is trained again until the total loss value meets the conditions for stopping training to obtain the twin network.

[0215] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: determining the first sample image feature similarity between the first sample image feature and the second sample image feature; converting the first sample image feature similarity into a first sample image feature similarity probability within a preset value range; determining a first loss value between the first sample image and the second sample image based on the first sample image feature similarity probability; and determining the second sample image feature similarity between the third sample image feature and the fourth sample image feature; converting the second sample image feature similarity into a second sample image feature similarity probability within a preset value range; and determining a fourth loss value between the first sample image and the third sample image based on the second sample image feature similarity probability.

[0216] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the first sample image feature into the first classification network included in the first sub-twin network in the twin network to be trained, and obtaining the first predicted behavior of the sample object; based on the first loss function, determining the second loss value between the first predicted behavior and the actual first behavior of the sample object; and, inputting the second sample image feature into the second classification network included in the first sub-twin network in the twin network to be trained, and obtaining the second predicted behavior of the sample object; based on the second loss function, determining the third loss value between the second predicted behavior and the actual first behavior.

[0217] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the third sample image feature into the third classification network included in the second sub-twin network in the twin network to be trained, and obtaining the third predicted behavior of the sample object; based on the third loss function, determining the fifth loss value between the third predicted behavior and the actual second behavior of the sample object; and, inputting the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained, and obtaining the fourth predicted behavior of the sample object; based on the fourth loss function, determining the sixth loss value between the fourth predicted behavior and the actual second behavior of the sample object.

[0218] In one embodiment, when a computer program is executed by a processor to determine a second image and a third image based on a first image, the following steps are specifically implemented: the first image is input into a face detection model, and when it is detected that the target object is contained in the first image, the face information corresponding to the target object is output; based on the face information, a face image corresponding to the target object is captured from the first image; the face image is input into a face key point detection model to obtain the face key point coordinates of the target object; the first area and the second area are determined based on the face key point coordinates, and the second image corresponding to the first area and the third image corresponding to the second area are obtained from the face image.

[0219] It should be noted that the data involved in this application (including but not limited to the first image, the second image, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations. In addition, the images of user objects involved in this application are also all images authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant images must comply with relevant regulations.

[0220] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0221] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0222] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying object behavior in a vehicle cabin, characterized in that: The method comprises: Acquire a first image of a target object in the cockpit, and determine a second image and a third image based on the first image; the second image is a partial image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a partial image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object; Inputting the first image and the second image into a first sub-twin network in the twin network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into a second sub-twin network in the twin network to obtain a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image; Determine, based on the first similarity probability, a recognition result of the first behavior to be recognized for the target object; and, based on the second similarity probability, determine a recognition result of the second behavior to be recognized for the target object; Wherein, the twin network is trained using a first sample pair and a second sample pair; the first sample pair is composed of a first sample image and a second sample image of the sample object, and the second sample image is a local image corresponding to the first area in the first sample image; the second sample pair is composed of the first sample image and a third sample image, and the third sample image is a local image corresponding to the second area in the first sample image.

2. The method according to claim 1, characterized in that The first sub-twin network includes a first feature extraction network and a second feature extraction network, and the second sub-twin network includes a third feature extraction network and a fourth feature extraction network; The step of inputting the first image and the second image into a first sub-twin network in the twin network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into a second sub-twin network in the twin network to obtain a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image, comprises: Inputting the first image and the second image into the first feature extraction network and the second feature extraction network respectively, to obtain a first image feature corresponding to the first image and a second image feature corresponding to the second image, and inputting the first image and the third image into the third feature extraction network and the fourth feature extraction network respectively, to obtain a third image feature corresponding to the first image and a fourth image feature corresponding to the third image; determining a first similarity between the first image feature and the second image feature, and determining a second similarity between the third image feature and the fourth image feature; The first similarity is converted into a first target value within a preset value range, and the first target value is used as a first similarity probability between the first image feature and the second image feature, and the second similarity is converted into a second target value within a preset value range, and the second target value is used as a second similarity probability between the third image feature and the fourth image feature.

3. The method according to claim 1, characterized in that The determining, based on the first similarity probability, a recognition result of the first behavior to be recognized for the target object, and the determining, based on the second similarity probability, a recognition result of the second behavior to be recognized for the target object, include: In the case where it is determined that the first similarity probability is greater than a first preset threshold, determining that the recognition result of the first behavior to be recognized for the target object is that the first behavior to be recognized occurs; and In a case where it is determined that the second similarity probability is greater than a second preset threshold, determining that the recognition result of the second behavior to be recognized for the target object is that the second behavior to be recognized occurs; The first preset threshold and the second preset threshold are the same or different.

4. The method according to claim 3, characterized in that The method further comprises: Output prompt information, where the prompt information is used to indicate recognition results of the first behavior to be recognized and the second behavior to be recognized for the target object.

5. The method according to any one of claims 1 to 4, characterized in that: The twin network is trained in the following way: For the first sample pair, a first sample image feature corresponding to the first sample image is obtained through a first feature extraction network included in the first sub-twin network in the to-be-trained twin network, and a second sample image feature corresponding to the second sample image is obtained through a second feature extraction network included in the first sub-twin network in the to-be-trained twin network; Determine a first total loss value corresponding to the first sample pair; wherein the first total loss value is composed of a first loss value, a second loss value and a third loss value; the first loss value is determined based on the first sample image feature and the second sample image feature; the second loss value is a loss value between a first predicted behavior of the sample object and an actual first behavior of the sample object, and the first predicted behavior is obtained by inputting the first sample image feature into a first classification network included in a first sub-twin network in the twin network to be trained; the third loss value is a loss value between a second predicted behavior of the sample object and the actual first behavior, and the second predicted behavior is obtained by inputting the second sample image feature into a second classification network included in the first sub-twin network in the twin network to be trained; the first predicted behavior and the second predicted behavior are both prediction results for the first behavior to be identified; For the second sample pair, a third sample image feature corresponding to the first sample image is obtained through a third feature extraction network included in the second sub-twin network in the to-be-trained twin network, and a fourth sample image feature corresponding to the third sample image is obtained through a fourth feature extraction network included in the second sub-twin network in the to-be-trained twin network; Determine a second total loss value corresponding to the second sample pair; the second total loss value is composed of a fourth loss value, a fifth loss value and a sixth loss value; the fourth loss value is determined based on the third sample image feature and the fourth sample image feature; the fifth loss value is a loss value between a third predicted behavior of the sample object and an actual second behavior of the sample object, and the third predicted behavior is obtained by inputting the third sample image feature into the third classification network included in the second sub-twin network in the twin network to be trained; the sixth loss value is a loss value between a fourth predicted behavior of the sample object and the actual second behavior, and the fourth predicted behavior is obtained by inputting the fourth sample image feature into the fourth classification network included in the second sub-twin network in the twin network to be trained; the third predicted behavior and the fourth predicted behavior are both prediction results for the second behavior to be identified; Based on the first total loss value and the second total loss value, determining a total loss value corresponding to the twin network; When it is determined that the total loss value does not meet the condition for stopping training, the network parameters of the twin network are updated based on the total loss value, and the twin network is trained again until the total loss value meets the condition for stopping training, thereby obtaining the twin network.

6. The method according to claim 5, characterized in that The first loss value and the fourth loss value are determined in the following manner: Determining a first sample image feature similarity between the first sample image feature and the second sample image feature; converting the first sample image feature similarity into a first sample image feature similarity probability within a preset value range; determining a first loss value between the first sample image and the second sample image based on the first sample image feature similarity probability; and, Determine the second sample image feature similarity between the third sample image feature and the fourth sample image feature; convert the second sample image feature similarity into a second sample image feature similarity probability within a preset value range; Based on the second sample image feature similarity probability, a fourth loss value between the first sample image and the third sample image is determined.

7. The method according to claim 5, characterized in that The second loss value and the third loss value are determined in the following manner: Inputting the first sample image feature into a first classification network included in a first sub-Twin network in the to-be-trained Twin network to obtain a first predicted behavior of the sample object; Based on a first loss function, determining a second loss value between the first predicted behavior and an actual first behavior of the sample object; as well as, The second sample image feature is input into the second classification network included in the first sub-twin network in the twin network to be trained to obtain a second predicted behavior of the sample object; based on a second loss function, a third loss value between the second predicted behavior and the actual first behavior is determined.

8. The method according to claim 5, characterized in that The fifth loss value and the sixth loss value are determined in the following manner: Inputting the third sample image feature into the third classification network included in the second sub-twin network in the to-be-trained twin network to obtain a third predicted behavior of the sample object; determining a fifth loss value between the third predicted behavior and the actual second behavior of the sample object based on a third loss function; and, The fourth sample image feature is input into the fourth classification network included in the second sub-twin network in the twin network to be trained to obtain the fourth predicted behavior of the sample object; based on the fourth loss function, the sixth loss value between the fourth predicted behavior and the actual second behavior of the sample object is determined.

9. A vehicle, characterized in that: It comprises a vehicle processor and an image acquisition device in the cabin; the image acquisition device is used to acquire an image including a target object in the cabin; the vehicle processor is used to process the image to obtain a first image of the target object, and execute the method as described in any one of claims 1 to 8 to determine the recognition result of the behavior to be recognized for the target object.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images