Twin network training method and device, computer equipment and readable storage medium

Abstract

The invention relates to a twin network training method and device, computer equipment and a computer readable storage medium, and the method comprises the steps: obtaining a sample pair set, inputting each type of sample pair in the sample pair set into a sub-twin network in a to-be-trained twin network, obtaining first and second sample image features outputted by a feature extraction network, the classification network obtains image features and outputs first and second prediction behaviors; determining a first loss value based on the difference similarity between the first and second sample image features; obtaining second and third loss values based on the first and second predicted behaviors and the actual behavior; and determining a sub-loss value of each sample pair based on the first to third loss values, the sub-loss values of the multiple types of sample pairs being a total loss value of the sample pair set, when at least one total loss value does not satisfy a training stopping condition, updating network parameters and performing re-training until the stopping condition is satisfied, and the twin network which can be applied to object behavior identification in the vehicle cabin after being trained is obtained.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a training method, apparatus, computer device, and computer-readable storage medium for a twin network. Background Art

[0002] With the increase in demand for vehicle use, traffic accidents occur from time to time. Many traffic accidents are related to the driver's bad driving behavior, and some traffic accidents are caused by the bad riding behavior of the passengers in the car. In the existing vehicle use environment, drivers often do not pay attention to their driving behavior during driving, and the passengers in the car do not pay attention to their riding behavior in the car. For example, there are common bad behaviors such as drivers and passengers smoking and making phone calls in the car. Among them, similar behaviors such as drivers making phone calls while driving can easily distract the driver's driving concentration, and the smoking behavior of passengers while driving can easily cause quarrels between passengers and drivers, which will also distract the driver's driving concentration.

[0003] Although the use of vehicles facilitates people's travel, the safety of drivers and passengers is of paramount importance. In order to improve the safety of drivers and passengers in the car, real-time monitoring and intelligent evaluation of the driver's driving behavior and the passengers' riding behavior will help to detect irregular or bad behaviors of the car members as early as possible, and then remind the car members to change their relevant behavioral habits, so as to effectively reduce the problem of the driver's attention being distracted during driving, thereby helping to reduce the occurrence of traffic accidents and protect the lives of drivers and passengers.

[0004] However, in the relevant existing technologies, the accuracy of intelligent detection of bad behaviors of vehicle occupants is not ideal. Therefore, there is an urgent need to provide a behavior recognition model that can be used to accurately identify the behaviors of vehicle occupants, so as to accurately identify the bad behaviors of vehicle occupants in a shorter time. Summary of the invention

[0005] Based on this, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium and computer program product for a twin network that can be trained to perform intelligent monitoring of the behavior of members inside a vehicle with high accuracy in order to address the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a training method for a twin network, the method comprising:

[0007] Acquire a sample pair set; the sample pair set includes at least a first type of sample pair and a second type of sample pair; the first type of sample pair includes a first sample image and a second sample image of a sample object, and the second type of sample pair includes a first sample image and a third sample image; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified;

[0008] Input the sample pairs of each type into the corresponding sub-twin network in the twin network to be trained, and obtain the first sample image feature and the second sample image feature corresponding to the two sample images in the sample pair based on the feature extraction network in the sub-twin network; and input the first sample image feature and the second sample image feature into the classification network in the sub-twin network to obtain the first prediction behavior and the second prediction behavior respectively; wherein different types of sample pairs correspond to different sub-twin networks;

[0009] Determine a first loss value based on a difference similarity between the first sample image feature and the second sample image feature; obtain a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtain a third loss value based on the second predicted behavior and the actual behavior;

[0010] Based on the first loss value, the second loss value and the third loss value, the sub-loss value corresponding to each of the sample pairs is determined, and based on the sub-loss values ​​corresponding to at least the first type and the second type of the sample pairs respectively, the total loss value of the sample pair set is determined, and when the total loss value does not meet the training stop condition, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognitions of objects in a vehicle cabin.

[0011] In one embodiment, determining the first loss value based on the difference similarity between the first sample image feature and the second sample image feature includes:

[0012] Obtaining a difference similarity between the first sample image feature and the second sample image feature;

[0013] Converting the difference similarity into a sample image feature similarity probability within a preset value interval;

[0014] The first loss value is determined based on the sample image feature similarity probability, the actual behavior type value of the sample pair and the preset loss weight.

[0015] In one of the embodiments, before determining the first loss value based on the difference similarity between the first sample image feature and the second sample image feature, the method further includes:

[0016] For each type of sample pair, determining the number of first elements and the number of second elements whose element values ​​are within a preset value range in the first sample image feature and the second sample image feature of the sample pair of the type; and based on the difference between the element values ​​in the first sample image feature and the second sample image feature, determining the number of third elements whose difference is within a preset difference range;

[0017] Based on the first number of elements, the second number of elements, the third number of elements, and the first sample image feature and the second sample image feature, the difference similarity between the first sample image feature and the second sample image feature in the type sample pair is determined.

[0018] In one embodiment, inputting the first sample image feature and the second sample image feature into the classification network in the sub-twin network to obtain a first predicted behavior and a second predicted behavior respectively includes:

[0019] Inputting the first sample image feature into the first classification network in the sub-twin network to obtain the first predicted behavior;

[0020] The second sample image feature is input into the second classification network in the sub-twin network to obtain the second predicted behavior.

[0021] In one embodiment, the sample pairs of each type are respectively input into the corresponding sub-twin network in the twin network to be trained, and based on the feature extraction network in the sub-twin network, the first sample image features and the second sample image features corresponding to the two sample images in the sample pair are obtained, including:

[0022] For each type of sample pair, a sample image in the sample pair of the type is input into the first feature extraction network in the sub-twin network to obtain a corresponding first sample image feature;

[0023] Another sample image in the sample pair of the type is input into the second feature extraction network in the sub-twin network to obtain corresponding second sample image features.

[0024] In one embodiment, each type of sample pair includes a positive sample pair and a negative sample pair;

[0025] Both sample images included in the positive sample pair are images containing behaviors to be identified;

[0026] Both sample images included in the negative sample pair are images in which the to-be-recognized behavior does not exist.

[0027] In one embodiment, obtaining a sample pair set includes:

[0028] Acquire a plurality of sample images including sample objects, and determine a positive sample image to be selected and a negative sample image to be selected from the plurality of sample images;

[0029] For the same behavior to be identified, obtaining a first similarity between the behavior in each of the to-be-selected positive sample images and the to-be-identified behavior, and screening a positive sample image from the plurality of to-be-selected positive sample images according to a comparison result of the first similarity and the first similarity threshold;

[0030] Obtaining a second similarity between the behavior in each of the negative sample images to be selected and the behavior to be identified for the same behavior, and screening a negative sample image from the plurality of negative sample images to be selected according to a comparison result of the second similarity and the second similarity threshold;

[0031] After completing the image screening, respectively obtain a second sample image corresponding to the first target area and a third sample image corresponding to the second target area from each of the screened positive sample images and each of the negative sample images; form a first type of positive sample pair based on the positive sample image and the corresponding second sample image, form a second type of positive sample pair based on the positive sample image and the corresponding third sample image, form a first type of negative sample pair based on the negative sample image and the corresponding second sample image, and form a second type of negative sample pair based on the negative sample image and the corresponding third sample image;

[0032] The sample pair set is obtained based on the positive sample pairs of the first type, the negative sample pairs of the first type, the positive sample pairs of the second type, and the negative sample pairs of the second type.

[0033] In a second aspect, the present application also provides a training device for a twin network, comprising:

[0034] A sample acquisition module, used to acquire a sample pair set; the sample pair set includes at least a first type of sample pair and a second type of sample pair; the first type of sample pair includes a first sample image and a second sample image of a sample object, and the second type of sample pair includes a first sample image and a third sample image; the second sample image is a local image of a first target area in the first sample image, and the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified;

[0035] A sample processing module, used to input the sample pairs of each type into the corresponding sub-twin network in the twin network to be trained, and obtain the first sample image feature and the second sample image feature corresponding to the two sample images in the sample pair based on the feature extraction network in the sub-twin network; and input the first sample image feature and the second sample image feature into the classification network in the sub-twin network to obtain the first predicted behavior and the second predicted behavior respectively; wherein different types of sample pairs correspond to different sub-twin networks; determine the first loss value based on the difference similarity between the first sample image feature and the second sample image feature; obtain the second loss value based on the first predicted behavior and the actual behavior corresponding to the sample pair; and obtain the third loss value based on the second predicted behavior and the actual behavior;

[0036] A parameter adjustment module is used to determine the sub-loss value corresponding to each of the sample pairs based on the first loss value, the second loss value and the third loss value, and determine the total loss value of the sample pair set based on the sub-loss values ​​corresponding to at least the first type and the second type of the sample pairs respectively, and when the total loss value does not meet the training stop condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognitions of objects in a vehicle cabin.

[0037] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any step in the first aspect when executing the computer program.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any step in the first aspect when executed by a processor.

[0039] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements any step in the first aspect when executed by a processor.

[0040] The present application provides a training method, apparatus, computer equipment, computer-readable storage medium and computer program product for a twin network. The training method for the twin network mentioned above obtains a sample pair set; the sample pair set includes at least a first type of sample pair and a second type of sample pair; the first type of sample pair includes a first sample image and a second sample image of a sample object, and the second type of sample pair includes a first sample image and a third sample image; the second sample image is a local image of a first target area in the first sample image, and the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified; each type of sample pair is respectively input into the corresponding sub-twin network in the twin network to be trained, and the first sample image features and the second sample image features corresponding to the two sample images in the sample pair are obtained based on the feature extraction network in the sub-twin network; and the first sample image is extracted from the first sample image. The first feature and the second sample image feature are input into the classification network in the sub-twin network to obtain the first predicted behavior and the second predicted behavior respectively; wherein, different types of sample pairs correspond to different sub-twin networks; based on the difference similarity between the first sample image feature and the second sample image feature, a first loss value is determined; based on the first predicted behavior and the actual behavior corresponding to the sample pair, a second loss value is obtained; and, based on the second predicted behavior and the actual behavior, a third loss value is obtained; based on the first loss value, the second loss value and the third loss value, the sub-loss value corresponding to each sample pair is determined, and based on the sub-loss values ​​corresponding to at least the first type and the second type of sample pairs, the total loss value of the sample pair set is determined, and when the total loss value does not meet the training stop condition, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the training stop condition is met, to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of objects in the vehicle cabin.

[0041] It can be seen that the present application forms a sample pair set through a first sample image, a local image of the first target area related to the first behavior to be identified in the first sample image (second sample image), and a local image of the second target area related to the second behavior to be identified in the first sample image (third sample image), as training samples for the twin network to be trained, so that the sample image simultaneously includes an image of the first target area and an image of the second target area of ​​the sample object, and also includes a global image of the sample object, thereby increasing the types of sample images; wherein other feature information associated with the target area (first target area and second target area) in the global image is beneficial to assist in identifying the specific behavior of the sample object in the target area, and is beneficial to improving the recognition accuracy of the subsequently trained twin network for the behavior to be identified of the object in the image; further, the present application classifies the sample images in a sample pair set into multiple types of sample pairs, for example, the first sample image and the second sample image constitute a first type of sample pair, and the first sample image and the third sample image constitute a second type of sample pair; based on this, the present application can also be classified into The twin network to be trained is divided into at least two sub-twin networks, with each type of sample pair as the input of a corresponding sub-twin network; and in the process of training the twin network, the difference similarity calculation method is adopted to calculate the difference similarity between the first sample image feature and the second sample image feature of the sample image in a sample pair, so as to determine the first loss value corresponding to the sample pair. The determination of the first loss value by the difference similarity is conducive to improving the accuracy of the obtained first loss value, and thus is conducive to improving the recognition accuracy of the trained twin network for the to-be-recognized behavior of the object in the image; at the same time, for each sample pair, the second loss value and the third loss value corresponding to the first sample image feature and the second sample image feature are obtained respectively; then, based on the first loss value and the second loss value and the third loss value, the sub-loss value corresponding to each sample pair is obtained, and then based on the sub-loss values ​​corresponding to at least the first type and the second type of sample pairs, the total loss value of the sample pair set is determined, so as to achieve good training of the twin network to be trained based on the total loss value. In summary, the twin network trained based on the method provided in this application can efficiently and accurately identify multiple types of specific behaviors of the occupants in the car based on pictures of the occupants in the car. For example, it can identify one or two types of bad behaviors of the occupants in the car. This helps to detect the irregular or bad behaviors of the occupants in the car as early as possible, and then remind the occupants in the car to change their relevant behavioral habits, so as to effectively reduce the problem of the driver's attention being distracted during driving, thereby helping to reduce the occurrence of traffic accidents and protect the lives of drivers and passengers. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A diagram of an application environment for training a twin network in one embodiment;

[0044] Figure 2 This is an application environment diagram of a trained twin network in one embodiment;

[0045] Figure 3 It is a flowchart of a training method of a twin network in one embodiment;

[0046] Figure 4 A schematic diagram of the structure of a twin network in one embodiment;

[0047] Figure 5 A schematic diagram of the structure of a twin network in another embodiment;

[0048] Figure 6 It is a structural block diagram of a training device of a twin network in one embodiment;

[0049] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] The training method of the twin network provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 902 communicates with the server 904 through the network. The terminal 902 can at least be used to obtain sample images for training the twin network to be trained, and perform certain preprocessing operations on the sample images to obtain the first sample image corresponding to the sample image; in addition, the classification labeling work for the sample image can also be realized based on the terminal 902. The data storage system can store the data that the server 904 needs to process. The data storage system can be integrated on the server 904, or it can be placed on the cloud or other network servers. The data storage system can be used to store at least a plurality of sample pair sets including a first sample image, a second sample image, and a third sample image, as well as a twin network to be trained and a twin network that has been trained. Among them, the terminal 902 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server 904 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0052] It should be noted that the training process of the twin network can be based on the server 904 to provide operations, or it can be based on the terminal 902 to provide related operations, or it can be based on the communication between the terminal 902 and the server 904 to implement related operations. This application does not make specific limitations on this.

[0053] Furthermore, the trained twin network provided in the embodiment of the present application can be applied to Figure 2 The application environment shown. Figure 2 This is only one application environment that can be selected for the trained twin network provided by this application, and this application is not limited to this; that is, the trained twin network can not only be used for Figure 2 The driver / passenger behavior recognition in the vehicle environment shown can also be used for example for worker behavior recognition in a factory environment, and can also be used for example for student behavior recognition in a teaching environment, etc. Specifically, the application scenario includes a vehicle 100, the vehicle 100 includes a vehicle controller 101, and the vehicle cabin includes an image acquisition device 102.

[0054] The vehicle controller 101 may first capture an image of a target object (eg, at least one of a driver and a passenger) in the cabin through the image capture device 102, and then process the captured image to obtain a first image of the target object, and store the first image. In the process of object behavior recognition of images based on the twin network trained in the present application, the vehicle controller 101 can first obtain a first image of the target object, and determine the second image and the third image based on the first image, that is, obtain the first local image (second image) and the second local image (third image) of the target object from the global image (first image) of the target object; wherein the first local image is an image corresponding to the area in the first image related to the first behavior to be recognized of the target object, and the second local image is an image corresponding to the area in the first image related to the second behavior to be recognized of the target object; further, the vehicle controller 101 inputs the first image, the second image and the third image belonging to the same sample pair set into the twin network trained in the present application, so as to obtain the image feature similarity probability between the two images in each sample pair based on the twin network trained in the present application; finally, based on the image feature similarity probabilities corresponding to multiple types of sample pairs, the recognition results of the first behavior to be recognized and the second behavior to be recognized of the target object are determined to determine the specific behavior of the target object.

[0055] In order to provide a behavior recognition model that can be used to accurately identify the behavior of members inside the vehicle, so as to accurately identify the bad behavior of the occupants in the vehicle in a shorter time, the present application provides a twin network training method to train a twin network that can recognize the behavior of members inside the vehicle with high accuracy.

[0056] In an exemplary embodiment, Figure 3 As shown, a training method for a twin network is provided, including but not limited to the following steps S201 to S204. Among them:

[0057] S201, obtaining a sample pair set; the sample pair set includes at least a first type of sample pair and a second type of sample pair; the first type of sample pair includes a first sample image and a second sample image of the sample object, and the second type of sample pair includes a first sample image and a third sample image; the second sample image is a local image of a first target area in the first sample image, and the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified.

[0058] The sample object may be a driver inside the vehicle or a passenger inside the vehicle. The first sample image may be a first sample image including at least one of the driver and the passenger; assuming that the first behavior to be identified is whether the driver smokes, the first target area is an area related to the behavior of smoking, such as the driver's mouth area. In this case, the second sample image is a local image corresponding to the mouth area in the first sample image of the driver. Assuming that the second behavior to be identified is whether the driver makes a phone call, the second target area is an area related to the behavior of making a phone call, such as the driver's eye area. In this case, the third sample image is a local image corresponding to the eye area in the first sample image of the driver. The local image corresponding to the target area is also a face ROI image. ROI (Region of Interest) refers to a specific area to be processed in an image, and the area to be processed is usually outlined by a box, a circle, an ellipse, an irregular polygon, etc.; that is, the second sample image and the third sample image may be, for example, images of two different areas framed from the first sample image by a box shape.

[0059] Among them, the first sample image obtained by the present application and the second sample image associated with the first sample image can form a first type of sample pair, and this first type of sample pair is, for example, a sample involving smoking behavior; the first sample image obtained by the present application and the third sample image associated with the first sample image can form a second type of sample pair, and this second type of sample pair is, for example, a sample involving phone call behavior. A sample pair set can be formed by the first type of sample pair and the second type of sample pair. It should be supplemented that the embodiment provided in the present application is that the sample pair set includes the first type of sample pair and the second type of sample pair, which is only an optional implementation provided by the present application, but the present application is not limited to this. A sample pair set can also include 3 or more types of sample pairs at the same time.

[0060] In an optional implementation, before step S201, an image including the sample object acquired by an image acquisition device in the vehicle cabin may be acquired, and the image may be preprocessed to obtain the image including the sample object as the first sample image. The image preprocessing may include denoising, cropping, etc., which is not specifically limited in this application.

[0061] The image acquisition device may be, for example, an infrared sensor or a visible light sensor. The infrared sensor may be, for example, an infrared camera for acquiring infrared images. The visible light sensor may be, for example, an ordinary camera for acquiring color images, which may also be referred to as RGB images, where R is a red channel, G is a green channel, and B is a blue channel. The image of the sample object may be a single-channel infrared image of the vehicle cabin including the sample object acquired by the infrared sensor, or an RGB color image of the vehicle cabin including the sample object acquired by the visible light sensor. The image in the infrared image sample pair may be an image formed based on obtaining the radiation of the sample object in the infrared band.

[0062] The present application provides an optional implementation manner in which a vehicle cabin includes both an infrared sensor and a visible light sensor, wherein the infrared sensor is used to monitor and photograph the driver's seat in the vehicle, and the visible light sensor is used to monitor and photograph the passenger seat in the vehicle; however, this is only an optional implementation manner provided by the present application, and the present application is not limited thereto.

[0063] A further optional implementation is provided, where the acquired image of the sample object is a single-channel infrared image, it can be further converted into a three-channel infrared image to serve as the first sample image of the sample object.

[0064] Each sample pair set includes three images, specifically a first sample image of a sample object, a second sample image cropped based on an area in the first sample image related to a first behavior to be identified of the sample object, and a third sample image cropped based on an area in the first sample image related to a second behavior to be identified of the sample object. Furthermore, the second sample image and its corresponding first sample image constitute a sample pair of one type, and the third sample image and its corresponding first sample image constitute a sample pair of another type.

[0065] S202, input each type of sample pair into the corresponding sub-twin network in the twin network to be trained, and obtain the first sample image feature and the second sample image feature corresponding to the two sample images in the sample pair based on the feature extraction network in the sub-twin network; and input the first sample image feature and the second sample image feature into the classification network in the sub-twin network to obtain the first prediction behavior and the second prediction behavior respectively; wherein different types of sample pairs correspond to different sub-twin networks.

[0066] Among them, the twin network (twin neural network) to be trained provided in the present application may specifically include a feature extraction network and a classification network, that is, the twin network is at least composed of a feature extraction network and a classification network, wherein the input data of the feature extraction network is a sample pair set, and the input data of the classification network is the data output by the feature extraction network after processing the sample pair set.

[0067] Since in step S202 of the present application, the second sample image and its corresponding first sample image constitute, for example, a sample pair of the first type, and the third sample image and its corresponding first sample image constitute, for example, a sample pair of the second type. Since a sample pair is simultaneously input into the twin network to be trained as a group of data, the present application provides an optional implementation method in which the twin network to be trained is divided into two sub-twin networks on the basis of the sample pair set including the first type of sample pairs and the second type of sample pairs, and each sub-twin network is provided with a corresponding feature extraction network and classification network. For example, it includes a first sub-twin network for receiving a sample pair of the first type, and a second sub-twin network for receiving a sample pair of the second type.

[0068] In an optional embodiment, two sample images (for example, the second sample image and the first sample image, or the third sample image and the first sample image) included in a sample pair of a type are respectively input into a sub-twin network in the twin network to be trained provided in the present application, and the feature extraction network in the sub-twin network receives the sample pair to obtain the first sample image feature and the second sample image feature outputted by the feature extraction network after processing the two sample images respectively. Then, the first sample image feature and the second sample image feature are respectively inputted into the classification network included in the sub-twin network to obtain the first prediction behavior outputted by the classification network after processing the first sample image feature, and to obtain the second prediction behavior outputted by the classification network after processing the second sample image feature.

[0069] In an exemplary embodiment, the first sample image feature may be an image feature vector, and the second sample image feature may also be an image feature vector; for example, features in the first sample image (e.g., the full image) are extracted through a feature extraction network to obtain a first feature vector (first sample image feature), and features in the second sample image (e.g., the ROI image) are extracted through a feature extraction network to obtain a second feature vector (second sample image feature).

[0070] In an exemplary embodiment, the classification network is used to receive a first sample image feature corresponding to a sample image in a sample pair to output a prediction result (first predicted behavior) for a specific behavior of a sample object in the sample image; the classification network is also used to receive a second sample image feature corresponding to another sample image in the sample pair to output a prediction result (second predicted behavior) for a specific behavior of a sample object in the sample image. For example, if the first sample image includes smoking behavior, then the corresponding first predicted behavior may be smoking behavior; if the sample image includes phone call behavior, then the corresponding first predicted behavior may be phone call behavior; the second predicted behavior is similar and will not be described here.

[0071] It should also be added that the classification network is based on predictions of pre-set behavior types that need to be identified. For example, when the pre-set behavior types that need to be identified include smoking behavior, the classification network will output an output result that the first predicted behavior is smoking behavior.

[0072] S203, determining a first loss value based on the difference similarity between the first sample image feature and the second sample image feature; obtaining a second loss value based on the first predicted behavior and the actual behavior corresponding to the sample pair; and obtaining a third loss value based on the second predicted behavior and the actual behavior.

[0073] The first predicted behavior obtained by executing step S202 of the present application is a prediction result of the to-be-identified behavior of the sample object in a sample image in a sample pair based on the feature extraction network and the classification network in the sub-twin network in the twin network to be trained provided by the present application; the user can also determine the actual behavior of the sample object included in the sample image through expert identification (artificial identification), and then, based on the difference between the first predicted behavior for the same sample image and the actual behavior corresponding to the sample pair, for example, by processing the numerical value corresponding to the behavior type of the first predicted behavior and the numerical value corresponding to the behavior type of the actual behavior, a second loss value for the first predicted behavior and the actual behavior can be obtained. The second predicted behavior obtained by executing step S202 of the present application is a prediction result of the behavior to be identified of the sample object in another sample image in a sample pair based on the feature extraction network and the classification network in the sub-twin network in the twin network to be trained provided by the present application; the user can also determine the actual behavior of the sample object included in the sample image, and then, based on the difference between the second predicted behavior for the same sample image and the actual behavior corresponding to the sample pair, for example, by processing the numerical value corresponding to the behavior type of the second predicted behavior and the numerical value corresponding to the behavior type of the actual behavior, the third loss value for the second predicted behavior and the actual behavior can be obtained. While obtaining the second loss value and the third loss value, the first loss value corresponding to the first sample image feature and the second sample image feature can also be obtained based on the difference similarity between the first sample image feature and the second sample image feature. Among them, the calculation of the loss value can be implemented based on the loss function.

[0074] S204, based on the first loss value, the second loss value and the third loss value, determine the sub-loss value corresponding to each sample pair, based on the sub-loss values ​​corresponding to at least the first type and the second type of sample pairs respectively, determine the total loss value of the sample pair set, and when the total loss value does not meet the training stop condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the training stop condition is met, to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of objects in the vehicle cabin.

[0075] Among them, after obtaining the first loss value, the second loss value and the third loss value of a type of sample pair based on the execution of step S203, the sub-loss value of the sample pair can be further determined based on these three loss values, and then, based on the same method, the sub-loss values ​​corresponding to the multiple types of sample pairs included in the sample pair set can be obtained; for example, the sub-loss value of the first type of sample pair and the sub-loss value of the second type of sample pair are obtained, and then, the total loss value corresponding to the sample pair set can be determined based on these two sub-loss values, and then it is judged whether the total loss value of the sample pair set meets the stop training condition of the twin network to be trained. When there are samples whose total loss values ​​do not meet the stop training condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample pair set used in a training process, and the twin network is trained again until the total loss values ​​of multiple sample pair sets meet the stop training condition. The training process of the twin network can be ended, and the trained twin network can be obtained at this time. Since the data input into the twin network to be trained in the present application is a cabin image including sample objects, and at least part of the sample objects included in the cabin image have behaviors to be identified, the twin network trained in the present application can be used to recognize the (pre-set) behaviors of objects in the vehicle cabin; furthermore, when the behavior of objects in the vehicle cabin is subsequently identified based on the trained twin network, the accuracy of identifying the behavior of objects in the vehicle cabin can be improved.

[0076] It should be noted that the sample pair set provided above is an embodiment of a cabin image including a sample object, which is only an optional implementation provided by the present application, and the present application is not limited thereto; the specific image type including the sample object used to train the twin network can be selected based on the different requirements of the application scenarios of the trained twin network. For example, the sample pair set that can be selected is a classroom image including a sample object (a student). In this case, the corresponding preset behaviors can be sleeping behavior, face covering behavior, eating behavior, etc.

[0077] The training method of the above twin network is by obtaining a sample pair set; the sample pair set includes at least a first type of sample pair and a second type of sample pair; the first type of sample pair includes a first sample image and a second sample image of the sample object, and the second type of sample pair includes a first sample image and a third sample image; the second sample image is a local image of a first target area in the first sample image, and the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to the first behavior to be identified, and the second target area is an area related to the second behavior to be identified; each type of sample pair is input into the corresponding sub-twin network in the twin network to be trained, and the first sample image features and the second sample image features corresponding to the two sample images in the sample pair are obtained based on the feature extraction network in the sub-twin network; and the first sample image features and the second sample image features are input into the classification network in the sub-twin network. network, and obtain the first predicted behavior and the second predicted behavior respectively; wherein, different types of sample pairs correspond to different sub-twin networks; based on the difference similarity between the first sample image feature and the second sample image feature, determine the first loss value; based on the first predicted behavior and the actual behavior corresponding to the sample pair, obtain the second loss value; and, based on the second predicted behavior and the actual behavior, obtain the third loss value; based on the first loss value, the second loss value and the third loss value, determine the sub-loss value corresponding to each sample pair, based on the sub-loss values ​​corresponding to at least the first type and the second type of sample pairs, respectively, determine the total loss value of the sample pair set, and when the total loss value does not meet the training stop condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the training stop condition is met, to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of objects in the vehicle cabin.

[0078] It can be seen that the present application forms a sample pair set through a first sample image, a local image of the first target area related to the first behavior to be identified in the first sample image (second sample image), and a local image of the second target area related to the second behavior to be identified in the first sample image (third sample image), as training samples for the twin network to be trained, so that the sample image simultaneously includes an image of the first target area and an image of the second target area of ​​the sample object, and also includes a global image of the sample object, thereby increasing the types of sample images; wherein other feature information associated with the target area (first target area and second target area) in the global image is beneficial to assist in identifying the specific behavior of the sample object in the target area, and is beneficial to improving the recognition accuracy of the subsequently trained twin network for the behavior to be identified of the object in the image; further, the present application classifies the sample images in a sample pair set into multiple types of sample pairs, for example, the first sample image and the second sample image constitute a first type of sample pair, and the first sample image and the third sample image constitute a second type of sample pair; based on this, the present application can also be classified into The twin network to be trained is divided into at least two sub-twin networks, with each type of sample pair as the input of a corresponding sub-twin network; and in the process of training the twin network, the difference similarity calculation formula is used to calculate the difference similarity between the first sample image feature and the second sample image feature of the sample image in a sample pair, so as to determine the first loss value corresponding to the sample pair. The determination of the first loss value by difference similarity is conducive to improving the accuracy of the obtained first loss value, and thus is conducive to improving the recognition accuracy of the trained twin network for the to-be-recognized behavior of the object in the image; at the same time, for each sample pair, the second loss value and the third loss value corresponding to the first sample image feature and the second sample image feature are obtained respectively; then, based on the first loss value and the second loss value and the third loss value, the sub-loss value corresponding to each sample pair is obtained, and then based on the sub-loss values ​​corresponding to at least the first type and the second type of sample pairs, the total loss value of the sample pair set is determined, so as to achieve good training of the twin network to be trained based on the total loss value. In summary, the twin network trained based on the method provided in this application can efficiently and accurately identify multiple types of specific behaviors of the occupants in the car based on pictures of the occupants in the car. For example, it can identify one or two types of bad behaviors of the occupants in the car. This helps to detect the irregular or bad behaviors of the occupants in the car as early as possible, and then remind the occupants in the car to change their relevant behavioral habits, so as to effectively reduce the problem of the driver's attention being distracted during driving, thereby helping to reduce the occurrence of traffic accidents and protect the lives of drivers and passengers.

[0079] In an exemplary embodiment, the step of determining the first loss value based on the difference similarity between the first sample image feature and the second sample image feature in step S203 may be specifically performed as follows, including steps S231 to S233 (not shown), wherein:

[0080] S231, obtaining a difference similarity between a first sample image feature and a second sample image feature;

[0081] S232, converting the difference similarity into a sample image feature similarity probability within a preset value range;

[0082] S233, determining a first loss value based on the sample image feature similarity probability, the actual behavior type value of the sample pair, and the preset loss weight.

[0083] Specifically, firstly, based on the calculation formula of difference similarity, the difference similarity between the first sample image feature and the second sample image feature is calculated, that is, the difference similarity corresponding to the two images in the sample pair is obtained.

[0084] Afterwards, the difference similarity is converted into a sample image feature similarity probability within a preset value interval, for example, the difference similarity is converted into a sample image feature relative probability within the interval [0,1]. The sample image feature relative probability is used to determine whether the behaviors of the two sample images in the sample pair are the same; for example, when the sample image feature relative probability is greater than the preset smoking probability, then it can be determined that the sample objects in the two images of the sample pair both have smoking behavior.

[0085] Finally, based on the sample image feature similarity probability obtained by executing step S232, the actual behavior type value of the sample object in the determined sample pair image, and the preset loss weight, a first loss value associated with a sample pair can be determined. The actual behavior type value is, for example, when the actual behavior is the same as the preset behavior to be detected, the actual behavior type value can be set to 1, and when the actual behavior is different from the preset behavior to be detected, the actual behavior type value can be set to 0.

[0086] In an exemplary embodiment, before determining the first loss value based on the difference similarity between the first sample image feature and the second sample image feature in step S203, the method further includes steps S321 and S322, wherein:

[0087] S321, for each type of sample pair, determining the number of first elements and the number of second elements in the first sample image feature and the second sample image feature of the sample pair of the type, whose element values ​​are within a preset value range; and determining the number of third elements whose difference values ​​are within a preset difference range based on the difference between the element values ​​in the first sample image feature and the second sample image feature;

[0088] S322, determining the difference similarity between the first sample image feature and the second sample image feature in the sample pair of the same type based on the first element number, the second element number, the third element number, and the first sample image feature and the second sample image feature.

[0089] Among them, the first number of elements is the element value of the element in the first sample image feature, and the number of elements within the preset value range; the second number of elements is the element value of the element in the second sample image feature, and the number of elements within the preset value range; regarding the determination of the third number of elements, the difference between the element values ​​of the elements in the first sample image feature and the second sample image feature is first determined, and the number of elements corresponding to the difference within the preset difference range is the third number of elements.

[0090] The difference similarity between the first sample image feature and the second sample image feature needs to be determined based on at least the first element number, the second element number, the third element number, and the first sample image feature and the second sample image feature.

[0091] In an exemplary embodiment, the above step S202, in which the first sample image feature and the second sample image feature are input into the classification network in the sub-twin network to obtain the first prediction behavior and the second prediction behavior respectively, can be selectively performed as follows: the first sample image feature is input into the first classification network in the sub-twin network to obtain the first prediction behavior; the second sample image feature is input into the second classification network in the sub-twin network to obtain the second prediction behavior.

[0092] In an exemplary embodiment, the above step S202 is performed by inputting each type of sample pair into the corresponding sub-twin network in the twin network to be trained, and obtaining the first sample image features and the second sample image features corresponding to the two sample images in the sample pair based on the feature extraction network in the sub-twin network. It can be optionally executed as follows: for each type of sample pair, one sample image in the sample pair of the type is input into the first feature extraction network in the sub-twin network to obtain the corresponding first sample image feature; the other sample image in the sample pair of the type is input into the second feature extraction network in the sub-twin network to obtain the corresponding second sample image feature.

[0093] Specifically, an optional implementation method is provided, in which the feature extraction network included in the sub-twin network in the twin network to be trained provided in the present application specifically includes a first feature extraction network and a second feature extraction network, and the classification network included in the twin network to be trained specifically includes a first classification network and a second classification network; wherein the first classification network is used to receive the result output by the corresponding first feature extraction network, and the second classification network is used to receive the result output by the corresponding second feature extraction network.

[0094] That is, it can be considered that the twin network to be trained provided by the present application includes two sub-twin networks, the first sub-twin network includes a first feature extraction network and a first classification network, and a second feature extraction network and a second classification network. The first sub-twin network is used to receive and process two images in a sample pair.

[0095] For example, the first sample image in the sample pair will be input into the first feature extraction network in the sub-twin network, the first feature extraction network outputs the first sample image feature, and then the first sample image feature will be input into the first classification network, and the first predicted behavior of the sample object in the first sample image will be obtained based on the corresponding first classification network output; the second sample image in the sample pair will be input into the second feature extraction network in the sub-twin network, the second feature extraction network outputs the second sample image feature, and then the second sample image feature will be input into the second classification network, and the second predicted behavior of the sample object in the second sample image will be obtained based on the corresponding second classification network output. For example, in the case where there is smoking behavior in the sample image in the sample pair, the sample pair is input into a sub-twin network of the twin network to be trained provided in the present application, and the first prediction result and the second prediction result output by the classification network may both be "smoking behavior".

[0096] It should also be added that, regarding the calculation of the second loss value and the third loss value, a suitable loss function for image classification can be selected for relevant calculations. For example, the first loss function used to calculate the second loss value and the second loss function used to calculate the third loss value may include but are not limited to the mean square error loss function, the mean absolute error loss function or the cross entropy loss function. Among them, the first loss function and the second loss function may be the same or different. For example, the loss function used to calculate the second loss value and the third loss value may both select the cross entropy loss function (Cross Entropy Loss), so as to calculate the loss value between the predicted behavior type value and the actual behavior type value of a corresponding set of predicted behaviors based on the cross entropy loss function.

[0097] In an exemplary embodiment, the first feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network; the second feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network.

[0098] Specifically, each sub-twin network of the twin network to be trained provided in the present application includes a first feature extraction network and a second feature extraction network. Both networks can choose to use convolutional neural networks. The specific type of convolutional neural network can be, for example, any one of the lightweight neural network MobileNet and the target detection network YOLO.

[0099] It should also be added that for the first feature extraction network and the second feature extraction network included in each sub-twin network, the same convolutional neural network can be selected, or different convolutional neural networks can be selected. For example, when the first feature extraction network selects MobileNet-V3, the second feature extraction network also selects MobileNet-V3; when the first feature extraction network selects MobileNet-V3, the second feature extraction network selects YOLO-V8; when the first feature extraction network selects YOLO-V8, the second feature extraction network selects MobileNet-V3; when the first feature extraction network selects YOLO-V8, the second feature extraction network also selects YOLO-V8.

[0100] The MobileNet-V3 and YOLO-V8 provided here are optional versions corresponding to the convolutional neural network MobileNet and the convolutional neural network YOLO, respectively, but this application does not limit the specific version selection and can be adjusted and used according to needs.

[0101] Specifically, in the twin network to be trained provided in the present application, when it includes two sub-twin networks, the specific types of the two first feature extraction networks and the two second feature extraction networks included in the two sub-twin networks can be selected from MobileNet-V3 and YOLO-V8, respectively.

[0102] In an exemplary embodiment, each type of sample pair includes a positive sample pair and a negative sample pair; both sample images included in the positive sample pair are images with behaviors to be identified; and both sample images included in the negative sample pair are images without behaviors to be identified.

[0103] Specifically, in the sample pair set collected by the present application for implementing the training of the twin network to be trained, each type of sample pair includes both positive sample pairs and negative sample pairs, wherein the behavior of the sample object in the image of the positive sample pair has a preset behavior to be identified. For example, when the preset behavior to be identified is smoking, the sample object (such as the driver) in the first sample image and the second sample image of the same sample pair both has smoking behavior. In the behavior of the sample object in the image of the negative sample pair, there is no preset behavior to be identified. For example, when the preset behavior to be identified is smoking, the sample object (such as the driver) in the first sample image and the second sample image of the same sample pair both has no smoking behavior.

[0104] By selecting training sample pairs for training the twin network, including both positive sample pairs and negative sample pairs, it is helpful to improve the recognition accuracy of the finally trained twin network for the behavior to be recognized.

[0105] It should also be added that the application of the trained twin network to the object behavior recognition in the vehicle cabin is only an optional application scenario provided by the application; however, in other similar scenarios, the twin network trained based on the training method provided by the application can also be used to recognize the behavior of objects in the scene; for example, the twin network trained based on the training method provided by the application can be used to recognize the behavior of workers in a factory environment; the twin network trained based on the training method provided by the application can be used to recognize the behavior of students in a classroom environment. Based on the different application scenarios of the twin network, the collection scenario of its training samples can also be adapted to the corresponding scenario, which is conducive to improving the behavior recognition effect of the trained twin network.

[0106] In an exemplary embodiment, the acquisition of the sample pair set in step S201 may be specifically implemented by executing steps S211 to S215 (not shown), wherein:

[0107] S211, acquiring a plurality of sample images including sample objects, and determining a positive sample image to be selected and a negative sample image to be selected from the plurality of sample images;

[0108] S212, for the same behavior to be identified, obtaining a first similarity between the behavior in each of the to-be-selected positive sample images and the behavior to be identified, and screening a positive sample image from a plurality of to-be-selected positive sample images according to a comparison result of the first similarity and a first similarity threshold;

[0109] S213, obtaining a second similarity between the behavior in each candidate negative sample image and the behavior to be identified for the same behavior, and screening a negative sample image from a plurality of candidate negative sample images according to a comparison result between the second similarity and a second similarity threshold;

[0110] S214, after completing the image screening, respectively obtain a second sample image corresponding to the first target area and a third sample image corresponding to the second target area from each of the screened positive sample images and each of the negative sample images; form a first type of positive sample pair based on the positive sample image and the corresponding second sample image, form a second type of positive sample pair based on the positive sample image and the corresponding third sample image, form a first type of negative sample pair based on the negative sample image and the corresponding second sample image, and form a second type of negative sample pair based on the negative sample image and the corresponding third sample image;

[0111] S215 , obtaining a sample pair set based on the positive sample pairs of the first type, the negative sample pairs of the first type, the positive sample pairs of the second type, and the negative sample pairs of the second type.

[0112] Specifically, a plurality of sample images including sample objects are first obtained, and then these sample images are divided into positive sample images to be selected and negative sample images to be selected; then, a first similarity between the behavior of the sample object in each positive sample image to be selected and the behavior to be identified is calculated, that is, the similarity probability between the behavior of the sample object in the positive sample image to be selected and the behavior to be identified, and then, according to the comparison result of the first similarity and the first similarity threshold, the positive sample images to be selected whose first similarity is greater than or equal to the first similarity threshold are screened out as the screened positive sample images; at the same time, a second similarity between the behavior of the sample object in each negative sample image to be selected and the behavior to be identified is calculated, that is, the similarity probability between the behavior of the sample object in the negative sample image to be selected and the behavior to be identified, and then, according to the comparison result of the second similarity and the second similarity threshold, the negative sample images to be selected whose second similarity is less than the second similarity threshold are screened out as the screened negative sample images.

[0113] It should be added that the present application does not limit the type of specific calculation formula applied to the behavior of the sample object in the sample image and the similarity probability between the behavior to be identified, and a suitable similarity calculation formula can be selected according to needs.

[0114] After completing the image screening, the corresponding second sample image and the third sample image are respectively intercepted from each screened positive sample image and each negative sample image, and then a first type of positive sample pair is formed based on the positive sample image and the corresponding second sample image, a second type of positive sample pair is formed based on the positive sample image and the corresponding third sample image, a first type of negative sample pair is formed based on the negative sample image and the corresponding second sample image, and a second type of negative sample pair is formed based on the negative sample image and the corresponding third sample image; so as to form a sample pair set by including multiple types of positive and negative sample pairs.

[0115] It should also be added that when the sample pair set selected is a vehicle cabin image that includes the sample object, the images used to train the twin network to be trained must include vehicle cabin images, but it is not limited to all images being vehicle cabin images, and some other images may also be included.

[0116] In addition, it should be added that for the twin network applied to the same application scenario, when training the twin network, the sample images of the sample pair set used can be collected in the corresponding same application scenario, or can be collected in multiple application scenarios. This application does not make specific limitations on this.

[0117] With respect to obtaining a sample pair set in step S201, the present application also provides an optional implementation method as follows:

[0118] Acquire a first sample image of a sample object in a vehicle cabin; input the first sample image into a face detection model, and when it is detected that the sample object is contained in the first sample image, output face information corresponding to the sample object; based on the face information, intercept a face image corresponding to the sample object from the first sample image; input the face image into a face key point detection model to obtain the face key point coordinates of the sample object; determine a first target area and a second target area based on the face key point coordinates, and acquire a second sample image corresponding to the first target area and a third sample image corresponding to the second target area from the face image; form a sample pair set based on the first sample image, the second sample image and the third sample image of the sample object.

[0119] With respect to the training method of the twin network for object behavior recognition in a vehicle cabin provided in the present application, the present application provides an optional embodiment, and the specific training process is shown in the following steps S1 to S7, wherein:

[0120] S1, obtaining multiple sample pair sets; wherein each sample pair set includes a full-image smoking or phone call sample image (first sample image), a mouth ROI smoking sample image (second sample image), and an eye ROI phone call sample image (third sample image), wherein the full-image smoking or phone call sample image is Figure 5 In the figure, D1 represents the mouth ROI smoking sample image. Figure 5 In the figure, D2 represents the eye ROI smoking sample image. Figure 5 For the same sample pair, the mouth ROI smoking sample image D2 is obtained by intercepting the area related to the smoking behavior to be identified in the full-image smoking or phone call sample image D1, and the eye ROI phone call sample image D3 is obtained by intercepting the area related to the phone call behavior to be identified in the full-image smoking or phone call sample image D1.

[0121] S2, extracts the ROI features of the data set D2 through convolutional neural network 2, and obtains the feature vector G w (D2), which is the second sample image feature mentioned above; extract the ROI feature of the data set D3 through the convolutional neural network 4 to obtain the feature vector G w (D3), which is the second sample image feature mentioned above; extract the full image feature of the data set D1 through the convolutional neural network 1, and extract the full image feature of the data set D1 through the convolutional neural network 3 to obtain the feature vector G w (D1), which is the first sample image feature mentioned above; wherein, the four convolutional neural networks, convolutional neural network 1 to convolutional neural network 4, can all adopt lightweight neural network MobileNet-V3 or target detection network YOLO-V8 based on demand. Among them, it should be added that, when the types of convolutional neural network 1 and convolutional neural network 3 are different, for example, convolutional neural network 1 adopts MobileNet-V3 and convolutional neural network 3 adopts YOLO-V8, even if convolutional neural network 1 and convolutional neural network 3 receive the same data set D1, but after being processed by convolutional neural network 1 and convolutional neural network 3 respectively, the first sample image features obtained respectively may also be different. It should also be added that at least parameter sharing can be achieved between the networks corresponding to the same type of sample pairs.

[0122] The subsequent steps S3 to S7 are as follows: Figure 5 The input of the convolutional neural network 3 shown in the figure is D4 on the left and the output is G on the right. w (D4) is used as an example for explanation.

[0123] S3, calculate the feature vector G by measuring the difference similarity w (D1) and the eigenvector G w The similarity measure of (D2), that is, the similarity of the first sample image features, is shown in formula (1).

[0124]

[0125] Among them, G in formula (1) w (D1) is the first feature vector, which represents the first sample image feature; G w (D2) is the second feature vector, which represents the second sample image feature; e It means G w The number of elements in (D1) that are not equal to 0; z f It means G w The number of elements in (D2) that are not equal to 0; ef It means G w (D1) and Gw The number of elements in the difference of (D2) that are not equal to 0.

[0126] By measuring the difference similarity, the feature vector G is calculated. w (D3) and the eigenvector G w The similarity measure of (D4), that is, the similarity of the second sample image features, is calculated as shown in formula (2).

[0127]

[0128] Among them, G in formula (2) w (D4) is the first feature vector, which represents the first sample image feature; G w (D3) is the second feature vector, which represents the second sample image feature; e It means G w The number of elements in (D3) that are not equal to 0; f It means G w The number of elements in (D4) that are not equal to 0; ef It means G w (D3) and G w The number of elements in the difference of (D4) that are not equal to 0.

[0129] S4, the feature vector G w (D1), eigenvector G w (D2) The similarity passes through the fully connected layer and then the Sigmoid layer, and is converted into the numerical sample relative probability RPS in the interval [0,1], that is, the similarity probability of judging that the sample is smoking. When RPS>0.5, it means that the model judges that the sample is smoking, indicating that the judgment of the ROI image classification model (the second classification network) on the mouth ROI smoking positive sample image and the global image classification task (the first classification network) on the full image smoking positive sample judgment are more consistent. Among them, when the twin network (model) is trained, the model will tend to judge the smoking positive sample.

[0130] Eigenvector G w (D3), eigenvector G w (D4) The similarity passes through the fully connected layer and then the Sigmoid layer, and is converted into the numerical sample relative probability RPS in the interval [0,1], that is, the similarity probability of judging that the sample is a phone call. When RPS>0.5, it means that the model judges that the sample is a phone call, indicating that the judgment of the ROI image classification model (the second classification network) on the mouth ROI phone call positive sample image and the global image classification task (the first classification network) on the full image phone call positive sample judgment are more consistent. Among them, when the twin network (model) is trained, the model will tend to judge the positive sample as a phone call.

[0131] That is, if the probability of similarity of sample image features is higher, it means that the recognition result of the first classification network in the twin network for the target object's to-be-identified behavior in the first sample image (full image) is more consistent with the recognition result of the second classification network for the target object's to-be-identified behavior in the second sample image (partial image).

[0132] S5, construct a cross entropy loss function Lrps based on RPS, and the relevant construction formula can be shown in formula (3).

[0133]

[0134] Among them, L in formula (3) rps represents the first loss value, that is, the loss value between the first sample image and the second sample image; RPS represents the probability of similarity of sample image features, that is, the relative probability of the feature vector of the sample image mentioned above being converted into a numerical sample within a preset interval; λ represents the loss weight (preset loss weight); l represents the label of the sample group, that is, the actual behavior type of the sample group. If the sample group is a positive sample group, l is 1, and if the sample group is a negative sample group, l is 0; It represents the regularization function, which can enhance the generalization ability of the model.

[0135] Among them, when training the model (twin network), the similarity metric and classification task can assist each other in learning, so that the model can not only distinguish whether two samples are similar, but also identify the category of a single sample; in addition, since the ROI image classification task and the full-image image classification task have different noise patterns, learning the two image classification tasks at the same time can average the noise, share their respective overfitting risks, and obtain a more generalized representation.

[0136] S6, the sub-twin network of the model includes two classification tasks (one is the ROI image classification task, the other is the full-image image classification task) and a similarity measurement task, so the sub-loss function consists of three parts, that is, the sub-loss value consists of the first loss value, the second loss value and the third loss value. The calculation method of the sub-loss value is shown in formula (4)(5).

[0137] L1=λ×(L cls (G w (D1)+L cls (G w (D2)+(1-2λ)×L rps (4)

[0138] L2=λ×(L cls (G w (D3)+L cls (G w (D4)+(1-2λ)×Lrps (5)

[0139] Among them, L1 and L2 in formula (4) and (5) represent the sub-loss values ​​of the corresponding sample pairs; λ represents the loss weight; G w (D1), G w (D4) are all first eigenvectors, representing the first sample image features; G w (D2), G w (D3) is the second feature vector, which represents the second sample image feature; L rps It represents the first loss value, that is, the loss value between two images in the same type of sample pair.

[0140] S7, based on the sum of the sub-loss value L1 corresponding to the first type of sample pair and the sub-loss value L2 corresponding to the second type of sample pair, the total loss value L of the sample pair set can be obtained. The relevant calculation formula is L=L1+L2.

[0141] In the training method of the twin network for object behavior recognition in the vehicle cabin shown in steps S1-S7, the first behavior to be recognized is set to smoking behavior, and the second behavior to be recognized is set to phone calling behavior, but this is only an optional embodiment provided by the present application; the behavior to be recognized can also be set to other behaviors as required. Among them, the first sample image and the second sample image in the same sample pair, or the first sample image and the third sample image in the same sample pair, can both be infrared images or RGB images, and the images in the same sample pair set can both be infrared images or RGB images; multiple sample pair sets can include both infrared images and RGB images.

[0142] In addition, the present application also provides an iterative process of a deep learning model of infrared images based on a twin neural network. When it is necessary to train the twin network to recognize smoking behavior and phone calling behavior, it may include the following contents: select 3,000 positive sample infrared images of the cockpit mouth ROI smoking and 3,000 negative sample infrared images of the cockpit mouth ROI non-smoking as the training set D_1; select 3,000 positive sample infrared images of the cockpit eye ROI calling and 3,000 negative sample infrared images of the cockpit eye ROI non-calling as the training set D_2; select 3,000 positive sample infrared images of the entire cockpit smoking, 3,000 positive sample infrared images of the entire cockpit calling and 3,000 negative sample infrared images of the entire cockpit non-smoking and non-calling as the training set D_3; use D_1, D_2 and D_3 as the initial training sets to train a multi-task twin neural network deep learning initial model based on the fusion of ROI image classification and infrared full image target detection.

[0143] Furthermore, a multi-task twin neural network deep learning initial model based on ROI image classification and infrared full-image target detection fusion can be used to screen a data set of 1 million images, specifically including the following:

[0144] When screening ROI smoking positive samples, a multi-task twin neural network deep learning initial model based on ROI image classification and infrared full image target detection fusion is used to output samples with a probability score of less than 0.1 for positive samples of mouth ROI smoking, and add them to the positive sample set in the dataset D_1;

[0145] When screening positive samples of ROI making phone calls, a multi-task twin neural network deep learning initial model based on the fusion of ROI image classification and infrared full-image target detection is used to output samples with a probability score of less than 0.1 for positive samples of eye ROI making phone calls, and add them to the positive sample set in the data set D_2;

[0146] When screening the full-image smoking positive samples, a multi-task twin neural network deep learning initial model based on the fusion of ROI image classification and infrared full-image target detection is used to output the full-image smoking positive samples with a probability score less than 0.1 and add them to the positive sample set in the data set D_3;

[0147] When screening full-image phone call positive samples, a multi-task twin neural network deep learning initial model based on ROI image classification and infrared full-image target detection fusion is used to output full-image phone call positive samples with a probability score less than 0.1 and add them to the positive sample set in data set D_3;

[0148] When screening the mouth ROI non-smoking negative samples, a multi-task twin neural network deep learning initial model based on the fusion of ROI image classification and infrared full image target detection is used to output the mouth ROI non-smoking negative samples with a probability score greater than 0.9 and add them to the negative sample set in the dataset D_1;

[0149] When screening eye ROI non-calling negative samples, a multi-task twin neural network deep learning initial model based on ROI image classification and infrared full-image target detection fusion is used to output eye ROI non-calling negative samples with a probability score greater than 0.9 and add them to the negative sample set in the dataset D_2;

[0150] When screening the negative samples of non-smoking and non-calling in the whole image, a multi-task twin neural network deep learning initial model based on ROI image classification and infrared full image target detection fusion is used to output the samples with probability scores greater than 0.9 of the negative samples of non-smoking and non-calling in the whole image and add them to the negative sample set in the data set D_3;

[0151] Before adding the above-screened ROI and full-image positive and negative samples to the initial training set, it is necessary to browse the samples to be added to check whether there are any negative samples mixed in the positive samples, and whether there are any positive samples mixed in the negative samples.

[0152] When the model outputs the positive sample threshold set to 0.1 and the negative sample threshold set to 0.9 to filter the sample set, and the number of positive and negative samples screened out is less than 1000, the positive sample threshold is set to 0.2 and the negative sample threshold is set to 0.8. This cycle is repeated, using thresholds of 0.1, 0.2, 0.3, and 0.4 to filter positive samples, and thresholds of 0.9, 0.8, 0.7, and 0.6 to filter negative samples. Before updating the training set, it is necessary to browse the positive and negative samples to be added to see if there are negative samples mixed with positive samples, and positive samples mixed with negative samples. By continuously adding positive and negative samples to the initial training set in the above way, a multi-task twin neural network deep learning model based on the fusion of ROI image classification and infrared full-image target detection is obtained.

[0153] In the prior art, for example, when using the cockpit perception system in a vehicle to detect whether the driver among the vehicle members has smoking behavior or making phone calls, the relevant detection process is as follows: face detection is performed on the image frame sequence of the main driver's seat. If a face is detected, face key point regression is performed in the face ROI area; based on the key points, the ROI area near the mouth is intercepted and input into the smoking image classification model to judge the smoking behavior of the main driver's seat, and the ROI area near the eyes is intercepted and input into the phone call image classification model to judge the phone call behavior of the main driver's seat; the smoking image classification model gives a judgment on whether the main driver is smoking, and the phone call classification model gives a judgment on whether the main driver is making a phone call. In this process, making a phone call and smoking are the same classification model, but two such classification models are required for deployment, which invisibly wastes the precious computing power resources of the platform.

[0154] Based on this, the present application provides an optional implementation method, which is to input the main driver's seat image frame sequence into a face detection model. If a face is detected, face key point regression is performed in the face ROI area; based on the key points, the ROI area near the mouth and the ROI area near the eyes are simultaneously captured and input into a multi-task image classification model to judge the smoking and phone call behaviors in the main driver's seat.

[0155] In the training process of the multi-task infrared image classification model, this method needs to use the positive samples of smoking in the ROI area near the mouth, the negative samples of making phone calls in the ROI area near the eyes, and the negative samples (including negative samples of smoking in the ROI area near the mouth and negative samples mixed with negative samples in the ROI area near the eyes) as training sets. The multi-task infrared image model can output the judgment of the phone call and smoking tasks at the same time, saving the computing power of the GPU on the board, thereby providing support for the realization of more DMS perception tasks. It can be seen that this application uses a multi-task approach to combine multiple functional tasks into one model to reduce the amount of model data.

[0156] Moreover, in the existing detection process, only the ROI area is used to judge smoking or making phone calls, but the global features are not correlated. For example, when smoking in the driver's seat, there will be a behavioral feature of arm movement, that is, the arm is different from the normal driving state, but the ROI infrared image only involves the image of the mouth area, missing the arm feature, which will make it impossible to accurately judge some short cigarette butts.

[0157] Based on this, the present application provides an optional implementation method, which can combine the full-image infrared image features to assist the ROI infrared image to determine whether there is smoking or making phone calls in the driver's seat.

[0158] When the twin network is trained in the present application, the sample images input into the twin network to be trained are two ROI images and a global image. When the ROI image is a mouth area image, the corresponding global image can include the driver's arm area. When the ROI image is an eye area image, the corresponding global image can also include the driver's arm area. This is because some drivers hold the phone to the eye area, that is, the temple area, when making a phone call. Therefore, based on the twin network training method provided in the present application, the trained twin network for object behavior recognition in the vehicle cabin can be used to determine whether the main driver's behavior is smoking or making a phone call.

[0159] Similarly, the above-mentioned judgment method for whether the driver is smoking or making a phone call can also be used to judge whether a member in the vehicle is smoking or making a phone call. That is, it can also provide support for more OMS perception tasks.

[0160] Among them, DMS stands for Driver Monitoring System, which monitors the driver and can monitor the driver's fatigue and dangerous driving behavior. OMS stands for Occupancy Monitoring System, which monitors the passengers and can monitor the passengers' age, status, emotions and behavior.

[0161] Based on this, the present application may also provide a method for identifying object behavior in a vehicle cabin using a twin network trained by the present application, specifically including: obtaining a first image of a target object (such as a driver or a member) in the cabin, and determining a second image and a third object based on the first image; the second image is a local image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be identified of the target object; the third image is a local image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be identified of the target object; the first image and the second image are input into a first sub-twin network in the twin network to obtain a first similarity probability between a first image feature corresponding to the first image and a second image feature corresponding to the second image, and the first image and the third image are input into a second sub-twin network in the twin network to obtain a second similarity probability between a third image feature corresponding to the first image and a fourth image feature corresponding to the third image; based on the first similarity probability, determining a recognition result for the first behavior to be identified of the target object, and based on the second similarity probability, determining a recognition result for the second behavior to be identified of the target object; wherein the twin network is a twin network trained based on the training method for the twin network provided by the present application.

[0162] Wherein, based on the first similarity probability, the recognition result of the first behavior to be recognized for the target object is determined, and, based on the second similarity probability, the recognition result of the second behavior to be recognized for the target object is determined, including: when it is determined that the first similarity probability is greater than a first preset threshold, the recognition result of the first behavior to be recognized for the target object is determined to be the occurrence of the first behavior to be recognized; and when it is determined that the second similarity probability is greater than a second preset threshold, the recognition result of the second behavior to be recognized for the target object is determined to be the occurrence of the second behavior to be recognized; wherein the first preset threshold and the second preset threshold are the same or different.

[0163] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0164] Based on the same inventive concept, the embodiment of the present application also provides a twin network training device for implementing the twin network training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more twin network training devices provided below can be referred to the limitations of the twin network training method above, and will not be repeated here.

[0165] In an exemplary embodiment, Figure 6 As shown, a training device 300 for a twin network is provided, comprising: a sample acquisition module 81, a sample processing module 82 and a parameter adjustment module 83, wherein:

[0166] The sample acquisition module 81 is used to acquire a sample pair set; the sample pair set includes at least a first type of sample pair and a second type of sample pair; the first type of sample pair includes a first sample image and a second sample image of the sample object, and the second type of sample pair includes a first sample image and a third sample image; the second sample image is a local image of a first target area in the first sample image, and the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to the first behavior to be identified, and the second target area is an area related to the second behavior to be identified;

[0167] The sample processing module 82 is used to input each type of sample pair into the corresponding sub-twin network in the twin network to be trained, and obtain the first sample image feature and the second sample image feature corresponding to the two sample images in the sample pair based on the feature extraction network in the sub-twin network; and input the first sample image feature and the second sample image feature into the classification network in the sub-twin network to obtain the first predicted behavior and the second predicted behavior respectively; wherein different types of sample pairs correspond to different sub-twin networks; determine the first loss value based on the difference similarity between the first sample image feature and the second sample image feature; obtain the second loss value based on the first predicted behavior and the actual behavior corresponding to the sample pair; and obtain the third loss value based on the second predicted behavior and the actual behavior;

[0168] The parameter adjustment module 83 is used to determine the sub-loss value corresponding to each sample pair based on the first loss value, the second loss value and the third loss value, and determine the total loss value of the sample pair set based on the sub-loss values ​​corresponding to at least the first type and the second type of sample pairs respectively, and when the total loss value does not meet the training stop condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of objects in the vehicle cabin.

[0169] In an exemplary embodiment, when the sample processing module 82 is used to determine the first loss value based on the difference similarity between the first sample image feature and the second sample image feature, it can be specifically used to: obtain the difference similarity between the first sample image feature and the second sample image feature; convert the difference similarity into a sample image feature similarity probability within a preset value range; determine the first loss value based on the sample image feature similarity probability, the actual behavior type value of the sample pair and the preset loss weight.

[0170] In an exemplary embodiment, the sample processing module 82 is also used to determine, for each type of sample pair, the number of first elements and the number of second elements in the first sample image feature and the second sample image feature of the type sample pair, whose element values ​​are within a preset value range; and based on the difference between the element values ​​in the first sample image feature and the second sample image feature, determine the number of third elements whose difference values ​​are within a preset difference range; based on the number of first elements, the number of second elements, the number of third elements, and the first sample image feature and the second sample image feature, determine the difference similarity between the first sample image feature and the second sample image feature in the type sample pair.

[0171] In an exemplary embodiment, when the sample processing module 82 is used to input the first sample image feature and the second sample image feature into the classification network in the sub-twin network to obtain the first prediction behavior and the second prediction behavior respectively, it is specifically used to: input the first sample image feature into the first classification network in the sub-twin network to obtain the first prediction behavior; input the second sample image feature into the second classification network in the sub-twin network to obtain the second prediction behavior.

[0172] In an exemplary embodiment, the sample processing module 82 is used to input sample pairs of each type into the corresponding sub-twin network in the twin network to be trained, and obtain the first sample image features and the second sample image features corresponding to the two sample images in the sample pairs based on the feature extraction network in the sub-twin network. It is specifically used for: for each type of sample pair, one sample image in the sample pair of the type is input into the first feature extraction network in the sub-twin network to obtain the corresponding first sample image feature; the other sample image in the sample pair of the type is input into the second feature extraction network in the sub-twin network to obtain the corresponding second sample image feature.

[0173] In an exemplary embodiment, the first feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network; the second feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network.

[0174] In an exemplary embodiment, each type of sample pair includes a positive sample pair and a negative sample pair; both sample images included in the positive sample pair are images with behaviors to be identified; and both sample images included in the negative sample pair are images without behaviors to be identified.

[0175] In an exemplary embodiment, the sample acquisition module 81 is used to acquire a sample pair set, including: acquiring multiple sample images including sample objects, and determining a positive sample image to be selected and a negative sample image to be selected from the multiple sample images; for the same behavior to be identified, acquiring a first similarity between the behavior in each of the positive sample images to be selected and the behavior to be identified, and filtering the positive sample image from the multiple positive sample images to be selected according to a comparison result of the first similarity and a first similarity threshold; acquiring a second similarity between the behavior in each of the negative sample images to be selected for the same behavior and the behavior to be identified, and filtering the negative sample image from the multiple negative sample images to be selected according to a comparison result of the second similarity and the second similarity threshold; After completing the image screening, the second sample image corresponding to the first target area and the third sample image corresponding to the second target area are respectively obtained from the screened positive sample images and negative sample images; a first type of positive sample pair is formed based on the positive sample image and the corresponding second sample image, a second type of positive sample pair is formed based on the positive sample image and the corresponding third sample image, a first type of negative sample pair is formed based on the negative sample image and the corresponding second sample image, and a second type of negative sample pair is formed based on the negative sample image and the corresponding third sample image; a sample pair set is obtained based on the first type of positive sample pairs, the first type of negative sample pairs, the second type of positive sample pairs and the second type of negative sample pairs.

[0176] Each module in the above-mentioned twin network training device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0177] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a training method for a twin network is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0178] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0179] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0180] Acquire a sample pair set; the sample pair set includes at least a first type of sample pair and a second type of sample pair; the first type of sample pair includes a first sample image and a second sample image of the sample object, and the second type of sample pair includes a first sample image and a third sample image; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified;

[0181] Input each type of sample pair into the corresponding sub-twin network in the twin network to be trained, and obtain the first sample image feature and the second sample image feature corresponding to the two sample images in the sample pair based on the feature extraction network in the sub-twin network; and input the first sample image feature and the second sample image feature into the classification network in the sub-twin network to obtain the first prediction behavior and the second prediction behavior respectively; wherein different types of sample pairs correspond to different sub-twin networks;

[0182] Determine a first loss value based on a difference similarity between a first sample image feature and a second sample image feature; obtain a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtain a third loss value based on the second predicted behavior and the actual behavior;

[0183] Based on the first loss value, the second loss value and the third loss value, the sub-loss value corresponding to each sample pair is determined. Based on the sub-loss values ​​corresponding to at least the first type and the second type of sample pairs, respectively, the total loss value of the sample pair set is determined. When the total loss value does not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the conditions for stopping training are met, so as to obtain a trained twin network. The trained twin network is applied to at least two behavior recognitions of objects in the vehicle cabin.

[0184] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0185] Acquire a sample pair set; the sample pair set includes at least a first type of sample pair and a second type of sample pair; the first type of sample pair includes a first sample image and a second sample image of the sample object, and the second type of sample pair includes a first sample image and a third sample image; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified;

[0186] Input each type of sample pair into the corresponding sub-twin network in the twin network to be trained, and obtain the first sample image feature and the second sample image feature corresponding to the two sample images in the sample pair based on the feature extraction network in the sub-twin network; and input the first sample image feature and the second sample image feature into the classification network in the sub-twin network to obtain the first prediction behavior and the second prediction behavior respectively; wherein different types of sample pairs correspond to different sub-twin networks;

[0187] Determine a first loss value based on a difference similarity between a first sample image feature and a second sample image feature; obtain a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtain a third loss value based on the second predicted behavior and the actual behavior;

[0188] Based on the first loss value, the second loss value and the third loss value, the sub-loss value corresponding to each sample pair is determined. Based on the sub-loss values ​​corresponding to at least the first type and the second type of sample pairs, respectively, the total loss value of the sample pair set is determined. When the total loss value does not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the conditions for stopping training are met, so as to obtain a trained twin network. The trained twin network is applied to at least two behavior recognitions of objects in the vehicle cabin.

[0189] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0190] Acquire a sample pair set; the sample pair set includes at least a first type of sample pair and a second type of sample pair; the first type of sample pair includes a first sample image and a second sample image of the sample object, and the second type of sample pair includes a first sample image and a third sample image; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified;

[0191] Input each type of sample pair into the corresponding sub-twin network in the twin network to be trained, and obtain the first sample image feature and the second sample image feature corresponding to the two sample images in the sample pair based on the feature extraction network in the sub-twin network; and input the first sample image feature and the second sample image feature into the classification network in the sub-twin network to obtain the first prediction behavior and the second prediction behavior respectively; wherein different types of sample pairs correspond to different sub-twin networks;

[0192] Determine a first loss value based on a difference similarity between a first sample image feature and a second sample image feature; obtain a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtain a third loss value based on the second predicted behavior and the actual behavior;

[0193] Based on the first loss value, the second loss value and the third loss value, the sub-loss value corresponding to each sample pair is determined. Based on the sub-loss values ​​corresponding to at least the first type and the second type of sample pairs, respectively, the total loss value of the sample pair set is determined. When the total loss value does not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the conditions for stopping training are met, so as to obtain a trained twin network. The trained twin network is applied to at least two behavior recognitions of objects in the vehicle cabin.

[0194] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0195] It should also be added that the images including user objects involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant images must comply with relevant regulations.

[0196] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0197] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0198] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a twin network, characterized in that: The method comprises: Acquire a sample pair set; the sample pair set includes at least a first type of sample pair and a second type of sample pair; the first type of sample pair includes a first sample image and a second sample image of a sample object, and the second type of sample pair includes a first sample image and a third sample image; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified; Input the sample pairs of each type into the corresponding sub-twin network in the twin network to be trained, and obtain the first sample image feature and the second sample image feature corresponding to the two sample images in the sample pair based on the feature extraction network in the sub-twin network; and input the first sample image feature and the second sample image feature into the classification network in the sub-twin network to obtain the first prediction behavior and the second prediction behavior respectively; wherein different types of sample pairs correspond to different sub-twin networks; Determine a first loss value based on a difference similarity between the first sample image feature and the second sample image feature; obtain a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtain a third loss value based on the second predicted behavior and the actual behavior; Based on the first loss value, the second loss value and the third loss value, the sub-loss value corresponding to each of the sample pairs is determined, and based on the sub-loss values ​​corresponding to at least the first type and the second type of the sample pairs respectively, the total loss value of the sample pair set is determined, and when the total loss value does not meet the training stop condition, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognitions of objects in a vehicle cabin.

2. The method according to claim 1, characterized in that The determining a first loss value based on the difference similarity between the first sample image feature and the second sample image feature comprises: Obtaining a difference similarity between the first sample image feature and the second sample image feature; Converting the difference similarity into a sample image feature similarity probability within a preset value interval; The first loss value is determined based on the sample image feature similarity probability, the actual behavior type value of the sample pair and the preset loss weight.

3. The method according to claim 1, characterized in that Before determining the first loss value based on the difference similarity between the first sample image feature and the second sample image feature, the method further includes: For each type of sample pair, determining the number of first elements and the number of second elements whose element values ​​are within a preset value range in the first sample image feature and the second sample image feature of the sample pair of the type; and based on the difference between the element values ​​in the first sample image feature and the second sample image feature, determining the number of third elements whose difference is within a preset difference range; Based on the first number of elements, the second number of elements, the third number of elements, and the first sample image feature and the second sample image feature, the difference similarity between the first sample image feature and the second sample image feature in the type sample pair is determined.

4. The method according to claim 1, characterized in that: The step of inputting the first sample image feature and the second sample image feature into the classification network in the sub-twin network to obtain a first predicted behavior and a second predicted behavior respectively includes: Inputting the first sample image feature into the first classification network in the sub-twin network to obtain the first predicted behavior; The second sample image feature is input into the second classification network in the sub-twin network to obtain the second predicted behavior.

5. The method according to claim 1, characterized in that The sample pairs of each type are respectively input into the corresponding sub-twin networks in the twin network to be trained, and based on the feature extraction network in the sub-twin network, first sample image features and second sample image features corresponding to the two sample images in the sample pairs are obtained, including: For each type of sample pair, input a sample image in the sample pair of the type into the first feature extraction network in the sub-twin network to obtain a corresponding first sample image feature; Another sample image in the sample pair of the type is input into the second feature extraction network in the sub-twin network to obtain corresponding second sample image features.

6. The method according to claim 1, characterized in that Each type of sample pair includes a positive sample pair and a negative sample pair; Both sample images included in the positive sample pair are images containing behaviors to be identified; Both sample images included in the negative sample pair are images in which the to-be-recognized behavior does not exist.

7. The method according to claim 6, characterized in that The obtaining of the sample pair set comprises: Acquire a plurality of sample images including sample objects, and determine a positive sample image to be selected and a negative sample image to be selected from the plurality of sample images; For the same behavior to be identified, obtaining a first similarity between the behavior in each of the to-be-selected positive sample images and the behavior to be identified, and screening a positive sample image from the plurality of to-be-selected positive sample images according to a comparison result of the first similarity and a first similarity threshold; Obtaining a second similarity between the behavior in each of the negative sample images to be selected and the behavior to be identified for the same behavior, and screening a negative sample image from the plurality of negative sample images to be selected according to a comparison result between the second similarity and a second similarity threshold; After completing the image screening, respectively obtain a second sample image corresponding to the first target area and a third sample image corresponding to the second target area from each of the screened positive sample images and each of the negative sample images; form a first type of positive sample pair based on the positive sample image and the corresponding second sample image, form a second type of positive sample pair based on the positive sample image and the corresponding third sample image, form a first type of negative sample pair based on the negative sample image and the corresponding second sample image, and form a second type of negative sample pair based on the negative sample image and the corresponding third sample image; The sample pair set is obtained based on the positive sample pairs of the first type, the negative sample pairs of the first type, the positive sample pairs of the second type, and the negative sample pairs of the second type.

8. A training device for a twin network, characterized in that: The device comprises: A sample acquisition module, used to acquire a sample pair set; the sample pair set includes at least a first type of sample pair and a second type of sample pair; the first type of sample pair includes a first sample image and a second sample image of a sample object, and the second type of sample pair includes a first sample image and a third sample image; the second sample image is a local image of a first target area in the first sample image, and the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified; A sample processing module, used to input the sample pairs of each type into the corresponding sub-twin network in the twin network to be trained, and obtain the first sample image feature and the second sample image feature corresponding to the two sample images in the sample pair based on the feature extraction network in the sub-twin network; and input the first sample image feature and the second sample image feature into the classification network in the sub-twin network, and the classification network obtains the first predicted behavior and the second predicted behavior based on the first sample image feature and the second sample image feature, respectively; wherein different types of sample pairs correspond to different sub-twin networks; determine the first loss value based on the difference similarity between the first sample image feature and the second sample image feature; obtain the second loss value based on the first predicted behavior and the actual behavior corresponding to the sample pair; and obtain the third loss value based on the second predicted behavior and the actual behavior; A parameter adjustment module is used to determine the sub-loss value corresponding to each of the sample pairs based on the first loss value, the second loss value and the third loss value, and determine the total loss value of the sample pair set based on the sub-loss values ​​corresponding to at least the first type and the second type of the sample pairs respectively, and when the total loss value does not meet the training stop condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognitions of objects in a vehicle cabin.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images