Twin network training method and device and computer equipment

Abstract

The invention relates to a twin network training method and device and computer equipment, and relates to the technical field of vehicles. The method comprises the steps of obtaining a plurality of sample pairs; inputting the sample pair into a twin network to be trained, and obtaining first and second sample image features of the first and second sample images based on a feature extraction network in the network; obtaining a first prediction behavior and a second prediction behavior of the first sample image feature and the second sample image feature through a classification network; determining a first loss value based on a difference similarity between the first and second sample image features; obtaining second and third loss values based on the first and second predicted behaviors and the actual behavior; and determining a total loss value of each sample pair based on the first to third loss values, updating network parameters and performing training again under the condition that at least one total loss value does not meet a training stopping condition until the training stopping condition is met, and obtaining a trained twin network which can be applied to object behavior recognition in the vehicle cabin.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a training method, apparatus and computer device for a twin network. Background Art

[0002] With the increase in demand for vehicle use, traffic accidents occur from time to time. Many traffic accidents are related to the driver's bad driving behavior, and some traffic accidents are caused by the bad riding behavior of the passengers in the car. In the existing vehicle use environment, drivers often do not pay attention to their driving behavior during driving, and the passengers in the car do not pay attention to their riding behavior in the car. For example, there are common bad behaviors such as drivers and passengers smoking and making phone calls in the car. Among them, similar behaviors such as drivers making phone calls while driving can easily distract the driver's driving concentration, and the smoking behavior of passengers while driving can easily cause quarrels between passengers and drivers, which will also distract the driver's driving concentration.

[0003] Although the use of vehicles facilitates people's travel, the safety of drivers and passengers is of paramount importance. In order to improve the safety of drivers and passengers in the car, real-time monitoring and intelligent evaluation of the driver's driving behavior and the passengers' riding behavior will help to detect irregular or bad behaviors of the car members as early as possible, and then remind the car members to change their relevant behavioral habits, so as to effectively reduce the problem of the driver's attention being distracted during driving, thereby helping to reduce the occurrence of traffic accidents and protect the lives of drivers and passengers.

[0004] However, in the relevant existing technologies, the accuracy of intelligent detection of bad behaviors of vehicle occupants is not ideal. Therefore, there is an urgent need to provide a behavior recognition model that can be used to accurately identify the behaviors of vehicle occupants, so as to accurately identify the bad behaviors of vehicle occupants in a shorter time. Summary of the invention

[0005] Based on this, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium and computer program product for a twin network that can be trained to perform intelligent monitoring of the behavior of members inside a vehicle with high accuracy in order to address the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a training method for a twin network, the method comprising:

[0007] Acquire multiple sample pairs; each of the sample pairs includes a first sample image and a second sample image of a sample object; the second sample image is a local image corresponding to a target area in the first sample image, and the target area is an area related to the behavior to be identified of the sample object;

[0008] Input the sample pair into the twin network to be trained, and obtain a first sample image feature corresponding to the first sample image and a second sample image feature corresponding to the second sample image in the sample pair based on a feature extraction network in the twin network to be trained; and input the first sample image feature and the second sample image feature into a classification network in the twin network to be trained, and the classification network obtains a first predicted behavior and a second predicted behavior based on the first sample image feature and the second sample image feature, respectively;

[0009] Determine a first loss value based on a difference similarity between the first sample image feature and the second sample image feature; obtain a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtain a third loss value based on the second predicted behavior and the actual behavior;

[0010] Based on the first loss value, the second loss value and the third loss value, the total loss value corresponding to each of the sample pairs is determined, and when the total loss value of at least one of the sample pairs does not meet the stop training condition, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the stop training condition is met, so as to obtain a trained twin network, and the trained twin network is applied to object behavior recognition in a vehicle cabin.

[0011] In one embodiment, determining the first loss value based on the difference similarity between the first sample image feature and the second sample image feature includes:

[0012] Obtaining a difference similarity between the first sample image feature and the second sample image feature;

[0013] Converting the difference similarity into a sample image feature similarity probability within a preset value interval;

[0014] The first loss value is determined based on the sample image feature similarity probability, the actual behavior type value of the sample pair and the preset loss weight.

[0015] In one of the embodiments, before determining the first loss value based on the difference similarity between the first sample image feature and the second sample image feature, the method further includes:

[0016] Determine respectively the number of first elements and the number of second elements in the first sample image feature and the second sample image feature, whose element values ​​are within a preset value range; and determine, based on the difference between the element values ​​in the first sample image feature and the second sample image feature, the number of third elements whose difference is within a preset difference range;

[0017] Based on the first number of elements, the second number of elements, the third number of elements, and the first sample image feature and the second sample image feature, a difference similarity between the first sample image feature and the second sample image feature is determined.

[0018] In one embodiment, the inputting the first sample image feature and the second sample image feature into the classification network in the twin network to be trained, wherein the classification network obtains a first predicted behavior and a second predicted behavior based on the first sample image feature and the second sample image feature, respectively, including:

[0019] Inputting the first sample image feature into the first classification network in the twin network to be trained to obtain the first predicted behavior;

[0020] The second sample image feature is input into the second classification network in the twin network to be trained to obtain the second predicted behavior.

[0021] In one embodiment, the step of inputting the sample pair into the twin network to be trained, and obtaining a first sample image feature corresponding to the first sample image and a second sample image feature corresponding to the second sample image in the sample pair based on a feature extraction network in the twin network to be trained, includes:

[0022] Inputting the first sample image in the sample pair into the first feature extraction network in the twin network to be trained to obtain a first sample image feature;

[0023] The second sample image in the sample pair is input into the second feature extraction network in the twin network to be trained to obtain the second sample image feature.

[0024] In one embodiment, the first feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network;

[0025] The second feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network.

[0026] In one embodiment, the plurality of sample pairs include positive sample pairs and negative sample pairs;

[0027] The first sample image and the second sample image included in the positive sample pair are both images containing behaviors to be identified;

[0028] The first sample image and the second sample image included in the negative sample pair are both images in which the to-be-recognized behavior does not exist.

[0029] In one embodiment, obtaining a plurality of sample pairs includes:

[0030] Acquire a plurality of sample images including sample objects, and determine a positive sample image to be selected and a negative sample image to be selected from the plurality of sample images;

[0031] Acquire a first similarity between the behavior in each of the to-be-selected positive sample images and the to-be-recognized behavior, and select a positive sample image from the plurality of to-be-selected positive sample images according to a comparison result of the first similarity and the first similarity threshold;

[0032] Obtaining a second similarity between the behavior in each of the negative sample images to be selected and the behavior to be identified, and screening a negative sample image from the plurality of negative sample images to be selected according to a comparison result of the second similarity and the second similarity threshold;

[0033] After completing the image screening, the second sample image corresponding to the target area is obtained from each of the screened positive sample images and each of the negative sample images; the positive sample pair is formed based on the positive sample image and the corresponding second sample image, and the negative sample pair is formed based on the negative sample image and the corresponding second sample image.

[0034] In a second aspect, the present application also provides a training device for a twin network, the device comprising:

[0035] A sample acquisition module, used to acquire a plurality of sample pairs; each of the sample pairs comprises a first sample image and a second sample image of a sample object; the second sample image is a local image corresponding to a target area in the first sample image, and the target area is an area related to the behavior to be identified of the sample object;

[0036] A sample processing module, used for inputting the sample pair into the twin network to be trained, obtaining a first sample image feature corresponding to the first sample image in the sample pair and a second sample image feature corresponding to the second sample image based on the feature extraction network in the twin network to be trained; and inputting the first sample image feature and the second sample image feature into a classification network in the twin network to be trained, wherein the classification network obtains a first predicted behavior and a second predicted behavior based on the first sample image feature and the second sample image feature, respectively; determining a first loss value based on a difference similarity between the first sample image feature and the second sample image feature; obtaining a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtaining a third loss value based on the second predicted behavior and the actual behavior;

[0037] A parameter adjustment module is used to determine the total loss value corresponding to each of the sample pairs based on the first loss value, the second loss value and the third loss value, and when the total loss value of at least one of the sample pairs does not meet the stop training condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the stop training condition is met, so as to obtain a trained twin network, and the trained twin network is applied to object behavior recognition in a vehicle cabin.

[0038] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any step in the above-mentioned first aspect when executing the computer program.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any step in the above-mentioned first aspect when executed by a processor.

[0040] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements any step in the above-mentioned first aspect when executed by a processor.

[0041] The present application provides a training method, apparatus, computer device, computer-readable storage medium and computer program product for a twin network. The training method for the twin network described above obtains a plurality of sample pairs; each sample pair includes a first sample image and a second sample image of a sample object; the second sample image is a local image corresponding to a target area in the first sample image, and the target area is an area related to the behavior to be identified of the sample object; the sample pairs are input into the twin network to be trained, and a first sample image feature corresponding to the first sample image in the sample pair and a second sample image feature corresponding to the second sample image in the sample pair are obtained based on a feature extraction network in the twin network to be trained; and the first sample image feature and the second sample image feature are input into a classification network in the twin network to be trained, and the classification network is based on the first The sample image feature and the second sample image feature respectively obtain the first predicted behavior and the second predicted behavior; based on the difference similarity between the first sample image feature and the second sample image feature, determine the first loss value; based on the first predicted behavior and the actual behavior corresponding to the sample pair, obtain the second loss value; and, based on the second predicted behavior and the actual behavior, obtain the third loss value; based on the first loss value, the second loss value and the third loss value, determine the total loss value corresponding to each sample pair, and when the total loss value of at least one sample pair does not meet the training stop condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the training stop condition is met, to obtain a trained twin network, and the trained twin network is applied to object behavior recognition in a vehicle cabin.

[0042] It can be seen that the present application uses the first sample image and the local image of the target area related to the behavior to be identified in the first sample image (the second sample image) as training samples of the twin network, so that the sample image includes not only the image of the target area of ​​the sample object, but also the global image of the sample object, which increases the type of sample images; and other feature information associated with the target area in the global image is conducive to auxiliary identification of the behavior of the sample object in the target area, and is conducive to improving the recognition accuracy of the subsequently trained twin network for the behavior to be identified of the object in the image; and in the process of training the twin network, based on the difference similarity between the first sample image feature and the second sample image feature, the first loss value of the first sample image feature and the second sample image feature corresponding to the sample image respectively is determined, and the use of the difference similarity is conducive to improving the accuracy of the obtained first loss value, and then it is conducive to improving the recognition accuracy of the trained twin network for the behavior to be identified of the object in the image; and then based on the first loss value and the second loss value and the third loss value corresponding to the first sample image feature and the second sample image feature respectively, the total loss value corresponding to each sample pair is obtained, so as to achieve good training of the twin network to be trained based on the total loss value. In summary, the twin network trained based on the method provided in this application can efficiently and accurately identify the specific behaviors of the occupants in the car based on pictures of the occupants in the car. For example, it can identify the bad behaviors of the occupants in the car. This helps to detect the irregular or bad behaviors of the occupants in the car as early as possible, and then remind the occupants in the car to change their relevant behavioral habits, so as to effectively reduce the problem of the driver's attention being distracted during driving, thereby helping to reduce the occurrence of traffic accidents and protect the lives of drivers and passengers. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A diagram of an application environment for training a twin network in one embodiment;

[0045] Figure 2 This is an application environment diagram of a trained twin network in one embodiment;

[0046] Figure 3 It is a flowchart of a training method of a twin network in one embodiment;

[0047] Figure 4A schematic diagram of the structure of a twin network in one embodiment;

[0048] Figure 5 A schematic diagram of the structure of a twin network in another embodiment;

[0049] Figure 6 It is a structural block diagram of a training device of a twin network in one embodiment;

[0050] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The training method of the twin network provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 902 communicates with the server 904 through the network. The terminal 902 can at least be used to obtain sample images for training the twin network to be trained, and perform certain preprocessing operations on the sample images to obtain the first sample image corresponding to the sample image; in addition, the classification labeling work for the sample image can also be realized based on the terminal 902. The data storage system can store the data that the server 904 needs to process. The data storage system can be integrated on the server 904, or it can be placed on the cloud or other network servers. The data storage system can be used to store at least a plurality of sample pairs including a first sample image and a second sample image, as well as a twin network to be trained and a twin network that has been trained. Among them, the terminal 902 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server 904 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0053] It should be noted that the training process of the twin network can be based on the server 904 to provide operations, or it can be based on the terminal 902 to provide related operations, or it can be based on the communication between the terminal 902 and the server 904 to implement related operations. This application does not make specific limitations on this.

[0054] Furthermore, the trained twin network provided in the embodiment of the present application can be applied to Figure 2 The application environment shown. Figure 2This is only one application environment that can be selected for the trained twin network provided by this application, and this application is not limited to this; that is, the trained twin network can not only be used for Figure 2 The driver / passenger behavior recognition in the vehicle environment shown can also be used for example for worker behavior recognition in a factory environment, and can also be used for example for student behavior recognition in a teaching environment, etc. Specifically, the application scenario includes a vehicle 100, the vehicle 100 includes a vehicle controller 101, and the vehicle cabin includes an image acquisition device 102.

[0055] The vehicle controller 101 may first acquire an image of a target object (e.g., at least one of the driver and the passenger) in the cabin through the image acquisition device 102, and then process the acquired image to obtain a first image of the target object, and store the first image. In the process of object behavior recognition based on the twin network trained in this application, the vehicle controller 101 may first acquire the first image of the target object, and determine the second image based on the first image, that is, obtain the local image (second image) of the target object from the global image (first image) of the target object; further, the vehicle controller 101 inputs the first image and the second image into the twin network trained in this application, so as to obtain the image feature similarity probability between the two images based on the twin network trained in this application; finally, based on the image feature similarity probability, determine the recognition result of the behavior to be recognized for the target object, so as to determine the specific behavior of the target object.

[0056] In order to provide a behavior recognition model that can be used to accurately identify the behavior of members inside the vehicle, so as to accurately identify the bad behavior of the occupants in the vehicle in a shorter time, the present application provides a twin network training method to train a twin network that can recognize the behavior of members inside the vehicle with high accuracy.

[0057] In an exemplary embodiment, Figure 3 As shown, a training method for a twin network is provided, including but not limited to the following steps S201 to S204. Among them:

[0058] S201, obtaining multiple sample pairs; each sample pair includes a first sample image and a second sample image of a sample object; the second sample image is a local image corresponding to a target area in the first sample image, and the target area is an area related to the behavior to be identified of the sample object.

[0059] The sample object may be a driver or a passenger in the vehicle. The first sample image may be a first sample image including at least one of the driver and the passenger. Assuming that the behavior to be identified is whether the driver smokes, the target area is an area related to the smoking behavior, such as the driver's mouth area. In this case, the second sample image is a local image corresponding to the mouth area in the first sample image of the driver. The local image corresponding to the target area is also a face ROI image. ROI (Region of Interest) refers to a specific area in an image that is processed. The area to be processed is usually outlined by a box, circle, ellipse, irregular polygon, etc.; that is, the second sample image may be, for example, an image framed by a box shape from the first sample image.

[0060] In an optional implementation, before step S201, an image including the sample object acquired by an image acquisition device in the vehicle cabin may be acquired, and the image may be preprocessed to obtain the image including the sample object as the first sample image. The image preprocessing may include denoising, cropping, etc., which is not specifically limited in this application.

[0061] The image acquisition device may be, for example, an infrared sensor or a visible light sensor. The infrared sensor may be, for example, an infrared camera for acquiring infrared images. The visible light sensor may be, for example, an ordinary camera for acquiring color images, which may also be referred to as RGB images, where R is a red channel, G is a green channel, and B is a blue channel. The image of the sample object may be a single-channel infrared image of the vehicle cabin including the sample object acquired by the infrared sensor, or an RGB color image of the vehicle cabin including the sample object acquired by the visible light sensor. The image in the infrared image sample pair may be an image formed based on obtaining the radiation of the sample object in the infrared band.

[0062] A further optional implementation is provided, where the acquired image of the sample object is a single-channel infrared image, it can be further converted into a three-channel infrared image to serve as the first sample image of the sample object.

[0063] Each sample pair includes two images, specifically a first sample image including a sample object, and a second sample image cropped based on a region of the first sample image that is related to the behavior to be identified of the sample object.

[0064] S202, input the sample pair into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image and the second sample image feature corresponding to the second sample image in the sample pair based on the feature extraction network in the twin network to be trained; and input the first sample image feature and the second sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first predicted behavior and the second predicted behavior based on the first sample image feature and the second sample image feature, respectively.

[0065] Among them, the twin network (twin neural network) to be trained provided in the present application may specifically include a feature extraction network and a classification network, that is, the twin network is at least composed of a feature extraction network and a classification network, wherein the input data of the feature extraction network is a sample pair, and the input data of the classification network is the data output by the feature extraction network after processing the sample pair.

[0066] In an optional embodiment, the first sample image and the second sample image included in the sample pair are respectively input into the feature extraction network in the twin network to be trained provided in the present application to obtain the first sample image feature based on the feature extraction network after processing the first sample image, and the second sample image feature based on the feature extraction network after processing the second sample image; then, the first sample image feature and the second sample image feature are respectively input into the classification network in the twin network to be trained provided in the present application to obtain the first prediction behavior based on the classification network after processing the first sample image feature, and the second prediction behavior based on the classification network after processing the second sample image feature.

[0067] In an exemplary embodiment, the first sample image feature may be an image feature vector, and the second sample image feature may be an image feature vector; for example, features in the first sample image (e.g., the full image) are extracted through a feature extraction network to obtain a first feature vector (first sample image feature), and features in the second sample image (e.g., the ROI image) are extracted through a feature extraction network to obtain a second feature vector (second sample image feature).

[0068] In an exemplary embodiment, the classification network is used to receive the first sample image features corresponding to the first sample image to output a prediction of a specific behavior of the sample object in the first sample image (first predicted behavior); the classification network is also used to receive the second sample image features corresponding to the second sample image to output a prediction of a specific behavior of the sample object in the second sample image (second predicted behavior). Wherein, if the first sample image includes, for example, smoking behavior, then the corresponding first predicted behavior may be smoking behavior; if the sample image includes, for example, making a phone call, then the corresponding first predicted behavior may be making a phone call; the second predicted behavior is similar and will not be described in detail here.

[0069] It should also be added that the classification network is based on predictions of pre-set behavior types that need to be identified. For example, when the pre-set behavior types that need to be identified include smoking behavior, the classification network will output an output result that the first predicted behavior is smoking behavior.

[0070] S203, determining a first loss value based on the difference similarity between the first sample image feature and the second sample image feature; obtaining a second loss value based on the first predicted behavior and the actual behavior corresponding to the sample pair; and obtaining a third loss value based on the second predicted behavior and the actual behavior.

[0071] The first predicted behavior obtained by executing step S202 of the present application is a prediction result of the behavior to be identified of the sample object in the first sample image based on the feature extraction network and the classification network in the twin network to be trained provided by the present application; the user can also determine the actual behavior of the sample object included in the first sample image through expert identification (artificial identification), and then, based on the difference between the first predicted behavior and the actual behavior corresponding to the sample pair for the same first sample image, for example, through data processing of the numerical value corresponding to the behavior type of the first predicted behavior and the numerical value corresponding to the behavior type of the actual behavior, a second loss value for the first predicted behavior and the actual behavior can be obtained. The second predicted behavior obtained by executing step S202 of the present application is a prediction result of the to-be-identified behavior of the sample object in the second sample image based on the feature extraction network and the classification network in the twin network to be trained provided by the present application; the user can also determine the actual behavior of the sample object in the second sample image based on the actual behavior of the sample object included in the first sample image corresponding to the determined second sample image, and then, based on the difference between the second predicted behavior for the same second sample image and the actual behavior corresponding to the sample pair, for example, by processing the numerical value corresponding to the behavior type of the second predicted behavior and the numerical value corresponding to the behavior type of the actual behavior, a third loss value for the second predicted behavior and the actual behavior can be obtained.

[0072] While obtaining the second loss value and the third loss value, the first loss value corresponding to the first sample image feature and the second sample image feature can also be obtained based on the difference similarity between the first sample image feature and the second sample image feature. The calculation of the loss value can be implemented based on the loss function.

[0073] S204, based on the first loss value, the second loss value and the third loss value, determine the total loss value corresponding to each sample pair, and when the total loss value of at least one sample pair does not meet the conditions for stopping training, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the conditions for stopping training are met, so as to obtain a trained twin network, and the trained twin network is applied to object behavior recognition in the vehicle cabin.

[0074] Among them, after obtaining the first loss value, the second loss value and the third loss value of the sample pair based on the execution of step S203, the total loss value of the sample pair can be further determined based on these three loss values, and then it can be judged whether the total loss value of the sample pair meets the stop training condition of the twin network to be trained. When the total loss value of the sample does not meet the stop training condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample pair used in a training process, and the twin network is trained again until the total loss values ​​of multiple sample pairs meet the stop training condition, and the training process of the twin network can be ended, and the trained twin network can be obtained at this time. Since the data input to the twin network to be trained in this application is a cabin image including sample objects, and at least part of the sample objects included in the cabin image have behaviors to be identified, the twin network trained in this application can be used for (pre-set) behavior recognition of objects in the vehicle cabin; and then, when the behavior of objects in the vehicle cabin is subsequently identified based on the trained twin network, the accuracy of object behavior recognition in the vehicle cabin can be improved.

[0075] It should be noted that the sample pair provided above is an embodiment of a cabin image including a sample object, which is only an optional implementation provided by the present application, and the present application is not limited thereto; the specific image type including the sample object used to train the twin network can be selected based on the different requirements of the application scenarios of the trained twin network. For example, the sample pair that can be selected is a classroom image including a sample object (a student). In this case, the corresponding preset behaviors can be sleeping behavior, face covering behavior, eating behavior, etc.

[0076] The training method of the above-mentioned twin network obtains multiple sample pairs; each sample pair includes a first sample image and a second sample image of the sample object; the second sample image is a local image corresponding to a target area in the first sample image, and the target area is an area related to the behavior to be identified of the sample object; the sample pairs are input into the twin network to be trained, and the first sample image features corresponding to the first sample image and the second sample image features corresponding to the second sample image in the sample pair are obtained based on the feature extraction network in the twin network to be trained; and the first sample image features and the second sample image features are input into the classification network in the twin network to be trained, and the classification network obtains the first prediction image based on the first sample image features and the second sample image features, respectively. behavior and a second predicted behavior; determine a first loss value based on the difference similarity between the first sample image feature and the second sample image feature; obtain a second loss value based on the first predicted behavior and the actual behavior corresponding to the sample pair; and obtain a third loss value based on the second predicted behavior and the actual behavior; determine the total loss value corresponding to each sample pair based on the first loss value, the second loss value and the third loss value, and when the total loss value of at least one sample pair does not meet the training stop condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to object behavior recognition in the vehicle cabin.

[0077] It can be seen that the present application uses the first sample image and the local image of the target area related to the behavior to be identified in the first sample image (the second sample image) as training samples of the twin network, and in the process of training the twin network, based on the difference similarity between the first sample image feature and the second sample image feature, the first loss value of the first sample image feature and the second sample image feature corresponding to the sample image is determined, and then based on the first loss value and the second loss value and the third loss value corresponding to the first sample image feature and the second sample image feature, the total loss value corresponding to each sample pair is obtained, so as to achieve good training of the twin network to be trained based on the total loss value; the twin network trained based on the method provided by the present application can efficiently and accurately identify the specific behavior of the occupants in the car based on the pictures including the occupants in the car, for example, it can identify the bad behavior of the occupants in the car, which helps to discover the irregular or bad behavior of the occupants in the car as early as possible, and then remind the occupants in the car to change their relevant behavioral habits, so as to effectively reduce the problem of distraction of the driver's attention during driving, thereby helping to reduce the occurrence of traffic accidents and protect the lives of drivers and passengers.

[0078] In an exemplary embodiment, the step of determining the first loss value based on the difference similarity between the first sample image feature and the second sample image feature in step S203 may be specifically performed as follows, including steps S231 to S233 (not shown), wherein:

[0079] S231, obtaining a difference similarity between a first sample image feature and a second sample image feature;

[0080] S232, converting the difference similarity into a sample image feature similarity probability within a preset value range;

[0081] S233, determining a first loss value based on the sample image feature similarity probability, the actual behavior type value of the sample pair, and the preset loss weight.

[0082] Specifically, firstly, based on the calculation formula of difference similarity, the difference similarity between the first sample image feature and the second sample image feature is calculated, that is, the difference similarity corresponding to the two images in the sample pair is obtained.

[0083] Afterwards, the difference similarity is converted into a sample image feature similarity probability within a preset value interval, for example, the difference similarity is converted into a sample image feature relative probability within the interval [0,1]. The sample image feature relative probability is used to determine whether the behaviors of the two sample images in the sample pair are the same; for example, when the sample image feature relative probability is greater than the preset smoking probability, then it can be determined that the sample objects in the two images of the sample pair both have smoking behavior.

[0084] Finally, based on the sample image feature similarity probability obtained by executing step S232, the actual behavior type value of the sample object in the determined sample pair image, and the preset loss weight, a first loss value associated with a sample pair can be determined. The actual behavior type value is, for example, when the actual behavior is the same as the preset behavior to be detected, the actual behavior type value can be set to 1, and when the actual behavior is different from the preset behavior to be detected, the actual behavior type value can be set to 0.

[0085] In an exemplary embodiment, before determining the first loss value based on the difference similarity between the first sample image feature and the second sample image feature in step S203, the method further includes steps 321 and 322, wherein:

[0086] Step 321, respectively determining the number of first elements and the number of second elements in the first sample image feature and the second sample image feature, whose element values ​​are within a preset value range; and determining the number of third elements whose difference values ​​are within a preset difference range based on the difference between the element values ​​in the first sample image feature and the second sample image feature;

[0087] Step 322: Determine the difference similarity between the first sample image feature and the second sample image feature based on the first element number, the second element number, the third element number, and the first sample image feature and the second sample image feature.

[0088] Among them, the first number of elements is the element value of the element in the first sample image feature, and the number of elements within the preset value range; the second number of elements is the element value of the element in the second sample image feature, and the number of elements within the preset value range; regarding the determination of the third number of elements, the difference between the element values ​​of the elements in the first sample image feature and the second sample image feature is first determined, and the number of elements corresponding to the difference within the preset difference range is the third number of elements.

[0089] The difference similarity between the first sample image feature and the second sample image feature needs to be determined based on at least the first element number, the second element number, the third element number, and the first sample image feature and the second sample image feature.

[0090] In an exemplary embodiment, the above step S202 is performed by inputting the first sample image feature and the second sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first prediction behavior and the second prediction behavior based on the first sample image feature and the second sample image feature, respectively. It can be optionally executed as follows: inputting the first sample image feature into the first classification network in the twin network to be trained to obtain the first prediction behavior; inputting the second sample image feature into the second classification network in the twin network to be trained to obtain the second prediction behavior.

[0091] In an exemplary embodiment, the above step S202 is performed by inputting the sample pair into the twin network to be trained, and obtaining the first sample image feature corresponding to the first sample image in the sample pair and the second sample image feature corresponding to the second sample image based on the feature extraction network in the twin network to be trained. It can be optionally executed as follows: inputting the first sample image in the sample pair into the first feature extraction network in the twin network to be trained to obtain the first sample image feature; inputting the second sample image in the sample pair into the second feature extraction network in the twin network to be trained to obtain the second sample image feature.

[0092] Specifically, an optional implementation method is provided, in which the feature extraction network included in the twin network to be trained provided in the present application specifically includes a first feature extraction network and a second feature extraction network, and the classification network included in the twin network to be trained specifically includes a first classification network and a second classification network; wherein the first classification network is used to receive the result output by the corresponding first feature extraction network, and the second classification network is used to receive the result output by the corresponding second feature extraction network.

[0093] That is, it can be considered that the twin network to be trained provided by the present application includes two sub-twin networks, the first sub-twin network includes a first feature extraction network and a first classification network, and the second sub-twin network includes a second feature extraction network and a second classification network. The first sub-twin network is used to receive and process one image in a sample pair, and the second sub-twin network is used to receive the other image in the same sample pair.

[0094] For example, the first sample image in the sample pair is input into the first feature extraction network, the first feature extraction network outputs the first sample image feature, and then the first sample image feature is input into the first classification network, and the first predicted behavior of the sample object in the first sample image is obtained based on the corresponding first classification network output; the second sample image in the sample pair is input into the second feature extraction network, the second feature extraction network outputs the second sample image feature, and then the second sample image feature is input into the second classification network, and the second predicted behavior of the sample object in the second sample image is obtained based on the corresponding second classification network output. For example, in the case where there is smoking behavior in the sample image of the sample pair, the sample pair is input into the twin network to be trained provided in the present application, and the first prediction result and the second prediction result output by the classification network may both be "smoking behavior".

[0095] It should also be added that, regarding the calculation of the second loss value and the third loss value, a suitable loss function for image classification can be selected for relevant calculations. For example, the first loss function used to calculate the second loss value and the second loss function used to calculate the third loss value may include but are not limited to the mean square error loss function, the mean absolute error loss function or the cross entropy loss function. Among them, the first loss function and the second loss function may be the same or different. For example, the loss function used to calculate the second loss value and the third loss value may both select the cross entropy loss function, so as to calculate the loss value between the predicted behavior type value and the actual behavior type value of a corresponding set of predicted behaviors based on the cross entropy loss function.

[0096] In an exemplary embodiment, the first feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network; the second feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network.

[0097] Specifically, the twin networks to be trained provided in the present application include a first feature extraction network and a second feature extraction network. Both networks can choose to use convolutional neural networks. The specific type of convolutional neural network can be, for example, any one of the lightweight neural network MobileNet and the target detection network YOLO.

[0098] It should also be added that for the first feature extraction network and the second feature extraction network included in the twin network to be trained, the same convolutional neural network can be selected, or different convolutional neural networks can be selected. For example, when the first feature extraction network selects MobileNet-V3, the second feature extraction network also selects MobileNet-V3; when the first feature extraction network selects MobileNet-V3, the second feature extraction network selects YOLO-V8; when the first feature extraction network selects YOLO-V8, the second feature extraction network selects MobileNet-V3; when the first feature extraction network selects YOLO-V8, the second feature extraction network also selects YOLO-V8.

[0099] The MobileNet-V3 and YOLO-V8 provided here are optional versions corresponding to the convolutional neural network MobileNet and the convolutional neural network YOLO, respectively, but this application does not limit the specific version selection and can be adjusted and used according to needs.

[0100] In an exemplary embodiment, the multiple sample pairs include positive sample pairs and negative sample pairs; the first sample image and the second sample image included in the positive sample pair are both images with behaviors to be identified; the first sample image and the second sample image included in the negative sample pair are both images without behaviors to be identified.

[0101] Specifically, the sample pairs collected by the present application for implementing the training of the twin network to be trained include both positive sample pairs and negative sample pairs, wherein the behaviors of the sample objects in the images of the positive sample pairs include preset behaviors to be identified. For example, when the preset behaviors to be identified are smoking behaviors, the sample objects (such as drivers) in the first sample image and the second sample image of the same sample pair both have smoking behaviors. The behaviors of the sample objects in the images of the negative sample pairs do not include preset behaviors to be identified. For example, when the preset behaviors to be identified are smoking behaviors, the sample objects (such as drivers) in the first sample image and the second sample image of the same sample pair both have smoking behaviors.

[0102] By selecting training sample pairs for training the twin network, including both positive sample pairs and negative sample pairs, it is helpful to improve the recognition accuracy of the finally trained twin network for the behavior to be recognized.

[0103] It should also be added that the application of the trained twin network to the object behavior recognition in the vehicle cabin is only an optional application scenario provided by the application; however, in other similar scenarios, the twin network trained based on the training method provided by the application can also be used to recognize the behavior of objects in the scene; for example, the twin network trained based on the training method provided by the application can be used to recognize the behavior of workers in a factory environment; the twin network trained based on the training method provided by the application can be used to recognize the behavior of students in a classroom environment. Based on the different application scenarios of the twin network, the collection scenario of its training samples can also be adapted to the corresponding scenario, which is conducive to improving the behavior recognition effect of the trained twin network.

[0104] In an exemplary embodiment, the acquisition of multiple sample pairs in step S201 may be specifically implemented by executing steps S211 to S214 (not shown), wherein:

[0105] S211, acquiring a plurality of sample images including sample objects, and determining a positive sample image to be selected and a negative sample image to be selected from the plurality of sample images;

[0106] S212, obtaining a first similarity between the behavior in each of the candidate positive sample images and the behavior to be identified, and screening a positive sample image from the plurality of candidate positive sample images according to a comparison result of the first similarity and a first similarity threshold;

[0107] S213, obtaining a second similarity between the behavior in each negative sample image to be selected and the behavior to be identified, and screening a negative sample image from a plurality of negative sample images to be selected according to a comparison result between the second similarity and a second similarity threshold;

[0108] S214, after completing the image screening, respectively obtain the second sample image corresponding to the target area from each screened positive sample image and each negative sample image; form a positive sample pair based on the positive sample image and the corresponding second sample image, and form a negative sample pair based on the negative sample image and the corresponding second sample image.

[0109] Specifically, a plurality of sample images including sample objects are first obtained, and then these sample images are divided into positive sample images to be selected and negative sample images to be selected; then, a first similarity between the behavior of the sample object in each positive sample image to be selected and the behavior to be identified is calculated, that is, the similarity probability between the behavior of the sample object in the positive sample image to be selected and the behavior to be identified, and then, according to the comparison result of the first similarity and the first similarity threshold, the positive sample images to be selected whose first similarity is greater than or equal to the first similarity threshold are screened out as the screened positive sample images; at the same time, a second similarity between the behavior of the sample object in each negative sample image to be selected and the behavior to be identified is calculated, that is, the similarity probability between the behavior of the sample object in the negative sample image to be selected and the behavior to be identified, and then, according to the comparison result of the second similarity and the second similarity threshold, the negative sample images to be selected whose second similarity is less than the second similarity threshold are screened out as the screened negative sample images.

[0110] It should be added that the present application does not limit the type of specific calculation formula applied to the behavior of the sample object in the sample image and the similarity probability between the behavior to be identified, and a suitable similarity calculation formula can be selected according to needs.

[0111] It should also be added that when the sample pair selected is a vehicle cabin image that includes a sample object, the image used to train the twin network to be trained must include a vehicle cabin image, but it is not limited to all images being vehicle cabin images, and some other images may also be included.

[0112] In addition, it should be added that for the twin network applied to the same application scenario, when training the twin network, the sample images of the sample pairs used can be collected in the corresponding same application scenario, or can be collected in multiple application scenarios. This application does not make specific limitations on this.

[0113] With respect to obtaining a sample pair in step S201, the present application also provides an optional implementation method: obtaining a first sample image of a sample object in a vehicle cabin; inputting the first sample image into a face detection model, and when it is detected that the sample object is contained in the first sample image, outputting face information corresponding to the sample object; based on the face information, capturing a face image corresponding to the sample object from the first sample image; inputting the face image into a face key point detection model to obtain the face key point coordinates of the sample object; determining a target area based on the face key point coordinates, and obtaining a second sample image corresponding to the target area from the face image; and forming a sample pair based on the first sample image and the second sample image of the sample object.

[0114] With respect to the training method of the twin network for object behavior recognition in a vehicle cabin provided in the present application, the present application provides an optional embodiment, and the specific training process is shown in the following steps S1 to S6, wherein:

[0115] S1, obtaining a plurality of sample pairs; wherein each sample pair includes a full-image smoking sample image (first sample image) and a mouth ROI smoking sample image (second sample image), wherein the mouth ROI smoking sample image is Figure 5 In the figure, D1 is used to represent the full smoking sample image. Figure 5 In the figure, D2 is used to represent the same sample pair. For the same sample pair, the mouth ROI smoking sample image D1 is obtained by intercepting the area related to the smoking behavior to be identified in the full-image smoking sample image D2.

[0116] S2, extract the ROI features of the data set D1 through the convolutional neural network 1, and obtain the feature vector G w (D1), which is the second sample image feature mentioned above; the full image feature of the data set D2 is extracted through the convolutional neural network 2 to obtain the feature vector G w (D2), that is, the first sample image feature mentioned above; wherein, the convolutional neural network 1 can adopt the lightweight neural network MobileNet-V3 or the target detection network YOLO-V8 based on demand, and the convolutional neural network 2 can adopt the lightweight neural network MobileNet-V3 or the target detection network YOLO-V8 based on demand. At the same time, parameter sharing can be achieved between the two networks.

[0117] S3, calculate the feature vector G by measuring the difference vector similarity w (D1) and the eigenvector G w The specific calculation formula is shown in formula (1).

[0118] (1)

[0119] Among them, G in formula (1) w (D2) is the first feature vector, which represents the first sample image feature; G w (D1) is the second feature vector, which represents the second sample image feature; e It means G w The number of elements in (D1) that are not equal to 0; f It means G w The number of elements in (D2) that are not equal to 0; ef It means G w (D1) and G w The number of elements in the difference of (D2) that are not equal to 0.

[0120] S4, the feature vector G w (D1) and the eigenvector G w (D2) The similarity passes through the fully connected layer and then the Sigmoid layer, and is converted into the numerical sample relative probability RPS in the interval [0,1], that is, the similar probability of judging that the samples are all smoking. When RPS>0.5, it means that the two models judge that the samples are all smoking, indicating that the judgment of the ROI image classification model (the second classification network) on the positive sample image of smoking in the mouth ROI and the judgment of the global image classification task (the first classification network) on the positive sample image of smoking in the whole image are more consistent. Among them, when the twin network (model) is trained, the model will tend to judge the positive sample of smoking.

[0121] That is, if the probability of similarity of sample image features is higher, it means that the recognition result of the first classification network in the twin network for the target object's to-be-identified behavior in the first sample image (full image) is more consistent with the recognition result of the second classification network for the target object's to-be-identified behavior in the second sample image (partial image).

[0122] S5, construct the cross entropy loss function Lrps based on RPS, and the relevant construction formula can be shown in formula (2).

[0123] (2)

[0124] Among them, L in formula (2) rps It represents the first loss value, that is, the loss value between the first sample image and the second sample image; RPS represents the probability of similarity of sample image features, that is, the relative probability of the feature vector of the sample image above being converted into a numerical sample within a preset interval; λ represents the loss weight (preset loss weight); It represents the label of the sample pair, that is, the actual behavior type of the sample pair. If the sample pair is a positive sample pair, then is 1, if the sample pair is a negative sample pair, then is 0; It represents the regularization function, which can enhance the generalization ability of the model.

[0125] Among them, when training the model (twin network), the similarity metric and classification task can assist each other in learning, so that the model can not only distinguish whether two samples are similar, but also identify the category of a single sample; in addition, since the ROI image classification task and the full-image image classification task have different noise patterns, learning the two image classification tasks at the same time can average the noise, share their respective overfitting risks, and obtain a more generalized representation.

[0126] S6, the model includes two classification tasks (one is the ROI image classification task, the other is the full image classification task) and a similarity measurement task, so the total loss function consists of three parts, that is, the total loss value consists of the first loss value, the second loss value and the third loss value. The calculation method of the total loss value is shown in formula (3).

[0127] (3)

[0128] Among them, L in formula (3) represents the total loss value of the sample pair; λ represents the loss weight; G w (D2) is the first feature vector, which represents the first sample image feature; G w (D1) is the second feature vector, which represents the second sample image feature; L rps It represents the first loss value, that is, the loss value between the first sample image and the second sample image; L cls (G w (D1)) represents the second loss value, that is, the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object; L cls (G w (D2)) represents the third loss value, that is, the loss value between the second predicted behavior and the actual behavior of the sample object.

[0129] In the training method of the twin network for object behavior recognition in the vehicle cabin shown in steps S1-S6, the behavior to be recognized is set to smoking behavior, but this is only an optional embodiment provided by the present application; the behavior to be recognized can also be set to making a phone call or other behaviors as required. The first sample image and the second sample image in the same sample pair can both be infrared images or RGB images, and multiple sample pairs can include both infrared images and RGB images.

[0130] In addition, the present application also provides an iterative process of a deep learning model of infrared images based on a twin neural network. When it is necessary to train the twin network to identify smoking behavior or phone calling behavior, it may include the following contents: select a positive sample training set Data1 containing 6,000 infrared cockpit images, of which Data1.1 is 3,000 infrared images of the phone calling ROI area near the eyes, and Data1.2 is 3,000 infrared images of the smoking ROI area near the mouth; the negative sample training set Data2 contains 6,000 infrared cockpit images, of which Data2.1 is 2,000 infrared images of the non-phone calling ROI area near the eyes, Data2.2 is 2,000 infrared images of the non-smoking ROI area near the mouth, and Data2.3 is 2,000 other negative samples of non-phone calling and non-smoking cockpit ROI infrared images. Use Data1 and Data2 as the initial training sets to train an initial model for multi-task infrared image classification.

[0131] Furthermore, the multi-task infrared image classification initial model can be used to screen a dataset of 1 million images, including the following:

[0132] Screening positive samples: When screening positive samples for making phone calls, positive samples with scores lower than 0.1 are added to the Data1.1 training set; when screening positive samples for smoking, positive samples with scores lower than 0.1 are added to the Data1.2 training set;

[0133] Filter negative samples: add non-calling negative samples with a screening score greater than 0.9 to the Data2.1 training set; add non-calling and smoking negative samples with a screening score greater than 0.9 to the Data2.2 training set; add non-calling and non-smoking negative samples with a screening score greater than 0.9 to the Data2.3 training set. Retrain the updated Data1 and Data2 with the multi-task infrared image classification initial model to obtain the multi-task infrared image classification model. Before adding to the initial training set, the positive and negative samples to be added will be browsed to see if there are negative samples mixed in the positive samples and positive samples mixed in the negative samples.

[0134] For example, when the positive sample threshold is set to 0.1 and the negative sample threshold is set to 0.9 to filter the sample set, if the number of positive and negative samples is less than 1000, the positive sample threshold is set to 0.2 and the negative sample threshold is set to 0.8. This cycle repeats, using thresholds of 0.1, 0.2, 0.3, and 0.4 to filter positive samples, and thresholds of 0.9, 0.8, 0.7, and 0.6 to filter negative samples. Before adding to the initial training set, the positive and negative samples to be added will be browsed to see if there are negative samples mixed in the positive samples, or positive samples mixed in the negative samples.

[0135] Furthermore, it is possible to detect false positive videos, add undetected positive and negative samples to the streamlined data set for training, and add detected positive and negative samples to the 1 million data set for later iterations. When adding undetected positive and negative samples to the initial training set, browse the added data and find that positive samples are mixed with negative samples, and negative samples are mixed with positive samples.

[0136] In the prior art, for example, when using the cockpit perception system in a vehicle to detect whether the driver among the vehicle members is smoking, the relevant detection process is as follows: face detection is performed on the image frame sequence of the main driver's seat. If a face is detected, face key point regression is performed in the face ROI area; based on the key points, the ROI area near the mouth is intercepted and input into the smoking image classification model to judge the smoking behavior of the main driver's seat. The smoking image classification model gives a judgment on whether the main driver is smoking.

[0137] Obviously, in the existing detection process, only the ROI area is used to judge smoking, but the global features are not correlated. For example, when smoking in the driver's seat, there will be a behavioral feature of arm movement, that is, the arm is different from the normal driving state, but the ROI infrared image only involves the image of the mouth area, missing the arm feature, which will cause some short cigarette butts to be unable to be accurately judged.

[0138] Based on this, the present application provides an optional implementation method, which can combine the full-image infrared image features to assist the ROI infrared image to determine whether there is smoking behavior in the driver's seat.

[0139] When training the twin network in the present application, the sample images input into the twin network to be trained are the ROI image and the global image. When the ROI image is an image of the mouth area, the corresponding global image can include the driver's arm area. Therefore, based on the twin network training method provided in the present application, the trained twin network for object behavior recognition in the vehicle cabin can be used to determine whether the main driver's behavior is smoking.

[0140] When using the twin network trained in this application to judge whether the driver is smoking, the image frame sequence of the main driver's seat is input into the face detection model. If a face is detected, the face key point regression is performed in the face ROI area; based on the key points, the ROI area near the mouth and the corresponding full image (that is, the first sample image and the second sample image) are simultaneously captured and input into the image deep learning network model (twin network) based on the twin neural network to judge the driving behavior of smoking in the main driver's seat.

[0141] This method combines ROI image and full-image feature sharing to judge driving behavior, and uses similarity measurement to assist model training. In this way, it can more accurately judge whether the driver has dangerous driving behavior (such as smoking). The image deep learning network model based on the twin neural network can output smoking task judgments more accurately, thereby providing support for more DMS perception tasks. Similarly, the above-mentioned judgment method for whether the driver has smoking behavior can also be used to judge whether the driver has made phone calls. Furthermore, the above-mentioned judgment method for whether the driver has smoking behavior can also be used to judge whether a member of the vehicle has smoked, and it can also be used to judge whether a member of the vehicle has made phone calls. That is, it can also provide support for more OMS perception tasks.

[0142] Among them, DMS stands for Driver Monitoring System, which monitors the driver and can monitor the driver's fatigue and dangerous driving behavior. OMS stands for Occupancy Monitoring System, which monitors the passengers and can monitor the passengers' age, status, emotions and behavior.

[0143] Based on this, the present application can also provide a method for identifying object behavior in a vehicle cabin using a twin network trained by the present application, specifically including: obtaining a first image of a target object (such as a driver or a member) in the cabin, and determining a second image based on the first image; the second image is a local image corresponding to a target area in the first image, and the target area is an area related to the behavior to be identified of the target object; the first image and the second image are input into the twin network to obtain a similarity probability between the first image feature and the second image feature; the first image feature is an image feature of the first image, and the second image feature is an image feature of the second image; based on the similarity probability, determining the recognition result of the behavior to be identified for the target object; wherein the twin network is a twin network trained based on the training method of the twin network provided in the present application.

[0144] Among them, based on the similarity probability, the recognition result of the behavior to be recognized for the target object is determined, specifically including: when it is determined that the similarity probability is greater than a preset threshold, the recognition result of the behavior to be recognized for the target object is determined as the occurrence of the behavior to be recognized; when it is determined that the similarity probability is less than or equal to the preset threshold, the recognition result of the behavior to be recognized for the target object is determined as the non-occurrence of the behavior to be recognized.

[0145] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0146] Based on the same inventive concept, the embodiment of the present application also provides a twin network training device for implementing the twin network training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more twin network training devices provided below can be referred to the limitations of the twin network training method above, and will not be repeated here.

[0147] In an exemplary embodiment, Figure 6 As shown, a training device 300 for a twin network is provided, comprising: a sample acquisition module 81, a sample processing module 82 and a parameter adjustment module 83, wherein:

[0148] The sample acquisition module 81 is used to acquire multiple sample pairs; each sample pair includes a first sample image and a second sample image of the sample object; the second sample image is a local image corresponding to a target area in the first sample image, and the target area is an area related to the behavior to be identified of the sample object;

[0149] The sample processing module 82 is used to input the sample pair into the twin network to be trained, obtain the first sample image feature corresponding to the first sample image in the sample pair and the second sample image feature corresponding to the second sample image based on the feature extraction network in the twin network to be trained; and input the first sample image feature and the second sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first predicted behavior and the second predicted behavior based on the first sample image feature and the second sample image feature respectively; determine the first loss value based on the difference similarity between the first sample image feature and the second sample image feature; obtain the second loss value based on the first predicted behavior and the actual behavior corresponding to the sample pair; and obtain the third loss value based on the second predicted behavior and the actual behavior;

[0150] The parameter adjustment module 83 is used to determine the total loss value corresponding to each sample pair based on the first loss value, the second loss value and the third loss value, and when the total loss value of at least one sample pair does not meet the stop training condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the stop training condition is met, so as to obtain a trained twin network. The trained twin network is applied to object behavior recognition in the vehicle cabin.

[0151] In one embodiment, when the sample processing module 82 is used to determine the first loss value based on the difference similarity between the first sample image feature and the second sample image feature, it is specifically used to: obtain the difference similarity between the first sample image feature and the second sample image feature; convert the difference similarity into a sample image feature similarity probability within a preset value range; determine the first loss value based on the sample image feature similarity probability, the actual behavior type value of the sample pair and the preset loss weight.

[0152] In one embodiment, it also includes: a sample processing module 82 for respectively determining the number of first elements and the number of second elements in the first sample image feature and the second sample image feature, whose element values ​​are within a preset value range; and based on the difference between the element values ​​in the first sample image feature and the second sample image feature, determining the number of third elements whose difference values ​​are within a preset difference range; based on the number of first elements, the number of second elements, the number of third elements, and the first sample image feature and the second sample image feature, determining the difference similarity between the first sample image feature and the second sample image feature.

[0153] In one embodiment, the sample processing module 82 is used to input the first sample image feature and the second sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first prediction behavior and the second prediction behavior based on the first sample image feature and the second sample image feature respectively. Specifically, it is used to: input the first sample image feature into the first classification network in the twin network to be trained to obtain the first prediction behavior; input the second sample image feature into the second classification network in the twin network to be trained to obtain the second prediction behavior.

[0154] In one embodiment, the sample processing module 82 is used to input the sample pair into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image in the sample pair and the second sample image feature corresponding to the second sample image based on the feature extraction network in the twin network to be trained. Specifically, it is used to: input the first sample image in the sample pair into the first feature extraction network in the twin network to be trained to obtain the first sample image feature; input the second sample image in the sample pair into the second feature extraction network in the twin network to be trained to obtain the second sample image feature.

[0155] In one embodiment, the first feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network; the second feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network.

[0156] In one embodiment, the multiple sample pairs include positive sample pairs and negative sample pairs; the first sample image and the second sample image included in the positive sample pair are both images with behaviors to be identified; the first sample image and the second sample image included in the negative sample pair are both images without behaviors to be identified.

[0157] In one embodiment, when the sample acquisition module 81 is used to acquire multiple sample pairs, it is specifically used to: acquire multiple sample images including sample objects, and determine a positive sample image to be selected and a negative sample image to be selected from the multiple sample images; acquire a first similarity between the behavior in each positive sample image to be selected and the behavior to be identified, and filter the positive sample image from the multiple positive sample images to be selected based on a comparison result of the first similarity and a first similarity threshold; acquire a second similarity between the behavior in each negative sample image to be selected and the behavior to be identified, and filter the negative sample image from the multiple negative sample images to be selected based on a comparison result of the second similarity and the second similarity threshold; after completing the image screening, respectively acquire a second sample image corresponding to the target area from each positive sample image and each negative sample image that have been screened; construct a positive sample pair based on the positive sample image and the corresponding second sample image, and construct a negative sample pair based on the negative sample image and the corresponding second sample image.

[0158] Each module in the above-mentioned twin network training device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0159] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a training method for a twin network is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0160] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0161] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0162] Acquire multiple sample pairs; each sample pair includes a first sample image and a second sample image of the sample object; the second sample image is a local image corresponding to a target area in the first sample image, and the target area is an area related to the behavior to be identified of the sample object;

[0163] Input the sample pair into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image and the second sample image feature corresponding to the second sample image in the sample pair based on the feature extraction network in the twin network to be trained; and input the first sample image feature and the second sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first prediction behavior and the second prediction behavior respectively based on the first sample image feature and the second sample image feature;

[0164] Determine a first loss value based on a difference similarity between a first sample image feature and a second sample image feature; obtain a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtain a third loss value based on the second predicted behavior and the actual behavior;

[0165] Based on the first loss value, the second loss value and the third loss value, the total loss value corresponding to each sample pair is determined. When the total loss value of at least one sample pair does not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the conditions for stopping training are met, so as to obtain a trained twin network. The trained twin network is applied to object behavior recognition in a vehicle cabin.

[0166] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0167] Acquire multiple sample pairs; each sample pair includes a first sample image and a second sample image of the sample object; the second sample image is a local image corresponding to a target area in the first sample image, and the target area is an area related to the behavior to be identified of the sample object;

[0168] Input the sample pair into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image and the second sample image feature corresponding to the second sample image in the sample pair based on the feature extraction network in the twin network to be trained; and input the first sample image feature and the second sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first prediction behavior and the second prediction behavior respectively based on the first sample image feature and the second sample image feature;

[0169] Determine a first loss value based on a difference similarity between a first sample image feature and a second sample image feature; obtain a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtain a third loss value based on the second predicted behavior and the actual behavior;

[0170] Based on the first loss value, the second loss value and the third loss value, the total loss value corresponding to each sample pair is determined. When the total loss value of at least one sample pair does not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the conditions for stopping training are met, so as to obtain a trained twin network. The trained twin network is applied to object behavior recognition in a vehicle cabin.

[0171] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0172] Acquire multiple sample pairs; each sample pair includes a first sample image and a second sample image of the sample object; the second sample image is a local image corresponding to a target area in the first sample image, and the target area is an area related to the behavior to be identified of the sample object;

[0173] Input the sample pair into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image and the second sample image feature corresponding to the second sample image in the sample pair based on the feature extraction network in the twin network to be trained; and input the first sample image feature and the second sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first prediction behavior and the second prediction behavior respectively based on the first sample image feature and the second sample image feature;

[0174] Determine a first loss value based on a difference similarity between a first sample image feature and a second sample image feature; obtain a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtain a third loss value based on the second predicted behavior and the actual behavior;

[0175] Based on the first loss value, the second loss value and the third loss value, the total loss value corresponding to each sample pair is determined. When the total loss value of at least one sample pair does not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the conditions for stopping training are met, so as to obtain a trained twin network. The trained twin network is applied to object behavior recognition in a vehicle cabin.

[0176] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0177] It should also be added that the images including user objects involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant images must comply with relevant regulations.

[0178] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0179] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0180] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a twin network, characterized in that: The method comprises: Acquire multiple sample pairs; each of the sample pairs includes a first sample image and a second sample image of a sample object; the second sample image is a local image corresponding to a target area in the first sample image, and the target area is an area related to the behavior to be identified of the sample object; Input the sample pair into the twin network to be trained, and obtain a first sample image feature corresponding to the first sample image and a second sample image feature corresponding to the second sample image in the sample pair based on a feature extraction network in the twin network to be trained; and input the first sample image feature and the second sample image feature into a classification network in the twin network to be trained, and the classification network obtains a first predicted behavior and a second predicted behavior based on the first sample image feature and the second sample image feature, respectively; Determine a first loss value based on a difference similarity between the first sample image feature and the second sample image feature; obtain a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtain a third loss value based on the second predicted behavior and the actual behavior; Based on the first loss value, the second loss value and the third loss value, the total loss value corresponding to each of the sample pairs is determined, and when the total loss value of at least one of the sample pairs does not meet the stop training condition, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the stop training condition is met, so as to obtain a trained twin network, and the trained twin network is applied to object behavior recognition in a vehicle cabin.

2. The method according to claim 1, characterized in that The determining a first loss value based on the difference similarity between the first sample image feature and the second sample image feature comprises: Obtaining a difference similarity between the first sample image feature and the second sample image feature; Converting the difference similarity into a sample image feature similarity probability within a preset value interval; The first loss value is determined based on the sample image feature similarity probability, the actual behavior type value of the sample pair and the preset loss weight.

3. The method according to claim 1, characterized in that Before determining the first loss value based on the difference similarity between the first sample image feature and the second sample image feature, the method further includes: Determine respectively the number of first elements and the number of second elements in the first sample image feature and the second sample image feature, whose element values ​​are within a preset value range; and determine, based on the difference between the element values ​​in the first sample image feature and the second sample image feature, the number of third elements whose difference is within a preset difference range; Based on the first number of elements, the second number of elements, the third number of elements, and the first sample image feature and the second sample image feature, a difference similarity between the first sample image feature and the second sample image feature is determined.

4. The method according to claim 1, characterized in that: The step of inputting the first sample image feature and the second sample image feature into a classification network in the twin network to be trained, wherein the classification network obtains a first prediction behavior and a second prediction behavior based on the first sample image feature and the second sample image feature, respectively, including: Inputting the first sample image feature into the first classification network in the twin network to be trained to obtain the first predicted behavior; The second sample image feature is input into the second classification network in the twin network to be trained to obtain the second predicted behavior.

5. The method according to claim 1, characterized in that The step of inputting the sample pair into the twin network to be trained, and obtaining a first sample image feature corresponding to the first sample image and a second sample image feature corresponding to the second sample image in the sample pair based on a feature extraction network in the twin network to be trained, comprises: Inputting the first sample image in the sample pair into the first feature extraction network in the twin network to be trained to obtain a first sample image feature; The second sample image in the sample pair is input into the second feature extraction network in the twin network to be trained to obtain the second sample image feature.

6. The method according to claim 5, characterized in that The first feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network; The second feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network.

7. The method according to claim 1, characterized in that The plurality of sample pairs include positive sample pairs and negative sample pairs; The first sample image and the second sample image included in the positive sample pair are both images containing behaviors to be identified; The first sample image and the second sample image included in the negative sample pair are both images in which the to-be-recognized behavior does not exist.

8. The method according to claim 7, characterized in that The obtaining of multiple sample pairs comprises: Acquire a plurality of sample images including sample objects, and determine a positive sample image to be selected and a negative sample image to be selected from the plurality of sample images; Obtaining a first similarity between the behavior in each of the to-be-selected positive sample images and the to-be-recognized behavior, and screening a positive sample image from the plurality of to-be-selected positive sample images according to a comparison result of the first similarity and a first similarity threshold; Obtaining a second similarity between the behavior in each of the negative sample images to be selected and the behavior to be identified, and screening a negative sample image from the plurality of negative sample images to be selected according to a comparison result between the second similarity and a second similarity threshold; After completing the image screening, the second sample image corresponding to the target area is obtained from each of the screened positive sample images and each of the negative sample images; the positive sample pair is formed based on the positive sample image and the corresponding second sample image, and the negative sample pair is formed based on the negative sample image and the corresponding second sample image.

9. A training device for a twin network, characterized in that: The device comprises: A sample acquisition module, used to acquire a plurality of sample pairs; each of the sample pairs comprises a first sample image and a second sample image of a sample object; the second sample image is a local image corresponding to a target area in the first sample image, and the target area is an area related to the behavior to be identified of the sample object; A sample processing module, used for inputting the sample pair into the twin network to be trained, obtaining a first sample image feature corresponding to the first sample image in the sample pair and a second sample image feature corresponding to the second sample image based on the feature extraction network in the twin network to be trained; and inputting the first sample image feature and the second sample image feature into a classification network in the twin network to be trained, wherein the classification network obtains a first predicted behavior and a second predicted behavior based on the first sample image feature and the second sample image feature, respectively; determining a first loss value based on a difference similarity between the first sample image feature and the second sample image feature; obtaining a second loss value based on the first predicted behavior and an actual behavior corresponding to the sample pair; and obtaining a third loss value based on the second predicted behavior and the actual behavior; A parameter adjustment module is used to determine the total loss value corresponding to each of the sample pairs based on the first loss value, the second loss value and the third loss value, and when the total loss value of at least one of the sample pairs does not meet the stop training condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the stop training condition is met, so as to obtain a trained twin network, and the trained twin network is applied to object behavior recognition in a vehicle cabin.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images