Twin network training method and device and computer equipment

Abstract

The invention relates to a twin network training method and device and computer equipment, and the method comprises the steps: obtaining a plurality of sample groups, inputting the sample groups into a to-be-trained twin network, and obtaining first to third sample image features of first to third sample images based on a feature extraction network in the network; obtaining first to third prediction behaviors of the first to third sample image features through a classification network; determining a first loss value based on difference similarity between the second sample image feature and the first sample image feature and between the third sample image feature and the first sample image feature; obtaining second to fourth loss values based on the first to third prediction behaviors and the actual behavior; the total loss value of the sample group is determined based on the four loss values, when at least one total loss value does not meet the training stopping condition, network parameters are updated, training is conducted again till the stopping condition is met, and a trained twin network is obtained and can be applied to object behavior recognition in the vehicle cabin.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a training method, apparatus and computer device for a twin network. Background Art

[0002] With the increase in demand for vehicle use, traffic accidents occur from time to time. Many traffic accidents are related to the driver's bad driving behavior, and some traffic accidents are caused by the bad riding behavior of the passengers in the car. In the existing vehicle use environment, drivers often do not pay attention to their driving behavior during driving, and the passengers in the car do not pay attention to their riding behavior in the car. For example, there are common bad behaviors such as drivers and passengers smoking and making phone calls in the car. Among them, similar behaviors such as drivers making phone calls while driving can easily distract the driver's driving concentration, and the smoking behavior of passengers while driving can easily cause quarrels between passengers and drivers, which will also distract the driver's driving concentration.

[0003] Although the use of vehicles facilitates people's travel, the safety of drivers and passengers is of paramount importance. In order to improve the safety of drivers and passengers in the car, real-time monitoring and intelligent evaluation of the driver's driving behavior and the passengers' riding behavior will help to detect irregular or bad behaviors of the car members as early as possible, and then remind the car members to change their relevant behavioral habits, so as to effectively reduce the problem of the driver's attention being distracted during driving, thereby helping to reduce the occurrence of traffic accidents and protect the lives of drivers and passengers.

[0004] However, in the relevant existing technologies, the accuracy of intelligent detection of bad behaviors of vehicle occupants is not ideal. Therefore, there is an urgent need to provide a behavior recognition model that can be used to accurately identify the behaviors of vehicle occupants, so as to accurately identify the bad behaviors of vehicle occupants in a shorter time. Summary of the invention

[0005] Based on this, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium and computer program product for a twin network that can be trained to perform intelligent monitoring of the behavior of members inside a vehicle with high accuracy in order to address the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a training method for a twin network, the method comprising:

[0007] Acquire multiple sample groups; each of the sample groups includes a first sample image, a second sample image, and a third sample image of a sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified;

[0008] Input the sample group into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image in the sample group based on the feature extraction network in the twin network to be trained; and input the first sample image feature, the second sample image feature, and the third sample image feature into the classification network in the twin network to be trained, respectively, to obtain the first predicted behavior, the second predicted behavior, and the third predicted behavior, respectively;

[0009] Determine a first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature respectively; obtain a second loss value based on the first predicted behavior and the actual behavior corresponding to the sample group, obtain a third loss value based on the second predicted behavior and the actual behavior, and obtain a fourth loss value based on the third predicted behavior and the actual behavior;

[0010] Based on the first loss value, the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is determined, and when the total loss value does not meet the condition for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the condition for stopping training is met, so as to obtain a trained twin network, which is applied to at least two behavior recognitions of objects in a vehicle cabin.

[0011] In one embodiment, determining the first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature respectively includes:

[0012] Acquire a first difference similarity between the first sample image feature and the second sample image feature, and a second difference similarity between the first sample image feature and the third sample image feature;

[0013] Determining a total sample image feature similarity based on the sum of the first difference similarity and the second difference similarity;

[0014] The first loss value is determined based on the total sample image feature similarity, the actual behavior type value of the sample group and a preset loss weight.

[0015] In one embodiment, before determining the first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature, the method further includes:

[0016] respectively determining the number of first elements, the number of second elements, and the number of third elements whose element values ​​are within a preset value range in the first sample image feature, the second sample image feature, and the third sample image feature; and determining the number of fourth elements whose first difference value is within a corresponding preset difference range and the number of fifth elements whose second difference value is within a corresponding preset difference range based on the first difference value and the second difference value between the element values ​​in the first sample image feature and the second sample image feature and the third sample image feature, respectively;

[0017] Determine a first difference similarity between the first sample image feature and the second sample image feature based on the first number of elements, the second number of elements, the fourth number of elements, and the first sample image feature and the second sample image feature;

[0018] Based on the first number of elements, the third number of elements, the fifth number of elements, and the first sample image feature and the third sample image feature, a second difference similarity between the first sample image feature and the third sample image feature is determined.

[0019] In one embodiment, the first sample image feature, the second sample image feature, and the third sample image feature are respectively input into the classification network in the twin network to be trained to obtain a first prediction behavior, a second prediction behavior, and a third prediction behavior, respectively, including:

[0020] Inputting the first sample image feature into the first classification network in the twin network to be trained to obtain the first predicted behavior;

[0021] Inputting the second sample image feature into the second classification network in the twin network to be trained to obtain the second predicted behavior;

[0022] The third sample image feature is input into the third classification network in the twin network to be trained to obtain the third predicted behavior.

[0023] In one embodiment, the step of inputting the sample group into the twin network to be trained, and obtaining a first sample image feature corresponding to the first sample image in the sample group, a second sample image feature corresponding to the second sample image, and a third sample image feature corresponding to the third sample image based on a feature extraction network in the twin network to be trained, includes:

[0024] Inputting the first sample image in the sample group into the first feature extraction network in the twin network to be trained to obtain corresponding first sample image features;

[0025] Inputting the second sample image in the sample group into the second feature extraction network in the twin network to be trained to obtain corresponding second sample image features;

[0026] The third sample image in the sample group is input into the third feature extraction network in the twin network to be trained to obtain the corresponding third sample image feature.

[0027] In one of the embodiments, the first feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network;

[0028] The second feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network;

[0029] The third feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network.

[0030] In one embodiment, the sample groups include a positive sample group and a negative sample group;

[0031] The three sample images included in the positive sample group are all images containing behaviors to be identified;

[0032] The three sample images included in the negative sample group are all images without the behavior to be identified.

[0033] In one embodiment, obtaining multiple sample groups includes:

[0034] Acquire a plurality of sample images including sample objects, and determine a positive sample image to be selected and a negative sample image to be selected from the plurality of sample images;

[0035] For the same behavior to be identified, obtaining a first similarity between the behavior in each of the to-be-selected positive sample images and the to-be-identified behavior, and screening a positive sample image from the plurality of to-be-selected positive sample images according to a comparison result of the first similarity and the first similarity threshold;

[0036] Obtaining a second similarity between the behavior in each of the negative sample images to be selected and the behavior to be identified for the same behavior, and screening a negative sample image from the plurality of negative sample images to be selected according to a comparison result of the second similarity and the second similarity threshold;

[0037] After completing the image screening, the second sample image corresponding to the first target area and the third sample image corresponding to the second target area are respectively obtained from the screened positive sample images and the screened negative sample images; a positive sample group is formed based on the positive sample image and the corresponding second sample image and the third sample image, and a negative sample group is formed based on the negative sample image and the corresponding second sample image and the third sample image.

[0038] In a second aspect, the present application also provides a training device for a twin network, comprising:

[0039] A sample acquisition module, used to acquire multiple sample groups; each of the sample groups includes a first sample image, a second sample image, and a third sample image of a sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified;

[0040] A sample processing module, used for inputting the sample group into the twin network to be trained, obtaining the first sample image feature corresponding to the first sample image in the sample group, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image based on the feature extraction network in the twin network to be trained; and inputting the first sample image feature, the second sample image feature and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first predicted behavior, the second predicted behavior and the third predicted behavior respectively based on the first sample image feature, the second sample image feature and the third sample image feature; determining the first loss value based on the difference similarity between the second sample image feature and the third sample image feature and the first sample image feature respectively; obtaining the second loss value based on the first predicted behavior and the actual behavior corresponding to the sample group, obtaining the third loss value based on the second predicted behavior and the actual behavior, and obtaining the fourth loss value based on the third predicted behavior and the actual behavior;

[0041] A parameter adjustment module is used to determine the total loss value corresponding to each of the sample groups based on the first loss value, the second loss value, the third loss value and the fourth loss value, and when the total loss value does not meet the training stop condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of objects in a vehicle cabin.

[0042] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any step in the above-mentioned first aspect when executing the computer program.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any step in the above-mentioned first aspect when executed by a processor.

[0044] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements any step in the above-mentioned first aspect when executed by a processor.

[0045] The present application provides a training method, apparatus, computer equipment, computer-readable storage medium and computer program product for a twin network. The training method for the twin network mentioned above obtains multiple sample groups; each sample group includes a first sample image, a second sample image and a third sample image of a sample object; the second sample image is a local image of a first target area in the first sample image, and the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified; the sample group is input into the twin network to be trained, and based on the feature extraction network in the twin network to be trained, a first sample image feature corresponding to the first sample image, a second sample image feature corresponding to the second sample image and a third sample image feature corresponding to the third sample image in the sample group are obtained; and the first sample image feature, the second sample image feature and the third sample image feature are input into the twin network to be trained. A classification network is provided, wherein the classification network obtains a first predicted behavior, a second predicted behavior, and a third predicted behavior based on the first sample image feature, the second sample image feature, and the third sample image feature, respectively; a first loss value is determined based on the difference similarity between the second sample image feature, the third sample image feature, and the first sample image feature, respectively; a second loss value is obtained based on the first predicted behavior and the actual behavior corresponding to the sample group, a third loss value is obtained based on the second predicted behavior and the actual behavior, and a fourth loss value is obtained based on the third predicted behavior and the actual behavior; a total loss value corresponding to each sample group is determined based on the first loss value, the second loss value, the third loss value, and the fourth loss value, and when the total loss value does not meet the training stop condition, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of objects in a vehicle cabin.

[0046] It can be seen that the present application forms a sample group through a first sample image, a local image of the first target area related to the first behavior to be identified in the first sample image (second sample image), and a local image of the second target area related to the second behavior to be identified in the first sample image (third sample image), as training samples for the twin network to be trained, so that the sample image simultaneously includes an image of the first target area and an image of the second target area of ​​the sample object, and also includes a global image of the sample object, thereby increasing the types of sample images; wherein other feature information associated with the target area (the first target area and the second target area) in the global image is conducive to auxiliary identification of the specific behavior of the sample object in the target area, and is conducive to improving the recognition accuracy of the subsequently trained twin network for the behavior to be identified of the object in the image; and in the twin In the process of training the twin network, the difference similarity calculation formula is used to calculate the difference similarity between the second sample image feature and the third sample image feature of the sample image and the first sample image feature, so as to determine the first loss value. The difference similarity is used to determine the first loss value, which is conducive to improving the accuracy of the obtained first loss value, and then to improving the recognition accuracy of the trained twin network for the to-be-recognized behavior of the object in the image; at the same time, the second loss value, the third loss value and the fourth loss value corresponding to the first sample image feature, the second sample image feature and the third sample image feature are obtained respectively; then, based on the first loss value and the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is obtained, so as to achieve good training of the twin network to be trained based on the total loss value. In summary, the twin network trained based on the method provided in this application can efficiently and accurately identify multiple types of specific behaviors of the occupants in the car based on pictures of the occupants in the car. For example, it can identify one or two types of bad behaviors of the occupants in the car. This helps to detect the irregular or bad behaviors of the occupants in the car as early as possible, and then remind the occupants in the car to change their relevant behavioral habits, so as to effectively reduce the problem of the driver's attention being distracted during driving, thereby helping to reduce the occurrence of traffic accidents and protect the lives of drivers and passengers. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A diagram of an application environment for training a twin network in one embodiment;

[0049] Figure 2 This is an application environment diagram of a trained twin network in one embodiment;

[0050] Figure 3 It is a flowchart of a training method of a twin network in one embodiment;

[0051] Figure 4 A schematic diagram of the structure of a twin network in one embodiment;

[0052] Figure 5 A schematic diagram of the structure of a twin network in another embodiment;

[0053] Figure 6 It is a structural block diagram of a training device of a twin network in one embodiment;

[0054] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The training method of the twin network provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 902 communicates with the server 904 through the network. The terminal 902 can at least be used to obtain sample images for training the twin network to be trained, and perform certain preprocessing operations on the sample images to obtain the first sample image corresponding to the sample image; in addition, the classification label work for the sample image can also be realized based on the terminal 902. The data storage system can store the data that the server 904 needs to process. The data storage system can be integrated on the server 904, or it can be placed on the cloud or other network servers. The data storage system can be used to store at least a plurality of sample groups including a first sample image, a second sample image, and a third sample image, as well as a twin network to be trained and a twin network that has been trained. Among them, the terminal 902 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server 904 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0057] It should be noted that the training process of the twin network can be based on the server 904 to provide operations, or it can be based on the terminal 902 to provide related operations, or it can be based on the communication between the terminal 902 and the server 904 to implement related operations. This application does not make specific limitations on this.

[0058] Furthermore, the trained twin network provided in the embodiment of the present application can be applied to Figure 2 The application environment shown. Figure 2 This is only one application environment that can be selected for the trained twin network provided by this application, and this application is not limited to this; that is, the trained twin network can not only be used for Figure 2 The driver / passenger behavior recognition in the vehicle environment shown can also be used for example for worker behavior recognition in a factory environment, and can also be used for example for student behavior recognition in a teaching environment, etc. Specifically, the application scenario includes a vehicle 100, the vehicle 100 includes a vehicle controller 101, and the vehicle cabin includes an image acquisition device 102.

[0059] The vehicle controller 101 may first capture an image of a target object (eg, at least one of a driver and a passenger) in the cabin through the image capture device 102, and then process the captured image to obtain a first image of the target object, and store the first image. In the process of object behavior recognition of images based on the twin network trained in the present application, the vehicle controller 101 can first obtain a first image of the target object, and determine the second image and the third image based on the first image, that is, obtain the first local image (second image) and the second local image (third image) of the target object from the global image (first image) of the target object; wherein the first local image is an image corresponding to the area in the first image related to the first behavior to be recognized of the target object, and the second local image is an image corresponding to the area in the first image related to the second behavior to be recognized of the target object; further, the vehicle controller 101 inputs the first image, the second image and the third image belonging to the same sample group into the twin network trained in the present application together, so as to obtain the similarity probabilities between the first image feature of the first image and the second image feature of the second image and the third image feature of the third image respectively based on the twin network trained in the present application; finally, based on the image feature similarity probability, determine the recognition result of the behavior to be recognized (the first behavior to be recognized and the second behavior to be recognized) of the target object to determine whether the specific behavior of the target object includes the behavior to be recognized.

[0060] Among them, for the recognition result of the first image by the twin network obtained by training based on this application, this application provides an optional implementation method, for example, in the case where the first image includes the driver's smoking or calling behavior, the final output result is that the specific behavior of the target object includes "smoking or calling behavior", as for whether it is smoking behavior or calling behavior, no specific distinction can be made. When the driver receives the reminder of bad driving behavior, the driver himself can clearly know which bad behavior he has, that is, the driver can correct the bad behavior that is occurring based on his actual situation. In addition, this application also provides an optional implementation method, for example, in the case where the first image includes the driver's smoking or calling behavior, the final output result is that the specific behavior of the target object "exists smoking behavior", or "exists calling behavior", or "exists smoking and calling behavior at the same time", so that when the driver receives the reminder of bad driving behavior, he can clearly know which bad behavior he has, which is conducive to improving the efficiency of the driver in correcting bad driving behavior. That is, if necessary, further analysis and processing of the recognition results can be performed to confirm whether the image shows smoking or making a phone call. It should be noted that for a driver, it is possible to make a phone call and smoke at the same time, that is, a driver may have two or more bad driving behaviors at the same time.

[0061] In order to provide a behavior recognition model that can be used to accurately identify the behavior of members inside the vehicle, so as to accurately identify the bad behavior of the occupants in the vehicle in a shorter time, the present application provides a twin network training method to train a twin network that can recognize the behavior of members inside the vehicle with high accuracy.

[0062] In an exemplary embodiment, Figure 3 As shown, a training method for a twin network is provided, including but not limited to the following steps S201 to S204. Among them:

[0063] S201, obtaining multiple sample groups; each sample group includes a first sample image, a second sample image, and a third sample image of the sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified.

[0064] The sample object may be a driver inside the vehicle or a passenger inside the vehicle. The first sample image may be a first sample image including at least one of the driver and the passenger; assuming that the first behavior to be identified is whether the driver smokes, the first target area is an area related to the behavior of smoking, such as the driver's mouth area. In this case, the second sample image is a local image corresponding to the mouth area in the first sample image of the driver. Assuming that the second behavior to be identified is whether the driver makes a phone call, the second target area is an area related to the behavior of making a phone call, such as the driver's eye area. In this case, the third sample image is a local image corresponding to the eye area in the first sample image of the driver. The local image corresponding to the target area is also a face ROI image. ROI (Region of Interest) refers to a specific area to be processed in an image, and the area to be processed is usually outlined by a box, a circle, an ellipse, an irregular polygon, etc.; that is, the second sample image and the third sample image may be, for example, images of two different areas framed from the first sample image by a box shape.

[0065] In an optional implementation, before step S201, an image including the sample object acquired by an image acquisition device in the vehicle cabin may be acquired, and the image may be preprocessed to obtain the image including the sample object as the first sample image. The image preprocessing may include denoising, cropping, etc., which is not specifically limited in this application.

[0066] The image acquisition device may be, for example, an infrared sensor or a visible light sensor. The infrared sensor may be, for example, an infrared camera for acquiring infrared images. The visible light sensor may be, for example, an ordinary camera for acquiring color images, which may also be called RGB images, where R is a red channel, G is a green channel, and B is a blue channel. The image of the sample object may be a single-channel infrared image of the vehicle cabin including the sample object acquired by the infrared sensor, or an RGB color image of the vehicle cabin including the sample object acquired by the visible light sensor. The images in the infrared image sample group may be images formed based on obtaining the radiation of the sample object in the infrared band.

[0067] The present application provides an optional implementation manner in which a vehicle cabin includes both an infrared sensor and a visible light sensor, wherein the infrared sensor is used to monitor and photograph the driver's seat in the vehicle, and the visible light sensor is used to monitor and photograph the passenger seat in the vehicle; however, this is only an optional implementation manner provided by the present application, and the present application is not limited thereto.

[0068] A further optional implementation is provided, where the acquired image of the sample object is a single-channel infrared image, it can be further converted into a three-channel infrared image to serve as the first sample image of the sample object.

[0069] Each sample group includes three images, specifically, a first sample image of a sample object, a second sample image cropped based on an area in the first sample image related to a first behavior to be identified of the sample object, and a third sample image cropped based on an area in the first sample image related to a second behavior to be identified of the sample object.

[0070] S202, input the sample group into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image in the sample group, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image based on the feature extraction network in the twin network to be trained; and input the first sample image feature, the second sample image feature, and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first predicted behavior, the second predicted behavior, and the third predicted behavior based on the first sample image feature, the second sample image feature, and the third sample image feature, respectively.

[0071] Among them, the twin network (twin neural network) to be trained provided in the present application may specifically include a feature extraction network and a classification network, that is, the twin network is at least composed of a feature extraction network and a classification network, wherein the input data of the feature extraction network is a sample group, and the input data of the classification network is the data output by the feature extraction network after processing the sample group.

[0072] In an optional embodiment, the first sample image, the second sample image, and the third sample image included in the sample group are respectively input into the feature extraction network in the twin network to be trained provided in the present application to obtain the first sample image feature based on the feature extraction network after processing the first sample image, and the second sample image feature based on the feature extraction network after processing the second sample image, and the third sample image feature based on the feature extraction network after processing the third sample image; then, the first sample image feature, the second sample image feature, and the third sample image feature are respectively input into the classification network in the twin network to be trained provided in the present application to obtain the first prediction behavior based on the classification network after processing the first sample image feature, and the second prediction behavior based on the classification network after processing the second sample image feature, and the third prediction behavior based on the classification network after processing the third sample image feature.

[0073] In an exemplary embodiment, the first sample image feature may be an image feature vector, and the second sample image feature and the third sample image feature may also be image feature vectors; for example, features in the first sample image (e.g., the full image) are extracted through a feature extraction network to obtain a first feature vector (first sample image feature), features in the second sample image (e.g., the first ROI image) are extracted through a feature extraction network to obtain a second feature vector (second sample image feature), and features in the third sample image (e.g., the second ROI image) are extracted through a feature extraction network to obtain a third feature vector (third sample image feature).

[0074] In an exemplary embodiment, the classification network is used to receive the first sample image feature corresponding to the first sample image to output the prediction result (first predicted behavior) of the specific behavior of the sample object in the first sample image; the classification network is also used to receive the second sample image feature corresponding to the second sample image to output the prediction result (second predicted behavior) of the specific behavior of the sample object in the second sample image; the classification network is also used to receive the third sample image feature corresponding to the third sample image to output the prediction result (third predicted behavior) of the specific behavior of the sample object in the third sample image. Wherein, in the case where the first sample image includes, for example, smoking behavior or phone call behavior, the corresponding first predicted behavior may be the existence of smoking behavior or phone call behavior; the second predicted behavior may be the existence of smoking behavior, and the third predicted behavior may be the existence of phone call behavior.

[0075] It should also be added that the classification network is based on predictions of pre-set behavior types that need to be identified. For example, when the pre-set behavior types that need to be identified include smoking behavior and phone call behavior, the classification network will output the output result that the first predicted behavior is smoking behavior or phone call behavior.

[0076] S203, determine the first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature respectively; obtain the second loss value based on the first predicted behavior and the actual behavior corresponding to the sample group, obtain the third loss value based on the second predicted behavior and the actual behavior, and obtain the fourth loss value based on the third predicted behavior and the actual behavior.

[0077] The first predicted behavior obtained by executing step S202 of the present application is a prediction result of the behavior to be identified of the sample object in the first sample image based on the feature extraction network and the classification network in the twin network to be trained provided by the present application; the user can also determine the actual behavior of the sample object included in the first sample image through expert identification (artificial identification), and then, based on the difference between the first predicted behavior and the actual behavior corresponding to the sample group for the same first sample image, for example, by processing the numerical value corresponding to the behavior type of the first predicted behavior and the numerical value corresponding to the behavior type of the actual behavior, a second loss value for the first predicted behavior and the actual behavior can be obtained. The second predicted behavior obtained by executing step S202 of the present application is a prediction result of the to-be-identified behavior of the sample object in the second sample image based on the feature extraction network and the classification network in the twin network to be trained provided by the present application; the user can also determine the actual behavior of the sample object in the second sample image based on the actual behavior of the sample object included in the first sample image corresponding to the determined second sample image, and then, based on the difference between the second predicted behavior for the same second sample image and the actual behavior corresponding to the sample group, for example, by processing the numerical value corresponding to the behavior type of the second predicted behavior and the numerical value corresponding to the behavior type of the actual behavior, a third loss value for the second predicted behavior and the actual behavior can be obtained. The third predicted behavior obtained by executing step S202 of the present application is a prediction result of the to-be-identified behavior of the sample object in the third sample image based on the feature extraction network and the classification network in the twin network to be trained provided by the present application; the user can also determine the actual behavior of the sample object in the third sample image based on the actual behavior of the sample object included in the first sample image corresponding to the determined third sample image, and then, based on the difference between the third predicted behavior for the same third sample image and the actual behavior corresponding to the sample group, for example, by processing the numerical value corresponding to the behavior type of the third predicted behavior and the numerical value corresponding to the behavior type of the actual behavior, a fourth loss value for the third predicted behavior and the actual behavior can be obtained.

[0078] While calculating and obtaining the second loss value, the third loss value and the fourth loss value, the first loss values ​​corresponding to the first sample image feature, the second sample image feature and the third sample image feature can be obtained based on the difference similarity between the first sample image feature and the second sample image feature, and the difference similarity between the first sample image feature and the third sample image feature. The calculation of the loss value can be implemented based on the loss function.

[0079] S204, based on the first loss value, the second loss value, the third loss value and the fourth loss value, determine the total loss value corresponding to each sample group, and when the total loss value of at least one sample group does not meet the conditions for stopping training, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the conditions for stopping training are met, to obtain a trained twin network, and the trained twin network is applied to object behavior recognition in the vehicle cabin.

[0080] Among them, after obtaining the first loss value, the second loss value, the third loss value and the fourth loss value of the sample group based on the execution of step S203, the total loss value of the sample group can be further determined based on the four loss values, and then it can be judged whether the total loss value of the sample group meets the stop training condition of the twin network to be trained. When the total loss value of the sample does not meet the stop training condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample group used in a training process, and the twin network is trained again until the total loss values ​​of multiple sample groups meet the stop training condition. The training process for the twin network can be ended, and the trained twin network can be obtained at this time. Since the data input to the twin network to be trained in this application is a cabin image including sample objects, and at least part of the sample objects included in the cabin image have behaviors to be identified, the twin network trained in this application can be used for (pre-set) behavior recognition of objects in the vehicle cabin; and then, when the object behavior in the vehicle cabin is subsequently identified based on the trained twin network, the accuracy of object behavior recognition in the vehicle cabin can be improved.

[0081] It should be noted that the sample group provided above is an embodiment of a cabin image including a sample object, which is only an optional implementation provided by the present application, and the present application is not limited thereto; the specific image type including the sample object used to train the twin network can be selected based on the different requirements of the application scenarios of the trained twin network. For example, the selectable sample group is a classroom image including a sample object (a student), in which case the corresponding preset behaviors can be sleeping behavior, face covering behavior, eating behavior, and so on.

[0082] The training method of the above-mentioned twin network obtains multiple sample groups; each sample group includes a first sample image, a second sample image and a third sample image of the sample object; the second sample image is a local image of the first target area in the first sample image, and the third sample image is a local image of the second target area in the first sample image, the first target area is an area related to the first behavior to be identified, and the second target area is an area related to the second behavior to be identified; the sample group is input into the twin network to be trained, and based on the feature extraction network in the twin network to be trained, the first sample image features corresponding to the first sample image, the second sample image features corresponding to the second sample image, and the third sample image features corresponding to the third sample image in the sample group are obtained; and the first sample image features, the second sample image features and the third sample image features are input into the classification network in the twin network to be trained, and the classification network is based on the first sample image features, the second sample image features and the third sample image features. The first predicted behavior, the second predicted behavior and the third predicted behavior are obtained based on the second sample image features and the third sample image features, respectively; the first loss value is determined based on the difference similarity between the second sample image features, the third sample image features and the first sample image features, respectively; the second loss value is obtained based on the first predicted behavior and the actual behavior corresponding to the sample group, the third loss value is obtained based on the second predicted behavior and the actual behavior, and the fourth loss value is obtained based on the third predicted behavior and the actual behavior; based on the first loss value, the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is determined, and when the total loss value does not meet the training stop condition, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of objects in the vehicle cabin.

[0083] It can be seen that the present application forms a sample group through a first sample image, a local image of the first target area related to the first behavior to be identified in the first sample image (second sample image), and a local image of the second target area related to the second behavior to be identified in the first sample image (third sample image), as training samples for the twin network to be trained, so that the sample image simultaneously includes an image of the first target area and an image of the second target area of ​​the sample object, and also includes a global image of the sample object, thereby increasing the types of sample images; wherein other feature information associated with the target area (the first target area and the second target area) in the global image is conducive to auxiliary identification of the specific behavior of the sample object in the target area, and is conducive to improving the recognition accuracy of the subsequently trained twin network for the behavior to be identified of the object in the image; and in the twin In the process of training the twin network, the difference similarity calculation formula is used to calculate the difference similarity between the second sample image feature and the third sample image feature of the sample image and the first sample image feature, so as to determine the first loss value. The difference similarity is used to determine the first loss value, which is conducive to improving the accuracy of the obtained first loss value, and then to improving the recognition accuracy of the trained twin network for the to-be-recognized behavior of the object in the image; at the same time, the second loss value, the third loss value and the fourth loss value corresponding to the first sample image feature, the second sample image feature and the third sample image feature are obtained respectively; then, based on the first loss value and the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is obtained, so as to achieve good training of the twin network to be trained based on the total loss value. In summary, the twin network trained based on the method provided in this application can efficiently and accurately identify multiple types of specific behaviors of occupants in the car based on pictures of the occupants in the car. For example, it can identify one or two types of bad behaviors of the occupants in the car. This helps to detect the irregular or bad behaviors of the occupants in the car as early as possible, and then remind the occupants in the car to change their relevant behavioral habits, so as to effectively reduce the problem of the driver's attention being distracted during driving, thereby helping to reduce the occurrence of traffic accidents and protect the lives of drivers and passengers.

[0084] In an exemplary embodiment, the step S203 of determining the first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature can be specifically performed as follows, including steps S231 to S234 (not shown), wherein:

[0085] S231, obtaining a first difference similarity between the first sample image feature and the second sample image feature, and determining the first sample image feature similarity and a second difference similarity between the first sample image feature and the third sample image feature;

[0086] S232, determining a total sample image feature similarity based on the sum of the first difference similarity and the second difference similarity;

[0087] S233, converting the total sample image feature similarity into a sample image feature similarity probability within a preset value range;

[0088] S234, determining a first loss value based on the sample image feature similarity probability, the actual behavior type value of the sample group and the preset loss weight.

[0089] Specifically, first, based on the calculation formula of difference similarity, the first difference similarity between the first sample image feature and the second sample image feature is calculated, and the second difference similarity between the first sample image feature and the third sample image feature is calculated; then, the corresponding first difference similarities and first difference similarities in the same sample group are added to obtain the total sample image feature similarity corresponding to the sample group.

[0090] Afterwards, the total sample image feature similarity is converted into a sample image feature similarity probability within a preset value range, for example, the sample image feature similarity is converted into a sample image feature relative probability within the interval [0,1]. The sample image feature relative probability is used to determine whether the behaviors of the three sample images in the sample group are the same; for example, when the sample image feature relative probability is greater than the preset smoking or phone call probability, then it can be obtained that the sample objects in the three images of the sample group all have the behavior of smoking or making phone calls.

[0091] Finally, based on the sample image feature similarity probability obtained by executing step S233, the actual behavior type value of the sample object in the determined sample group image, and the preset loss weight, a first loss value associated with a sample group can be determined. The actual behavior type value is, for example, when the actual behavior is the same as the preset behavior to be detected, the actual behavior type value can be set to 1, and when the actual behavior is different from the preset behavior to be detected, the actual behavior type value can be set to 0.

[0092] In an exemplary embodiment, before the above-mentioned determination of the first loss value based on the difference similarity between the second sample image feature and the third sample image feature and the first sample image feature, an execution method of determining the second difference similarity between the first sample image feature and the third sample image feature may also be included, wherein:

[0093] Determine the number of first elements, the number of second elements, and the number of third elements whose element values ​​are within a preset value range in the first sample image feature, the second sample image feature, and the third sample image feature, respectively; and determine the number of fourth elements whose first difference values ​​are within a corresponding preset difference range and the number of fifth elements whose second difference values ​​are within a corresponding preset difference range based on the first difference and the second difference between the element values ​​in the first sample image feature and the second sample image feature and the third sample image feature, respectively;

[0094] Determine a first difference similarity between the first sample image feature and the second sample image feature based on the first element number, the second element number, the fourth element number, and the first sample image feature and the second sample image feature;

[0095] Based on the first number of elements, the third number of elements, the fifth number of elements, and the first sample image feature and the third sample image feature, a second difference similarity between the first sample image feature and the third sample image feature is determined.

[0096] Among them, the first number of elements is the element value of the element in the first sample image feature, and the number of elements within the preset value range; the second number of elements is the element value of the element in the second sample image feature, and the number of elements within the preset value range; the third number of elements is the element value of the element in the third sample image feature, and the number of elements within the preset value range.

[0097] Regarding the determination of the fourth number of elements, the difference between the element values ​​of the elements in the first sample image feature and the second sample image feature is first determined, and the number of elements corresponding to the difference within the preset difference range is the fourth number of elements. Regarding the determination of the fifth number of elements, the difference between the element values ​​of the elements in the first sample image feature and the third sample image feature is first determined, and the number of elements corresponding to the difference within the preset difference range is the fifth number of elements.

[0098] The first difference similarity between the first sample image feature and the second sample image feature needs to be determined based on at least the first element number, the second element number, the fourth element number, and the first sample image feature and the second sample image feature. The second difference similarity between the first sample image feature and the third sample image feature needs to be determined based on at least the first element number, the third element number, the fifth element number, and the first sample image feature and the second sample image feature.

[0099] In an exemplary embodiment, the above step S202 is performed by inputting the first sample image feature, the second sample image feature and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first prediction behavior, the second prediction behavior and the third prediction behavior based on the first sample image feature, the second sample image feature and the third sample image feature, respectively. It can be optionally executed as follows: inputting the first sample image feature into the first classification network in the twin network to be trained to obtain the first prediction behavior; inputting the second sample image feature into the second classification network in the twin network to be trained to obtain the second prediction behavior; inputting the third sample image feature into the third classification network in the twin network to be trained to obtain the third prediction behavior.

[0100] In an exemplary embodiment, the above step S202 is executed by inputting the sample group into the twin network to be trained, and obtaining the first sample image feature corresponding to the first sample image in the sample group, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image based on the feature extraction network in the twin network to be trained. It can be optionally executed as follows: inputting the first sample image in the sample group into the first feature extraction network in the twin network to be trained to obtain the first sample image feature; inputting the second sample image in the sample group into the second feature extraction network in the twin network to be trained to obtain the second sample image feature; inputting the third sample image in the sample group into the third feature extraction network in the twin network to be trained to obtain the third sample image feature.

[0101] Specifically, an optional implementation method is provided, in which the feature extraction network included in the twin network to be trained provided in the present application specifically includes a first feature extraction network, a second feature extraction network and a third feature extraction network, and the classification network included in the twin network to be trained specifically includes a first classification network, a second classification network and a third classification network; wherein the first classification network is used to receive the result output by the corresponding first feature extraction network, the second classification network is used to receive the result output by the corresponding second feature extraction network, and the third classification network is used to receive the result output by the corresponding third feature extraction network.

[0102] That is, it can be considered that the twin network to be trained provided by the present application includes three sub-twin networks, the first sub-twin network includes the first feature extraction network and the first classification network, the second sub-twin network includes the second feature extraction network and the second classification network, and the third sub-twin network includes the third feature extraction network and the third classification network. Among them, the first sub-twin network is used to receive and process an image in a sample group, the second sub-twin network is used to receive another image in the same sample group, and the third sub-twin network is used to receive the last image in the same sample group.

[0103] For example, the first sample image in the sample group is input into the first feature extraction network, the first feature extraction network outputs the first sample image feature, and then the first sample image feature is input into the first classification network, and the first predicted behavior of the sample object in the first sample image is obtained based on the corresponding first classification network; the second sample image in the sample group is input into the second feature extraction network, the second feature extraction network outputs the second sample image feature, and then the second sample image feature is input into the second classification network, and the second predicted behavior of the sample object in the second sample image is obtained based on the corresponding second classification network; the third sample image in the sample group is input into the third feature extraction network, the third feature extraction network outputs the third sample image feature, and then the third sample image feature is input into the third classification network, and the third predicted behavior of the sample object in the third sample image is obtained based on the corresponding third classification network. For example, in the case where the sample images in the sample group have smoking or phone calls, the sample group is input into the twin network to be trained provided in the present application, and the first prediction result output by the classification network may be "there is smoking behavior or phone calls", the second prediction result may be "there is smoking behavior", and the third prediction result may be "there is phone calls".

[0104] It should also be added that, regarding the calculation of the second loss value, the third loss value, and the fourth loss value, a suitable loss function for image classification can be selected for relevant calculations, for example, the first loss function used to calculate the second loss value, the second loss function used to calculate the third loss value, and the third loss function used to calculate the fourth loss value may include but are not limited to the mean square error loss function, the mean absolute error loss function, or the cross entropy loss function. Among them, the first loss function, the second loss function, and the third loss function may be the same or different. For example, the loss function used to calculate the second loss value, the third loss value, and the fourth loss value may all select the cross entropy loss function (Cross Entropy Loss), so as to calculate the loss value between the predicted behavior type value and the actual behavior type value of a corresponding set of predicted behaviors based on the cross entropy loss function.

[0105] In an exemplary embodiment, the first feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network; the second feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network; and the third feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network.

[0106] Specifically, the twin networks to be trained provided in the present application include a first feature extraction network, a second feature extraction network and a third feature extraction network. These three networks can all choose to use convolutional neural networks. The specific type of convolutional neural network can be, for example, any one of the lightweight neural network MobileNet and the target detection network YOLO.

[0107] It should also be added that for the first feature extraction network, the second feature extraction network and the third feature extraction network included in the twin network to be trained, the same convolutional neural network can be selected, or at least two convolutional neural networks can be selected. That is, the type of any one of the first feature extraction network, the second feature extraction network and the third feature extraction network can be selected from MobileNet-V3 and YOLO-V8.

[0108] The MobileNet-V3 and YOLO-V8 provided here are optional versions corresponding to the convolutional neural network MobileNet and the convolutional neural network YOLO, respectively, but this application does not limit the specific version selection and can be adjusted and used according to needs.

[0109] In an exemplary embodiment, the multiple sample groups include a positive sample group and a negative sample group; the first sample image, the second sample image, and the third sample image (three sample images) included in the positive sample group are all images in which behaviors to be identified (first behaviors to be identified and / or second behaviors to be identified); the first sample image, the second sample image, and the third sample image (three sample images) included in the negative sample group are all images in which behaviors to be identified (first behaviors to be identified and second behaviors to be identified) do not exist.

[0110] Specifically, the sample group collected by the present application for implementing the training of the twin network to be trained includes both a positive sample group and a negative sample group, wherein the behavior of the sample objects in the image in the positive sample group includes at least one of the preset first behavior to be identified and the second behavior to be identified. For example, when the preset behavior to be identified is smoking behavior, the sample objects (such as the driver) in the first sample image, the second sample image, and the third sample image of the same sample group all have smoking behavior; for example, when the preset behavior to be identified is making a phone call, the sample objects (such as the driver) in the first sample image, the second sample image, and the third sample image of the same sample group all have making a phone call; in addition, there is another possibility that, for example, when the preset behavior to be identified is "smoking behavior or making a phone call", the first sample image of the same sample group has "smoking behavior or making a phone call", the second sample image corresponding to the first sample image including smoking behavior has "smoking behavior", and the third sample image corresponding to the first sample image including making a phone call has "making a phone call". Among the behaviors of the sample objects in the images in the negative sample group, there is no preset behavior to be identified. For example, when the preset behavior to be identified is smoking behavior, the sample objects (such as the driver) in the first sample image, the second sample image, and the third sample image of the same sample group do not have smoking behavior.

[0111] By selecting a training sample group for training the twin network, including both a positive sample group and a negative sample group, it is helpful to improve the recognition accuracy of the finally trained twin network for the behavior to be recognized.

[0112] It should also be added that the application of the trained twin network to the object behavior recognition in the vehicle cabin is only an optional application scenario provided by the application; however, in other similar scenarios, the twin network trained based on the training method provided by the application can also be used to recognize the behavior of objects in the scene; for example, the twin network trained based on the training method provided by the application can be used to recognize the behavior of workers in a factory environment; the twin network trained based on the training method provided by the application can be used to recognize the behavior of students in a classroom environment. Based on the different application scenarios of the twin network, the collection scenario of its training samples can also be adapted to the corresponding scenario, which is conducive to improving the behavior recognition effect of the trained twin network.

[0113] In an exemplary embodiment, the acquisition of multiple sample groups in step S201 may be specifically implemented by executing steps S211 to S214 (not shown), wherein:

[0114] S211, acquiring a plurality of sample images including sample objects, and determining a positive sample image to be selected and a negative sample image to be selected from the plurality of sample images;

[0115] S212, for the same behavior to be identified, obtaining a first similarity between the behavior in each of the to-be-selected positive sample images and the behavior to be identified, and screening a positive sample image from a plurality of to-be-selected positive sample images according to a comparison result of the first similarity and a first similarity threshold;

[0116] S213, obtaining a second similarity between the behavior in each candidate negative sample image and the behavior to be identified for the same behavior, and screening a negative sample image from a plurality of candidate negative sample images according to a comparison result between the second similarity and a second similarity threshold;

[0117] S214, after completing the image screening, respectively obtain the second sample image corresponding to the first target area and the third sample image corresponding to the second target area from the screened positive sample images and negative sample images; form a positive sample group based on the positive sample image and the corresponding second sample image and third sample image, and form a negative sample group based on the negative sample image and the corresponding second sample image and third sample image.

[0118] Specifically, a plurality of sample images including sample objects are first obtained, and then these sample images are divided into positive sample images to be selected and negative sample images to be selected; then, for the same behavior to be identified, a first similarity between the behavior of the sample object in each positive sample image to be selected and the behavior to be identified is calculated, that is, the similarity probability between the behavior of the sample object in the positive sample image to be selected and the behavior to be identified, and then, according to the comparison result of the first similarity and the first similarity threshold, the positive sample images to be selected whose first similarity is greater than or equal to the first similarity threshold are screened out as the screened positive sample images; at the same time, for the same behavior to be identified, a second similarity between the behavior of the sample object in each negative sample image to be selected and the behavior to be identified is calculated, that is, the similarity probability between the behavior of the sample object in the negative sample image to be selected and the behavior to be identified, and then, according to the comparison result of the second similarity and the second similarity threshold, the negative sample images to be selected whose second similarity is less than the second similarity threshold are screened out as the screened negative sample images.

[0119] It should be added that the present application does not limit the type of specific calculation formula applied to the behavior of the sample object in the sample image and the similarity probability between the behavior to be identified, and a suitable similarity calculation formula can be selected according to needs.

[0120] It should also be added that when the sample group selected is a vehicle cabin image that includes a sample object, the images used to train the twin network to be trained must include vehicle cabin images, but it is not limited to all images being vehicle cabin images, and some other images may also be included.

[0121] In addition, it should be added that for the twin network applied to the same application scenario, when training the twin network, the sample images of the sample pairs used can be collected in the corresponding same application scenario, or can be collected in multiple application scenarios. This application does not make specific limitations on this.

[0122] Regarding the acquisition of the sample group in step S201, the present application also provides an optional implementation method:

[0123] Acquire a first sample image of a sample object in a vehicle cabin; input the first sample image into a face detection model, and when it is detected that the sample object is contained in the first sample image, output face information corresponding to the sample object; based on the face information, intercept a face image corresponding to the sample object from the first sample image; input the face image into a face key point detection model to obtain the face key point coordinates of the sample object; determine a first target area and a second target area based on the face key point coordinates, and acquire a second sample image corresponding to the first target area and a third sample image corresponding to the second target area from the face image; form a sample group based on the first sample image, the second sample image and the third sample image of the sample object.

[0124] With respect to the training method of the twin network for object behavior recognition in a vehicle cabin provided in the present application, the present application provides an optional embodiment, and the specific training process is shown in the following steps S1 to S6, wherein:

[0125] S1, obtaining multiple sample groups; wherein each sample group includes a full-image smoking or phone call sample image (first sample image), a mouth ROI smoking sample image (second sample image), and an eye ROI phone call sample image (third sample image), wherein the mouth ROI smoking sample image is Figure 5 In the figure, D1 represents the eye ROI smoking sample image. Figure 5 In the figure, D2 represents the whole smoking or phone call sample image. Figure 5 For the same sample group, the mouth ROI smoking sample image D1 is obtained by intercepting the area related to the smoking behavior to be identified in the full-image smoking or phone call sample image D3, and the eye ROI phone call sample image D2 is obtained by intercepting the area related to the phone call behavior to be identified in the full-image smoking or phone call sample image D3.

[0126] S2, extract the ROI features of the data set D1 through the convolutional neural network 1, and obtain the feature vector G w (D1), which is the second sample image feature mentioned above; the ROI feature of the data set D2 is extracted through the convolutional neural network 2 to obtain the feature vector G w (D2), which is the third sample image feature mentioned above; the full image feature of the data set D3 is extracted through the convolutional neural network 3 to obtain the feature vector G w (D3), that is, the first sample image feature mentioned above; wherein, convolutional neural network 1 can adopt lightweight neural network MobileNet-V3 or target detection network YOLO-V8 based on demand, convolutional neural network 2 can adopt lightweight neural network MobileNet-V3 or target detection network YOLO-V8 based on demand, and similarly, convolutional neural network 3 can also adopt lightweight neural network MobileNet-V3 or target detection network YOLO-V8 based on demand. At the same time, parameter sharing can be achieved among the three networks.

[0127] S3, calculate the feature vector G by measuring the difference similarity w (D1) and the eigenvector G w The similarity measure of (D3), that is, the first sample image feature similarity. The specific calculation formula is shown in formula (1).

[0128]

[0129] Among them, G in formula (1) w (D3) is, for example, the first feature vector, which represents the first sample image feature, G w (D1) is, for example, a second feature vector, which represents a second sample image feature; e It means G w The number of elements in (D1) that are not equal to 0; f It means G w The number of elements in (D3) that are not equal to 0; ef It means G w (D1) and G w The number of elements in the difference of (D3) that are not equal to 0.

[0130] By measuring the difference similarity, the feature vector G is calculated. w (D2) and the eigenvector G w The similarity measure of (D3), that is, the similarity of the second sample image features, is specifically calculated as shown in formula (2).

[0131]

[0132] Among them, G in formula (2) w (D2) is the third eigenvector, which represents the third sample image feature, z e It means G w The number of elements in (D2) that are not equal to 0; f It means G w The number of elements in (D3) that are not equal to 0; ef It means G w (D2) and G w The number of elements in the difference of (D3) that are not equal to 0.

[0133] Based on the sum of the first sample image feature similarity and the second sample image feature similarity, the total sample image feature similarity is determined. The specific calculation formula is shown in formula (3).

[0134] S=S1(G w (D1), G w (D3))+S2(G w (D2), G w (D3)) (3)

[0135] Among them, S1(G w (D1),G w (D3) represents the similarity of the first sample image features, S2(G w (D2),G w (D3) represents the second sample image feature similarity, and S represents the total sample image feature similarity.

[0136] S4, the feature vector G w (D1), eigenvector G w (D2), eigenvector G w (D3) The similarity passes through the fully connected layer and then the Sigmoid layer, and is converted into the numerical sample relative probability RPS in the interval [0,1], that is, the similarity probability of judging whether the sample is smoking or making a phone call. When RPS>0.5, it means that the model judges that the sample is smoking or making a phone call, indicating that the ROI image classification model (second classification network) judges the positive sample image of smoking in the mouth ROI and the global image classification task (first classification network) judge the positive sample image of smoking in the whole picture more consistent, and indicating that the ROI image classification model (third classification network) judges the positive sample image of making a phone call in the eye ROI and the global image classification task (first classification network) judge the positive sample image of making a phone call in the whole picture more consistent. Among them, when the twin network (model) is trained, the model will tend to judge the positive sample of smoking and the positive sample of making a phone call.

[0137] Specifically, based on the feature vector G w(D1) and the eigenvector G w (D3), we can get the corresponding numerical sample relative probability RPS1, that is, the similarity probability of judging that the sample is smoking. When RPS1>0.5, it means that the model judges that the sample is smoking. Based on the feature vector G w (D2) and the eigenvector G w (D3), we can get the corresponding numerical sample relative probability RPS2, that is, the similarity probability of judging that the sample is a phone call. When RPS1>0.5, it means that the model judges that the sample is a phone call.

[0138] That is, if the probability of similarity of sample image features is higher, it means that the recognition result of the first classification network in the twin network for the target object's to-be-identified behavior in the first sample image (full image) is more consistent with the recognition result of the second classification network for the target object's to-be-identified behavior in the second sample image (partial image).

[0139] S5, construct a cross entropy loss function Lrps based on RPS, and the relevant construction formula can be shown in formula (4).

[0140]

[0141] Among them, L in formula (4) rps represents the first loss value, that is, the loss value between the first sample image and the second sample image; RPS represents the probability of similarity of sample image features, that is, the relative probability of the feature vector of the sample image mentioned above being converted into a numerical sample within a preset interval; λ represents the loss weight (preset loss weight); l represents the label of the sample group, that is, the actual behavior type of the sample group. If the sample group is a positive sample group, l is 1, and if the sample group is a negative sample group, l is 0; It represents the regularization function, which can enhance the generalization ability of the model.

[0142] Among them, when training the model (twin network), the similarity metric and classification task can assist each other in learning, so that the model can not only distinguish whether two samples are similar, but also identify the category of a single sample; in addition, since the ROI image classification task and the full-image image classification task have different noise patterns, learning the three image classification tasks at the same time can average the noise, share their respective overfitting risks, and obtain a more generalized representation.

[0143] S6, the model includes three classification tasks (two are ROI image classification tasks, one is the whole image classification task) and one similarity measurement task, so the total loss function consists of four parts, that is, the total loss value consists of the first loss value, the second loss value, the third loss value and the third loss value. The calculation method of the total loss value is shown in formula (5).

[0144] L = λ×(L cls (G w (D1)+L cls (G w (D2)+L cls (G w (D3)))+(1-2λ)×L rps (5)

[0145] Among them, L in formula (5) represents the total loss value of the sample group; λ represents the loss weight; G w (D3) is, for example, the first feature vector, which represents the first sample image feature, G w (D1) is the second eigenvector, which represents the second sample image feature; G w (D2) is the third eigenvector, which represents the third sample image feature; L rps It represents the first loss value, that is, the loss value between the first sample image and the second sample image; L cls (G w (D1)) represents the second loss value, that is, the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object; L cls (G w (D2)) represents the third loss value, that is, the loss value between the second predicted behavior and the actual behavior of the sample object; L cls (G w (D3)) represents the third loss value, that is, the loss value between the third predicted behavior and the actual behavior of the sample object.

[0146] In the training method of the twin network for object behavior recognition in the vehicle cabin shown in steps S1-S6, the first behavior to be recognized is set as smoking behavior, and the second behavior to be recognized is set as making a phone call behavior, but this is only an optional embodiment provided by the present application; the behavior to be recognized can also be set to other behaviors as required. Among them, the first sample image, the second sample image, and the third sample image in the same sample group can all be infrared images or RGB images, and multiple sample groups can include both infrared images and RGB images.

[0147] In addition, the present application also provides an iterative process of a deep learning model of infrared images based on a twin neural network. When it is necessary to train the twin network to recognize smoking behavior and phone calling behavior, it may include the following contents: select 3,000 positive sample infrared images of the cockpit mouth ROI smoking and 3,000 negative sample infrared images of the cockpit mouth ROI non-smoking as the training set D_1; select 3,000 positive sample infrared images of the cockpit eye ROI calling and 3,000 negative sample infrared images of the cockpit eye ROI non-calling as the training set D_2; select 3,000 positive sample infrared images of the entire cockpit smoking, 3,000 positive sample infrared images of the entire cockpit calling and 3,000 negative sample infrared images of the entire cockpit non-smoking and non-calling as the training set D_3; use D_1, D_2 and D_3 as the initial training sets to train a multi-task twin neural network deep learning initial model based on the fusion of ROI image classification and infrared full image target detection.

[0148] Furthermore, a multi-task twin neural network deep learning initial model based on ROI image classification and infrared full-image target detection fusion can be used to screen a data set of 1 million images, specifically including the following:

[0149] When screening ROI smoking positive samples, a multi-task twin neural network deep learning initial model based on ROI image classification and infrared full image target detection fusion is used to output samples with a probability score of less than 0.1 for positive samples of mouth ROI smoking, and add them to the positive sample set in the dataset D_1;

[0150] When screening positive samples of ROI making phone calls, a multi-task twin neural network deep learning initial model based on the fusion of ROI image classification and infrared full-image target detection is used to output samples with a probability score of less than 0.1 for positive samples of eye ROI making phone calls, and add them to the positive sample set in the data set D_2;

[0151] When screening the full-image smoking positive samples, a multi-task twin neural network deep learning initial model based on the fusion of ROI image classification and infrared full-image target detection is used to output the full-image smoking positive samples with a probability score less than 0.1 and add them to the positive sample set in the data set D_3;

[0152] When screening full-image phone call positive samples, a multi-task twin neural network deep learning initial model based on ROI image classification and infrared full-image target detection fusion is used to output full-image phone call positive samples with a probability score less than 0.1 and add them to the positive sample set in data set D_3;

[0153] When screening the mouth ROI non-smoking negative samples, a multi-task twin neural network deep learning initial model based on the fusion of ROI image classification and infrared full image target detection is used to output the mouth ROI non-smoking negative samples with a probability score greater than 0.9 and add them to the negative sample set in the dataset D_1;

[0154] When screening eye ROI non-calling negative samples, a multi-task twin neural network deep learning initial model based on ROI image classification and infrared full-image target detection fusion is used to output eye ROI non-calling negative samples with a probability score greater than 0.9 and add them to the negative sample set in the dataset D_2;

[0155] When screening the negative samples of non-smoking and non-calling in the whole image, a multi-task twin neural network deep learning initial model based on ROI image classification and infrared full image target detection fusion is used to output the samples with probability scores greater than 0.9 of the negative samples of non-smoking and non-calling in the whole image and add them to the negative sample set in the data set D_3;

[0156] Before adding the above-screened ROI and full-image positive and negative samples to the initial training set, it is necessary to browse the samples to be added to check whether there are any negative samples mixed in the positive samples, and whether there are any positive samples mixed in the negative samples.

[0157] When the model outputs the positive sample threshold set to 0.1 and the negative sample threshold set to 0.9 to filter the sample set, and the number of positive and negative samples screened out is less than 1000, the positive sample threshold is set to 0.2 and the negative sample threshold is set to 0.8. This cycle is repeated, using thresholds of 0.1, 0.2, 0.3, and 0.4 to filter positive samples, and thresholds of 0.9, 0.8, 0.7, and 0.6 to filter negative samples. Before updating the training set, it is necessary to browse the positive and negative samples to be added to see if there are negative samples mixed with positive samples, and positive samples mixed with negative samples. By continuously adding positive and negative samples to the initial training set in the above way, a multi-task twin neural network deep learning model based on the fusion of ROI image classification and infrared full-image target detection is obtained.

[0158] In the prior art, for example, when using the cockpit perception system in a vehicle to detect whether the driver among the vehicle members has smoking behavior or making phone calls, the relevant detection process is as follows: face detection is performed on the image frame sequence of the main driver's seat. If a face is detected, face key point regression is performed in the face ROI area; based on the key points, the ROI area near the mouth is intercepted and input into the smoking image classification model to judge the smoking behavior of the main driver's seat, and the ROI area near the eyes is intercepted and input into the phone call image classification model to judge the phone call behavior of the main driver's seat; the smoking image classification model gives a judgment on whether the main driver is smoking, and the phone call classification model gives a judgment on whether the main driver is making a phone call. In this process, making a phone call and smoking are the same classification model, but two such classification models are required for deployment, which invisibly wastes the precious computing power resources of the platform.

[0159] Based on this, the present application provides an optional implementation method, which is to input the main driver's seat image frame sequence into a face detection model. If a face is detected, face key point regression is performed in the face ROI area; based on the key points, the ROI area near the mouth and the ROI area near the eyes are simultaneously captured and input into a multi-task image classification model to judge the smoking and phone call behaviors in the main driver's seat.

[0160] In the training process of the multi-task infrared image classification model, this method needs to use the positive samples of smoking in the ROI area near the mouth, the negative samples of making phone calls in the ROI area near the eyes, and the negative samples (including negative samples of smoking in the ROI area near the mouth and negative samples mixed with negative samples in the ROI area near the eyes) as training sets. The multi-task infrared image model can output the judgment of the phone call and smoking tasks at the same time, saving the computing power of the GPU on the board, thereby providing support for the realization of more DMS perception tasks. It can be seen that this application uses a multi-task approach to combine multiple functional tasks into one model to reduce the amount of model data.

[0161] Moreover, in the existing detection process, only the ROI area is used to judge smoking or making phone calls, but the global features are not correlated. For example, when smoking in the driver's seat, there will be a behavioral feature of arm movement, that is, the arm is different from the normal driving state, but the ROI infrared image only involves the image of the mouth area, missing the arm feature, which will make it impossible to accurately judge some short cigarette butts.

[0162] Based on this, the present application provides an optional implementation method, which can combine the full-image infrared image features to assist the ROI infrared image to determine whether there is smoking or making phone calls in the driver's seat.

[0163] When the twin network is trained in the present application, the sample images input into the twin network to be trained are two ROI images and a global image. When the ROI image is a mouth area image, the corresponding global image can include the driver's arm area. When the ROI image is an eye area image, the corresponding global image can include the driver's arm area. This is because some drivers hold the phone to the eye area, that is, the temple area, when making a call. Therefore, based on the twin network training method provided in the present application, the trained twin network for object behavior recognition in the vehicle cabin can be used to determine whether the main driver's behavior is smoking or making a phone call.

[0164] Similarly, the above-mentioned judgment method for whether the driver is smoking or making a phone call can also be used to judge whether a member in the vehicle is smoking or making a phone call. That is, it can also provide support for more OMS perception tasks.

[0165] Among them, DMS stands for Driver Monitoring System, which monitors the driver and can monitor the driver's fatigue and dangerous driving behavior. OMS stands for Occupancy Monitoring System, which monitors the passengers and can monitor the passengers' age, status, emotions and behavior.

[0166] Based on this, the present application can also provide a method for object behavior recognition in a vehicle cabin using a twin network trained by the present application, specifically including: obtaining a first image of a target object (such as a driver or a member) in the cabin, and determining a second image and a third object based on the first image; the second image is a local image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be recognized of the target object; the third image is a local image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be recognized of the target object; the first image, the second image, and the third image are input into the twin network to obtain the similarity probability between the first image feature of the first image, the second image feature of the second image, and the third image feature of the third image; based on the similarity probability, determining a behavior recognition result for the target object; wherein the twin network is a twin network trained based on the twin network training method provided in the present application.

[0167] Among them, based on the similarity probability, the recognition result of the behavior to be identified for the target object is determined, specifically including: when it is determined that the similarity probability is greater than a preset threshold, the behavior recognition result for the target object is determined to be the occurrence of the first behavior to be identified or the second behavior to be identified; when it is determined that the similarity probability is less than or equal to the preset threshold, the behavior recognition result for the target object is determined to be the non-occurrence of the first behavior to be identified or the second behavior to be identified.

[0168] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0169] Based on the same inventive concept, the embodiment of the present application also provides a twin network training device for implementing the twin network training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more twin network training devices provided below can be referred to the limitations of the twin network training method above, and will not be repeated here.

[0170] In an exemplary embodiment, Figure 6 As shown, a training device 300 for a twin network is provided, comprising: a sample acquisition module 81, a sample processing module 82 and a parameter adjustment module 83, wherein:

[0171] The sample acquisition module 81 is used to acquire multiple sample groups; each sample group includes a first sample image, a second sample image and a third sample image of the sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to the first behavior to be identified, and the second target area is an area related to the second behavior to be identified;

[0172] The sample processing module 82 is used to input the sample group into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image in the sample group, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image based on the feature extraction network in the twin network to be trained; and input the first sample image feature, the second sample image feature, and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first predicted behavior, the second predicted behavior, and the third predicted behavior based on the first sample image feature, the second sample image feature, and the third sample image feature, respectively; determine the first loss value based on the difference similarity between the second sample image feature, the third sample image feature, and the first sample image feature, respectively; obtain the second loss value based on the first predicted behavior and the actual behavior corresponding to the sample group, obtain the third loss value based on the second predicted behavior and the actual behavior, and obtain the fourth loss value based on the third predicted behavior and the actual behavior;

[0173] The parameter adjustment module 83 is used to determine the total loss value corresponding to each sample group based on the first loss value, the second loss value, the third loss value and the fourth loss value, and when the total loss value does not meet the stop training condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the stop training condition is met, so as to obtain a trained twin network. The trained twin network is applied to at least two behavior recognitions of objects in the vehicle cabin.

[0174] In one embodiment, when the sample processing module 82 is used to determine the first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature, respectively, it is specifically used to: obtain the first difference similarity between the first sample image feature and the second sample image feature, and the second difference similarity between the first sample image feature and the third sample image feature; determine the total sample image feature similarity based on the sum of the first difference similarity and the second difference similarity; convert the total sample image feature similarity into a sample image feature similarity probability within a preset value range; determine the first loss value based on the sample image feature similarity probability, the actual behavior type value of the sample group and a preset loss weight.

[0175] In one embodiment, the sample processing module 82 is further used to respectively determine the number of first elements, the number of second elements, and the number of third elements whose element values ​​are within a preset value range in the first sample image feature, the second sample image feature, and the third sample image feature; and determine the number of fourth elements whose first difference is within the corresponding preset difference range and the number of fifth elements whose second difference is within the corresponding preset difference range based on the first difference and the second difference between the element values ​​in the first sample image feature and the second sample image feature and the third sample image feature respectively; determine the first difference similarity between the first sample image feature and the second sample image feature based on the first number of elements, the second number of elements, the fourth number of elements, and the first sample image feature and the second sample image feature; and determine the second difference similarity between the first sample image feature and the third sample image feature based on the first number of elements, the third number of elements, the fifth number of elements, and the first sample image feature and the third sample image feature.

[0176] In one embodiment, the sample processing module 82 is used to input the first sample image feature, the second sample image feature and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first prediction behavior, the second prediction behavior and the third prediction behavior respectively based on the first sample image feature, the second sample image feature and the third sample image feature. It is specifically used to: input the first sample image feature into the first classification network in the twin network to be trained to obtain the first prediction behavior; input the second sample image feature into the second classification network in the twin network to be trained to obtain the second prediction behavior; input the third sample image feature into the third classification network in the twin network to be trained to obtain the third prediction behavior.

[0177] In one embodiment, the sample processing module 82 is used to input the sample group into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image in the sample group, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image based on the feature extraction network in the twin network to be trained. It is specifically used to: input the first sample image in the sample group into the first feature extraction network in the twin network to be trained to obtain the corresponding first sample image feature; input the second sample image in the sample group into the second feature extraction network in the twin network to be trained to obtain the corresponding second sample image feature; input the third sample image in the sample group into the third feature extraction network in the twin network to be trained to obtain the corresponding third sample image feature.

[0178] In one embodiment, the first feature extraction network is one of the MobileNet feature extraction network and the YOLO feature extraction network; the second feature extraction network is one of the MobileNet feature extraction network and the YOLO feature extraction network; and the third feature extraction network is one of the MobileNet feature extraction network and the YOLO feature extraction network.

[0179] In one embodiment, the sample groups include a positive sample group and a negative sample group; the three sample images included in the positive sample group are images with behaviors to be identified; the three sample images included in the negative sample group are images without behaviors to be identified.

[0180] In one embodiment, when the sample acquisition module 81 is used to acquire multiple sample groups, it is specifically used to: acquire multiple sample images including sample objects, and determine the positive sample images to be selected and the negative sample images to be selected from the multiple sample images; for the same behavior to be identified, acquire the first similarity between the behavior in each of the positive sample images to be selected and the behavior to be identified, and filter the positive sample images from the multiple positive sample images to be selected based on the comparison result of the first similarity and the first similarity threshold; acquire the second similarity between the behavior in each of the negative sample images to be selected and the behavior to be identified for the same behavior, and filter the negative sample images from the multiple negative sample images to be selected based on the comparison result of the second similarity and the second similarity threshold; after completing the image screening, respectively acquire the second sample image corresponding to the first target area and the third sample image corresponding to the second target area from the filtered positive sample images and negative sample images; form a positive sample group based on the positive sample image and the corresponding second sample image and third sample image, and form a negative sample group based on the negative sample image and the corresponding second sample image and third sample image.

[0181] Each module in the above-mentioned twin network training device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0182] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a training method for a twin network is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0183] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0184] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0185] Acquire multiple sample groups; each sample group includes a first sample image, a second sample image, and a third sample image of the sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to the first behavior to be identified, and the second target area is an area related to the second behavior to be identified;

[0186] Input the sample group into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image in the sample group based on the feature extraction network in the twin network to be trained; and input the first sample image feature, the second sample image feature, and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first predicted behavior, the second predicted behavior, and the third predicted behavior respectively based on the first sample image feature, the second sample image feature, and the third sample image feature;

[0187] Determine a first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature respectively; obtain a second loss value based on the first predicted behavior and the actual behavior corresponding to the sample group, obtain a third loss value based on the second predicted behavior and the actual behavior, and obtain a fourth loss value based on the third predicted behavior and the actual behavior;

[0188] Based on the first loss value, the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is determined, and when the total loss value does not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the conditions for stopping training are met, so as to obtain a trained twin network, which is applied to at least two behavior recognitions of objects in the vehicle cabin.

[0189] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0190] Acquire multiple sample groups; each sample group includes a first sample image, a second sample image, and a third sample image of the sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to the first behavior to be identified, and the second target area is an area related to the second behavior to be identified;

[0191] Input the sample group into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image in the sample group based on the feature extraction network in the twin network to be trained; and input the first sample image feature, the second sample image feature, and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first predicted behavior, the second predicted behavior, and the third predicted behavior respectively based on the first sample image feature, the second sample image feature, and the third sample image feature;

[0192] Determine a first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature respectively; obtain a second loss value based on the first predicted behavior and the actual behavior corresponding to the sample group, obtain a third loss value based on the second predicted behavior and the actual behavior, and obtain a fourth loss value based on the third predicted behavior and the actual behavior;

[0193] Based on the first loss value, the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is determined, and when the total loss value does not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the conditions for stopping training are met, so as to obtain a trained twin network, which is applied to at least two behavior recognitions of objects in the vehicle cabin.

[0194] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0195] Acquire multiple sample groups; each sample group includes a first sample image, a second sample image, and a third sample image of the sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to the first behavior to be identified, and the second target area is an area related to the second behavior to be identified;

[0196] Input the sample group into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image in the sample group based on the feature extraction network in the twin network to be trained; and input the first sample image feature, the second sample image feature, and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first predicted behavior, the second predicted behavior, and the third predicted behavior respectively based on the first sample image feature, the second sample image feature, and the third sample image feature;

[0197] Determine a first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature respectively; obtain a second loss value based on the first predicted behavior and the actual behavior corresponding to the sample group, obtain a third loss value based on the second predicted behavior and the actual behavior, and obtain a fourth loss value based on the third predicted behavior and the actual behavior;

[0198] Based on the first loss value, the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is determined, and when the total loss value does not meet the conditions for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the conditions for stopping training are met, so as to obtain a trained twin network, which is applied to at least two behavior recognitions of objects in the vehicle cabin.

[0199] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0200] It should also be added that the images including user objects involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant images must comply with relevant regulations.

[0201] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0202] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0203] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a twin network, characterized in that: The method comprises: Acquire multiple sample groups; each of the sample groups includes a first sample image, a second sample image, and a third sample image of a sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified; Input the sample group into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image in the sample group based on the feature extraction network in the twin network to be trained; and input the first sample image feature, the second sample image feature, and the third sample image feature into the classification network in the twin network to be trained, respectively, to obtain the first predicted behavior, the second predicted behavior, and the third predicted behavior, respectively; Determine a first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature respectively; obtain a second loss value based on the first predicted behavior and the actual behavior corresponding to the sample group, obtain a third loss value based on the second predicted behavior and the actual behavior, and obtain a fourth loss value based on the third predicted behavior and the actual behavior; Based on the first loss value, the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is determined, and when the total loss value does not meet the condition for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the condition for stopping training is met, so as to obtain a trained twin network, which is applied to at least two behavior recognitions of objects in a vehicle cabin.

2. The method according to claim 1, characterized in that The determining of the first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature respectively includes: Acquire a first difference similarity between the first sample image feature and the second sample image feature, and a second difference similarity between the first sample image feature and the third sample image feature; Determining a total sample image feature similarity based on the sum of the first difference similarity and the second difference similarity; The first loss value is determined based on the total sample image feature similarity, the actual behavior type value of the sample group and a preset loss weight.

3. The method according to claim 1 or 2, characterized in that: Before determining the first loss value based on the difference similarity between the second sample image feature, the third sample image feature and the first sample image feature, the method further includes: respectively determining the number of first elements, the number of second elements, and the number of third elements whose element values ​​are within a preset value range in the first sample image feature, the second sample image feature, and the third sample image feature; and determining the number of fourth elements whose first difference value is within a corresponding preset difference range and the number of fifth elements whose second difference value is within a corresponding preset difference range based on the first difference value and the second difference value between the element values ​​in the first sample image feature and the second sample image feature and the third sample image feature, respectively; Determine a first difference similarity between the first sample image feature and the second sample image feature based on the first number of elements, the second number of elements, the fourth number of elements, and the first sample image feature and the second sample image feature; Based on the first number of elements, the third number of elements, the fifth number of elements, and the first sample image feature and the third sample image feature, a second difference similarity between the first sample image feature and the third sample image feature is determined.

4. The method according to claim 1, characterized in that: The first sample image feature, the second sample image feature and the third sample image feature are respectively input into the classification network in the twin network to be trained to obtain a first prediction behavior, a second prediction behavior and a third prediction behavior, respectively, including: Inputting the first sample image feature into the first classification network in the twin network to be trained to obtain the first predicted behavior; Inputting the second sample image feature into the second classification network in the twin network to be trained to obtain the second predicted behavior; The third sample image feature is input into the third classification network in the twin network to be trained to obtain the third predicted behavior.

5. The method according to claim 1, characterized in that: The step of inputting the sample group into the twin network to be trained, and obtaining, based on a feature extraction network in the twin network to be trained, a first sample image feature corresponding to the first sample image in the sample group, a second sample image feature corresponding to the second sample image, and a third sample image feature corresponding to the third sample image, comprises: Inputting the first sample image in the sample group into the first feature extraction network in the twin network to be trained to obtain corresponding first sample image features; Inputting the second sample image in the sample group into the second feature extraction network in the twin network to be trained to obtain corresponding second sample image features; The third sample image in the sample group is input into the third feature extraction network in the twin network to be trained to obtain the corresponding third sample image feature.

6. The method according to claim 5, characterized in that The first feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network; The second feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network; The third feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network.

7. The method according to claim 1, characterized in that The sample groups include a positive sample group and a negative sample group; The three sample images included in the positive sample group are all images containing behaviors to be identified; The three sample images included in the negative sample group are all images without the behavior to be identified.

8. The method according to claim 7, characterized in that The obtaining of multiple sample groups comprises: Acquire a plurality of sample images including sample objects, and determine a positive sample image to be selected and a negative sample image to be selected from the plurality of sample images; For the same behavior to be identified, obtaining a first similarity between the behavior in each of the to-be-selected positive sample images and the behavior to be identified, and screening a positive sample image from the plurality of to-be-selected positive sample images according to a comparison result of the first similarity and a first similarity threshold; Obtaining a second similarity between the behavior in each of the negative sample images to be selected and the behavior to be identified for the same behavior, and screening a negative sample image from the plurality of negative sample images to be selected according to a comparison result between the second similarity and a second similarity threshold; After completing the image screening, the second sample image corresponding to the first target area and the third sample image corresponding to the second target area are respectively obtained from the screened positive sample images and the screened negative sample images; a positive sample group is formed based on the positive sample image and the corresponding second sample image and the third sample image, and a negative sample group is formed based on the negative sample image and the corresponding second sample image and the third sample image.

9. A training device for a twin network, characterized in that: The device comprises: A sample acquisition module, used to acquire multiple sample groups; each of the sample groups includes a first sample image, a second sample image, and a third sample image of a sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified; A sample processing module, used for inputting the sample group into the twin network to be trained, and obtaining a first sample image feature corresponding to the first sample image in the sample group, a second sample image feature corresponding to the second sample image, and a third sample image feature corresponding to the third sample image based on a feature extraction network in the twin network to be trained; and inputting the first sample image feature, the second sample image feature, and the third sample image feature into a classification network in the twin network to be trained, respectively, to obtain a first predicted behavior, a second predicted behavior, and a third predicted behavior, respectively; determining a first loss value based on the difference similarity between the second sample image feature, the third sample image feature, and the first sample image feature, respectively; obtaining a second loss value based on the first predicted behavior and the actual behavior corresponding to the sample group, obtaining a third loss value based on the second predicted behavior and the actual behavior, and obtaining a fourth loss value based on the third predicted behavior and the actual behavior; A parameter adjustment module is used to determine the total loss value corresponding to each of the sample groups based on the first loss value, the second loss value, the third loss value and the fourth loss value, and when the total loss value does not meet the training stop condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of objects in a vehicle cabin.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images