Twin network training method, device, computer equipment and readable storage medium

Abstract

The present application relates to a training method, apparatus, computer equipment and readable storage medium for a twin network. The method comprises obtaining a plurality of sample groups and inputting them into a twin network to be trained, obtaining first to third sample image features of the first to third sample images based on a feature extraction network in the network; and obtaining first to third predicted behaviors of the first to third sample image features through a classification network; determining a first loss value based on the distance similarity and cosine similarity between the second and third sample image features and the first sample image feature respectively; obtaining second to fourth loss values ​​based on the first to third predicted behaviors and actual behaviors; determining a total loss value of the sample group based on the four loss values, and when at least one total loss value does not meet a condition for stopping training, updating network parameters and training again until the stopping condition is met, thereby obtaining a trained twin network, which can be applied to user object behavior recognition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, apparatus, computer device and readable storage medium for a twin network. Background Art

[0002] With the development of computer technology, image recognition technology is being used more and more widely in all walks of life; for example, in the use of vehicles, many traffic accidents are related to the driver's bad driving behavior. Therefore, accurate analysis and determination of the driver's real-time driving behavior based on image recognition technology is conducive to reducing the occurrence of traffic accidents; for example, in the use of machines in factories, many dangerous accidents are related to the workers' bad operating behavior. Therefore, accurate analysis and determination of the workers' real-time operating behavior based on image recognition technology is conducive to reducing the occurrence of work accidents; for example, in teaching scenarios, the decline in students' grades is closely related to the students' small movements in class. Therefore, analysis and determination of students' real-time listening behavior based on image recognition technology is conducive to improving students' academic performance.

[0003] In summary, in all walks of life, there is a demand for real-time identification of user behavior; however, in the existing technology, the accuracy of intelligent detection of user bad behavior is not ideal. Therefore, it is urgent to provide a behavior recognition model that can be used to accurately identify the user's specific behavior, so as to accurately identify the user's bad behavior in a shorter time. Summary of the invention

[0004] Based on this, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium and computer program product for a twin network that can train a highly accurate intelligent monitoring of user behavior in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a training method for a twin network, the method comprising: obtaining multiple sample groups; each of the sample groups comprises a first sample image, a second sample image and a third sample image of a sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified; the sample group is input into the twin network to be trained, and based on the feature extraction network in the twin network to be trained, first sample image features corresponding to the first sample image in the sample group, second sample image features corresponding to the second sample image, and third sample image features corresponding to the third sample image are obtained; and the first predicted behavior, the second predicted behavior, and the second predicted behavior are obtained respectively by inputting the sample groups into the classification network in the twin network to be trained. and a third predicted behavior; determine a first loss value based on the distance similarity and cosine similarity between the second sample image feature, the third sample image feature and the first sample image feature respectively; obtain a second loss value based on the difference between the first predicted behavior and the actual behavior corresponding to the sample group, obtain a third loss value based on the difference between the second predicted behavior and the actual behavior, and obtain a fourth loss value based on the difference between the third predicted behavior and the actual behavior; determine the total loss value corresponding to each of the sample groups based on the first loss value, the second loss value, the third loss value and the fourth loss value, and if the total loss value does not meet the training stop condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of user objects.

[0006] In the second aspect, the present application also provides a training device for a twin network, including: a sample acquisition module for acquiring multiple sample groups; each sample group includes a first sample image, a second sample image and a third sample image of a sample object; the second sample image is a local image of a first target area in the first sample image, and the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified; a sample processing module for inputting the sample group into the twin network to be trained, and obtaining a first sample image feature corresponding to the first sample image in the sample group, a second sample image feature corresponding to the second sample image, and a third sample image feature corresponding to the third sample image based on a feature extraction network in the twin network to be trained; and inputting the first sample image feature, the second sample image feature and the third sample image feature into a classification network in the twin network to be trained, respectively. network, respectively obtain the first predicted behavior, the second predicted behavior and the third predicted behavior; determine the first loss value based on the distance similarity and cosine similarity between the second sample image feature and the third sample image feature and the first sample image feature respectively; obtain the second loss value based on the difference between the first predicted behavior and the actual behavior corresponding to the sample group, obtain the third loss value based on the difference between the second predicted behavior and the actual behavior, and obtain the fourth loss value based on the difference between the third predicted behavior and the actual behavior; a parameter adjustment module is used to determine the total loss value corresponding to each of the sample groups based on the first loss value, the second loss value, the third loss value and the fourth loss value, and update the network parameters of the twin network to be trained when the total loss value does not meet the training stop condition, and train the twin network to be trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of user objects.

[0007] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any step in the above-mentioned first aspect when executing the computer program.

[0008] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any step in the above-mentioned first aspect when executed by a processor.

[0009] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements any step in the above-mentioned first aspect when executed by a processor.

[0010] The present application provides a training method, apparatus, computer equipment, computer-readable storage medium and computer program product for a twin network. The training method for the twin network mentioned above obtains multiple sample groups; each sample group includes a first sample image, a second sample image and a third sample image of a sample object; the second sample image is a local image of a first target area in the first sample image, and the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified; the sample group is input into the twin network to be trained, and based on the feature extraction network in the twin network to be trained, a first sample image feature corresponding to the first sample image, a second sample image feature corresponding to the second sample image and a third sample image feature corresponding to the third sample image in the sample group are obtained; and the first sample image feature, the second sample image feature and the third sample image feature are input into a classification network in the twin network to be trained, and the classification network Based on the first sample image feature, the second sample image feature and the third sample image feature, the first predicted behavior, the second predicted behavior and the third predicted behavior are obtained respectively; based on the distance similarity and cosine similarity between the second sample image feature and the third sample image feature and the first sample image feature respectively, the first loss value is determined; based on the difference between the first predicted behavior and the actual behavior corresponding to the sample group, the second loss value is obtained, based on the difference between the second predicted behavior and the actual behavior, the third loss value is obtained, and based on the difference between the third predicted behavior and the actual behavior, the fourth loss value is obtained; based on the first loss value, the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is determined, and when the total loss value does not meet the training stop condition, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of user objects.

[0011] It can be seen that the present application forms a sample group through a first sample image, a local image of the first target area related to the first behavior to be identified in the first sample image (second sample image), and a local image of the second target area related to the second behavior to be identified in the first sample image (third sample image), as training samples for the twin network to be trained, so that the sample image simultaneously includes an image of the first target area and an image of the second target area of ​​the sample object, and also includes a global image of the sample object, thereby increasing the types of sample images; wherein other feature information associated with the target area (the first target area and the second target area) in the global image is conducive to auxiliary identification of the specific behavior of the sample object in the target area, and is conducive to improving the recognition accuracy of the subsequently trained twin network for the behavior to be identified of the object in the image; and in the process of training the twin network , the distance similarity and cosine similarity are combined to calculate the distance similarity and cosine similarity between the second sample image feature and the third sample image feature of the sample image and the first sample image feature, so as to determine the first loss value. The combination of distance similarity and cosine similarity to determine the first loss value is conducive to improving the accuracy of the obtained first loss value, and thus is conducive to improving the recognition accuracy of the trained twin network for the to-be-recognized behavior of the object in the image; at the same time, the second loss value, the third loss value and the fourth loss value corresponding to the first sample image feature, the second sample image feature and the third sample image feature are obtained respectively; then, based on the first loss value and the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is obtained, so as to achieve good training of the twin network to be trained based on the total loss value. In summary, the twin network trained based on the method provided in this application can efficiently and accurately identify multiple types of specific behaviors of the target object based on pictures including the target object. For example, it can identify one or two types of bad behaviors of the target object. This helps to detect the target object's irregular or bad behavior as early as possible, and then remind the target object to change relevant behavioral habits, so as to effectively avoid the occurrence of adverse events caused by the target object's bad behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 A diagram of an application environment for training a twin network in one embodiment;

[0014] Figure 2 This is an application environment diagram of a trained twin network in one embodiment;

[0015] Figure 3 It is a flowchart of a training method of a twin network in one embodiment;

[0016] Figure 4 A schematic diagram of the structure of a twin network in one embodiment;

[0017] Figure 5 A schematic diagram of the structure of a twin network in another embodiment;

[0018] Figure 6 It is a structural block diagram of a training device of a twin network in one embodiment;

[0019] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0021] The training method of the twin network provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 902 communicates with the server 904 through the network. The terminal 902 can at least be used to obtain sample images for training the twin network to be trained, and perform certain preprocessing operations on the sample images to obtain the first sample image corresponding to the sample image; in addition, the classification label work for the sample image can also be realized based on the terminal 902. The data storage system can store the data that the server 904 needs to process. The data storage system can be integrated on the server 904, or it can be placed on the cloud or other network servers. The data storage system can be used to store at least a plurality of sample groups including a first sample image, a second sample image, and a third sample image, as well as a twin network to be trained and a twin network that has been trained. Among them, the terminal 902 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server 904 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0022] It should be noted that the training process of the twin network can be based on the server 904 to provide operations, or it can be based on the terminal 902 to provide related operations, or it can be based on the communication between the terminal 902 and the server 904 to implement related operations. This application does not make specific limitations on this.

[0023] Furthermore, the trained twin network provided in the embodiment of the present application can be applied to Figure 2 The application environment shown. Figure 2 This is only one application environment that can be selected for the trained twin network provided by this application, and this application is not limited to this; that is, the trained twin network can not only be used for Figure 2 The driver / passenger behavior recognition in the vehicle environment shown can also be used for example for worker behavior recognition in a factory environment, and can also be used for example for student behavior recognition in a teaching environment, etc. Specifically, the application scenario includes a vehicle 100, the vehicle 100 includes a vehicle controller 101, and the vehicle cabin includes an image acquisition device 102.

[0024] The vehicle controller 101 may first capture an image of a target object (eg, at least one of a driver and a passenger) in the cabin through the image capture device 102, and then process the captured image to obtain a first image of the target object, and store the first image. In the process of object behavior recognition of images based on the twin network trained in the present application, the vehicle controller 101 can first obtain a first image of the target object, and determine the second image and the third image based on the first image, that is, obtain the first local image (second image) and the second local image (third image) of the target object from the global image (first image) of the target object; wherein the first local image is an image corresponding to the area in the first image related to the first behavior to be recognized of the target object, and the second local image is an image corresponding to the area in the first image related to the second behavior to be recognized of the target object; further, the vehicle controller 101 inputs the first image, the second image and the third image belonging to the same sample group into the twin network trained in the present application together, so as to obtain the similarity probabilities between the first image feature of the first image and the second image feature of the second image and the third image feature of the third image respectively based on the twin network trained in the present application; finally, based on the image feature similarity probability, determine the recognition result of the behavior to be recognized (the first behavior to be recognized and the second behavior to be recognized) of the target object to determine whether the specific behavior of the target object includes the behavior to be recognized.

[0025] Among them, for the recognition result of the first image by the twin network obtained by training based on this application, this application provides an optional implementation method, for example, in the case where the first image includes the driver's smoking or calling behavior, the final output result is that the specific behavior of the target object includes "smoking or calling behavior", as for whether it is smoking behavior or calling behavior, no specific distinction can be made. When the driver receives the reminder of bad driving behavior, the driver himself can clearly know which bad behavior he has, that is, the driver can correct the bad behavior that is occurring based on his actual situation. In addition, this application also provides an optional implementation method, for example, in the case where the first image includes the driver's smoking or calling behavior, the final output result is that the specific behavior of the target object "exists smoking behavior", or "exists calling behavior", or "exists smoking and calling behavior at the same time", so that when the driver receives the reminder of bad driving behavior, he can clearly know which bad behavior he has, which is conducive to improving the efficiency of the driver in correcting bad driving behavior. That is, if necessary, further analysis and processing of the recognition results can be performed to confirm whether the image shows smoking or making a phone call. It should be noted that for a driver, it is possible to make a phone call and smoke at the same time, that is, a driver may have two or more bad driving behaviors at the same time.

[0026] In order to provide a behavior recognition model that can be used to accurately identify the specific bad behaviors of the target object, so as to accurately identify the bad behaviors and types of bad behaviors of the target object in a shorter time, the present application provides a twin network training method to train a twin network that can identify the behavior of the target object with high accuracy.

[0027] In an exemplary embodiment, Figure 3 As shown, a training method for a twin network is provided, including but not limited to the following steps S201 to S204. Among them:

[0028] S201, obtaining multiple sample groups; each sample group includes a first sample image, a second sample image, and a third sample image of the sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified.

[0029] Wherein, taking the template object of the training sample as the driver or passenger inside the vehicle as an example, the sample object can be the driver inside the vehicle or the passenger inside the vehicle. The first sample image can be a first sample image including at least one of the driver and the passenger; assuming that the first behavior to be identified is whether the driver smokes (smoking is a bad behavior), the first target area is an area related to the smoking behavior, such as the driver's mouth area. In this case, the second sample image is a local image corresponding to the mouth area in the first sample image of the driver. Assuming that the second behavior to be identified is whether the driver makes a phone call (making a phone call is a bad behavior), the second target area is an area related to the phone call behavior, such as the driver's eye area. In this case, the third sample image is a local image corresponding to the eye area in the first sample image of the driver. Wherein, the local image corresponding to the target area is also a face ROI image, and ROI (Region of Interest) refers to a specific area to be processed in an image, and the area to be processed is usually outlined by a box, a circle, an ellipse, an irregular polygon, etc.; that is, the second sample image and the third sample image can be, for example, images of two different areas framed from the first sample image by a box shape.

[0030] In an optional implementation, before step S201, an image including the sample object acquired by an image acquisition device in the vehicle cabin may be acquired, and the image may be preprocessed to obtain the image including the sample object as the first sample image. The image preprocessing may include denoising, cropping, etc., which is not specifically limited in this application.

[0031] The image acquisition device may be, for example, an infrared sensor or a visible light sensor. The infrared sensor may be, for example, an infrared camera for acquiring infrared images. The visible light sensor may be, for example, an ordinary camera for acquiring color images, which may also be called RGB images, where R is a red channel, G is a green channel, and B is a blue channel. The image of the sample object may be a single-channel infrared image of the vehicle cabin including the sample object acquired by the infrared sensor, or an RGB color image of the vehicle cabin including the sample object acquired by the visible light sensor. The images in the infrared image sample group may be images formed based on obtaining the radiation of the sample object in the infrared band.

[0032] The present application provides an optional implementation manner in which a vehicle cabin includes both an infrared sensor and a visible light sensor, wherein the infrared sensor is used to monitor and photograph the driver's seat in the vehicle, and the visible light sensor is used to monitor and photograph the passenger seat in the vehicle; however, this is only an optional implementation manner provided by the present application, and the present application is not limited thereto.

[0033] A further optional implementation is provided, where the acquired image of the sample object is a single-channel infrared image, it can be further converted into a three-channel infrared image to serve as the first sample image of the sample object.

[0034] Each sample group includes three images, specifically, a first sample image of a sample object, a second sample image cropped based on an area in the first sample image related to a first behavior to be identified of the sample object, and a third sample image cropped based on an area in the first sample image related to a second behavior to be identified of the sample object.

[0035] S202, input the sample group into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image in the sample group, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image based on the feature extraction network in the twin network to be trained; and input the first sample image feature, the second sample image feature, and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first predicted behavior, the second predicted behavior, and the third predicted behavior based on the first sample image feature, the second sample image feature, and the third sample image feature, respectively.

[0036] Among them, the twin network (twin neural network) to be trained provided in the present application may specifically include a feature extraction network and a classification network, that is, the twin network is at least composed of a feature extraction network and a classification network, wherein the input data of the feature extraction network is a sample group, and the input data of the classification network is the data output by the feature extraction network after processing the sample group.

[0037] In an optional embodiment, the first sample image, the second sample image, and the third sample image included in the sample group are respectively input into the feature extraction network in the twin network to be trained provided in the present application to obtain the first sample image feature based on the feature extraction network after processing the first sample image, and the second sample image feature based on the feature extraction network after processing the second sample image, and the third sample image feature based on the feature extraction network after processing the third sample image; then, the first sample image feature, the second sample image feature, and the third sample image feature are respectively input into the classification network in the twin network to be trained provided in the present application to obtain the first prediction behavior based on the classification network after processing the first sample image feature, and the second prediction behavior based on the classification network after processing the second sample image feature, and the third prediction behavior based on the classification network after processing the third sample image feature.

[0038] In an exemplary embodiment, the first sample image feature may be an image feature vector, and the second sample image feature and the third sample image feature may also be image feature vectors; for example, features in the first sample image (e.g., the full image) are extracted through a feature extraction network to obtain a first feature vector (first sample image feature), features in the second sample image (e.g., the first ROI image) are extracted through a feature extraction network to obtain a second feature vector (second sample image feature), and features in the third sample image (e.g., the second ROI image) are extracted through a feature extraction network to obtain a third feature vector (third sample image feature).

[0039] In an exemplary embodiment, the classification network is used to receive the first sample image feature corresponding to the first sample image to output the prediction result (first predicted behavior) of the specific behavior of the sample object in the first sample image; the classification network is also used to receive the second sample image feature corresponding to the second sample image to output the prediction result (second predicted behavior) of the specific behavior of the sample object in the second sample image; the classification network is also used to receive the third sample image feature corresponding to the third sample image to output the prediction result (third predicted behavior) of the specific behavior of the sample object in the third sample image. Wherein, in the case where the first sample image includes, for example, smoking behavior or phone call behavior, the corresponding first predicted behavior may be the existence of smoking behavior or phone call behavior; the second predicted behavior may be the existence of smoking behavior, and the third predicted behavior may be the existence of phone call behavior.

[0040] It should also be added that the classification network is based on predictions of pre-set behavior types that need to be identified. For example, when the pre-set behavior types that need to be identified include smoking behavior and phone call behavior, the classification network will output the output result that the first predicted behavior is smoking behavior or phone call behavior.

[0041] S203, determine the first loss value based on the distance similarity and cosine similarity between the second sample image feature, the third sample image feature and the first sample image feature respectively; obtain the second loss value based on the difference between the first predicted behavior and the actual behavior corresponding to the sample group, obtain the third loss value based on the difference between the second predicted behavior and the actual behavior, and obtain the fourth loss value based on the difference between the third predicted behavior and the actual behavior.

[0042] Among them, for the calculation of distance similarity, the Euclidean distance can be used to calculate the similarity of the images. The smaller the Euclidean distance, the greater the similarity. For the calculation of cosine similarity, the cosine value of the angle between the two eigenvectors is used to measure the cosine similarity between the two eigenvectors. The more similar the two vectors are, the smaller the angle is, and the cosine value is closer to 1.

[0043] The first predicted behavior obtained by executing step S202 of the present application is a prediction result of the behavior to be identified of the sample object in the first sample image based on the feature extraction network and the classification network in the twin network to be trained provided by the present application; the user can also determine the actual behavior of the sample object included in the first sample image through expert identification (artificial identification), and then, based on the difference between the first predicted behavior and the actual behavior corresponding to the sample group for the same first sample image, for example, by processing the numerical value corresponding to the behavior type of the first predicted behavior and the numerical value corresponding to the behavior type of the actual behavior, a second loss value for the first predicted behavior and the actual behavior can be obtained. The second predicted behavior obtained by executing step S202 of the present application is a prediction result of the to-be-identified behavior of the sample object in the second sample image based on the feature extraction network and the classification network in the twin network to be trained provided by the present application; the user can also determine the actual behavior of the sample object in the second sample image based on the actual behavior of the sample object included in the first sample image corresponding to the determined second sample image, and then, based on the difference between the second predicted behavior for the same second sample image and the actual behavior corresponding to the sample group, for example, by processing the numerical value corresponding to the behavior type of the second predicted behavior and the numerical value corresponding to the behavior type of the actual behavior, a third loss value for the second predicted behavior and the actual behavior can be obtained. The third predicted behavior obtained by executing step S202 of the present application is a prediction result of the to-be-identified behavior of the sample object in the third sample image based on the feature extraction network and the classification network in the twin network to be trained provided by the present application; the user can also determine the actual behavior of the sample object in the third sample image based on the actual behavior of the sample object included in the first sample image corresponding to the determined third sample image, and then, based on the difference between the third predicted behavior for the same third sample image and the actual behavior corresponding to the sample group, for example, by processing the numerical value corresponding to the behavior type of the third predicted behavior and the numerical value corresponding to the behavior type of the actual behavior, a fourth loss value for the third predicted behavior and the actual behavior can be obtained.

[0044] While calculating and obtaining the second loss value, the third loss value and the fourth loss value, the first loss values ​​corresponding to the first sample image feature, the second sample image feature and the third sample image feature can be obtained based on the distance similarity and cosine similarity between the first sample image feature and the second sample image feature, and the distance similarity and cosine similarity between the first sample image feature and the third sample image feature, respectively. The calculation of the loss value can be implemented based on the loss function.

[0045] S204, based on the first loss value, the second loss value, the third loss value and the fourth loss value, determine the total loss value corresponding to each sample group, and when the total loss value of at least one sample group does not meet the conditions for stopping training, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the conditions for stopping training are met, to obtain a trained twin network, and the trained twin network is applied to user object behavior recognition.

[0046] Among them, after obtaining the first loss value, the second loss value, the third loss value and the fourth loss value of the sample group based on the execution of step S203, the total loss value of the sample group can be further determined based on these four loss values, and then it can be judged whether the total loss value of the sample group meets the stop training condition of the twin network to be trained. When the total loss value of the sample does not meet the stop training condition, the network parameters of the twin network to be trained are updated based on the total loss value of each sample group used in a training process, and the twin network is trained again until the total loss values ​​of multiple sample groups meet the stop training condition. The training process of the twin network can be ended, and the trained twin network can be obtained at this time.

[0047] Since in the above process of illustrating the training method of the twin network provided by the present application, the data input to the twin network to be trained by the present application is a cabin image including sample objects, and at least part of the sample objects included in the cabin image have behaviors to be identified, therefore, the twin network trained by the present application can be used to recognize the (pre-set) behaviors of objects in the vehicle cabin; furthermore, when the behavior of objects in the vehicle cabin is subsequently identified based on the trained twin network, the accuracy of identifying the behavior of objects in the vehicle cabin can be improved.

[0048] It should be noted that the sample group provided above is an embodiment of a cabin image including a sample object, which is only an optional implementation provided by the present application, and the present application is not limited thereto; the specific image type including the sample object used to train the twin network can be selected based on the different requirements of the application scenarios of the trained twin network. For example, the selectable sample group is a classroom image including a sample object (a student), in which case the corresponding preset behaviors can be sleeping behavior, face covering behavior, eating behavior, and so on.

[0049] The training method of the above-mentioned twin network obtains multiple sample groups; each sample group includes a first sample image, a second sample image and a third sample image of the sample object; the second sample image is a local image of the first target area in the first sample image, and the third sample image is a local image of the second target area in the first sample image, the first target area is an area related to the first behavior to be identified, and the second target area is an area related to the second behavior to be identified; the sample group is input into the twin network to be trained, and based on the feature extraction network in the twin network to be trained, the first sample image feature corresponding to the first sample image, the second sample image feature corresponding to the second sample image and the third sample image feature corresponding to the third sample image in the sample group are obtained; and the first sample image feature, the second sample image feature and the third sample image feature are input into the classification network in the twin network to be trained, and the classification network is based on the first sample image feature, the second sample image feature and the third sample image feature. Sample image features are used to obtain the first predicted behavior, the second predicted behavior and the third predicted behavior respectively; the first loss value is determined based on the distance similarity and cosine similarity between the second sample image features and the third sample image features and the first sample image features respectively; the second loss value is obtained based on the difference between the first predicted behavior and the actual behavior corresponding to the sample group, the third loss value is obtained based on the difference between the second predicted behavior and the actual behavior, and the fourth loss value is obtained based on the difference between the third predicted behavior and the actual behavior; based on the first loss value, the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is determined, and when the total loss value does not meet the training stop condition, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of user objects.

[0050] It can be seen that the present application forms a sample group through a first sample image, a local image of the first target area related to the first behavior to be identified in the first sample image (second sample image), and a local image of the second target area related to the second behavior to be identified in the first sample image (third sample image), as training samples for the twin network to be trained, so that the sample image simultaneously includes an image of the first target area and an image of the second target area of ​​the sample object, and also includes a global image of the sample object, thereby increasing the types of sample images; wherein other feature information associated with the target area (the first target area and the second target area) in the global image is conducive to auxiliary identification of the specific behavior of the sample object in the target area, and is conducive to improving the recognition accuracy of the subsequently trained twin network for the behavior to be identified of the object in the image; and in the process of training the twin network In the method, a combination of distance similarity and cosine similarity is used to calculate the distance similarity and cosine similarity between the second sample image feature and the third sample image feature of the sample image and the first sample image feature, respectively, so as to determine the first loss value. The combination of distance similarity and cosine similarity to determine the first loss value is conducive to improving the accuracy of the obtained first loss value, and thus is conducive to improving the recognition accuracy of the trained twin network for the to-be-recognized behavior of the object in the image; at the same time, the second loss value, the third loss value and the fourth loss value corresponding to the first sample image feature, the second sample image feature and the third sample image feature are obtained respectively; then, based on the first loss value and the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each sample group is obtained, so as to achieve good training of the twin network to be trained based on the total loss value. In summary, the twin network trained based on the method provided in this application can efficiently and accurately identify multiple types of specific behaviors of the target object based on pictures including the target object. For example, it can identify one or two types of bad behaviors of the target object. This helps to detect the target object's irregular or bad behavior as early as possible, and then remind the target object to change relevant behavioral habits, so as to effectively avoid the occurrence of adverse events caused by the target object's bad behavior.

[0051] When a vehicle cabin image including at least one of the driver and the passenger is used as the sample group for training the twin network of the present application, multiple types of specific behaviors of the occupants in the vehicle can be efficiently and accurately identified based on the pictures including the occupants in the vehicle. For example, one or two types of bad behaviors of the occupants in the vehicle can be identified. This helps to detect the irregular or bad behaviors of the occupants in the vehicle as early as possible, and then remind the occupants to change their relevant behavioral habits, so as to effectively reduce the problem of the driver's attention being distracted during driving, thereby helping to reduce the occurrence of traffic accidents and protect the lives of the driver and passengers.

[0052] In an exemplary embodiment, the first loss value is determined based on the distance similarity and cosine similarity between the second sample image feature and the third sample image feature and the first sample image feature, respectively, in step S203, which can be specifically executed as follows, including steps S231 to S235 (not shown), wherein: S231, based on the product of the first distance similarity and the first cosine similarity between the first sample image feature and the second sample image feature, the first sample image feature similarity is determined; S232, based on the product of the second distance similarity and the second cosine similarity between the first sample image feature and the third sample image feature, the second sample image feature similarity is determined; S233, based on the sum of the first sample image feature similarity and the second sample image feature similarity, the total sample image feature similarity is determined; S234, the total sample image feature similarity is converted into a sample image feature similarity probability within a preset value range; S235, based on the sample image feature similarity probability, the actual behavior type value of the sample group and the preset loss weight, the first loss value is determined.

[0053] Specifically, first, based on the calculation formula of distance similarity, such as Euclidean distance, the first distance similarity between the first sample image feature and the second sample image feature is calculated, and the second distance similarity between the first sample image feature and the third sample image feature is calculated; and based on the calculation formula of cosine similarity, the first cosine similarity between the first sample image feature and the second sample image feature is calculated, and the second cosine similarity between the first sample image feature and the third sample image feature is calculated; then, based on the product of the first distance similarity and the first cosine similarity corresponding to the same sample group, the first sample image feature similarity corresponding to the sample group is calculated, and at the same time, based on the product of the second distance similarity and the second cosine similarity corresponding to the same sample group, the second sample image feature similarity corresponding to the sample group is calculated. Then, the first sample image feature similarity and the second sample image feature similarity corresponding to the same sample group are added to obtain the total sample image feature similarity corresponding to the sample group.

[0054] Afterwards, the total sample image feature similarity is converted into a sample image feature similarity probability within a preset value range, for example, the sample image feature similarity is converted into a sample image feature relative probability within the interval [0,1]. The sample image feature relative probability is used to determine whether the behaviors of the three sample images in the sample group are the same; for example, when the sample image feature relative probability is greater than the preset smoking or phone call probability, then it can be obtained that the sample objects in the three images of the sample group all have the behavior of smoking or making phone calls.

[0055] Finally, based on the total sample image feature similarity probability obtained by executing step S233, the actual behavior type value of the sample object in the determined sample group image, and the preset loss weight, a first loss value associated with a sample group can be determined. The actual behavior type value is, for example, when the actual behavior is the same as the preset behavior to be detected, the actual behavior type value can be set to 1, and when the actual behavior is different from the preset behavior to be detected, the actual behavior type value can be set to 0.

[0056] In an exemplary embodiment, the above step S202 is performed by inputting the first sample image feature, the second sample image feature and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first prediction behavior, the second prediction behavior and the third prediction behavior respectively based on the first sample image feature, the second sample image feature and the third sample image feature. It can be optionally executed as follows: inputting the first sample image feature into the first classification network in the twin network to be trained to obtain the first prediction behavior of the sample object; inputting the second sample image feature into the second classification network in the twin network to be trained to obtain the second prediction behavior of the sample object; inputting the third sample image feature into the third classification network in the twin network to be trained to obtain the third prediction behavior of the sample object.

[0057] In an exemplary embodiment, the above step S202 is executed by inputting the sample group into the twin network to be trained, and obtaining the first sample image feature corresponding to the first sample image in the sample group, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image based on the feature extraction network in the twin network to be trained. It can be optionally executed as follows: inputting the first sample image in the sample group into the first feature extraction network in the twin network to be trained to obtain the first sample image feature; inputting the second sample image in the sample group into the second feature extraction network in the twin network to be trained to obtain the second sample image feature; inputting the third sample image in the sample group into the third feature extraction network in the twin network to be trained to obtain the third sample image feature.

[0058] Specifically, an optional implementation method is provided, in which the feature extraction network included in the twin network to be trained provided in the present application specifically includes a first feature extraction network, a second feature extraction network and a third feature extraction network, and the classification network included in the twin network to be trained specifically includes a first classification network, a second classification network and a third classification network; wherein the first classification network is used to receive the result output by the corresponding first feature extraction network, the second classification network is used to receive the result output by the corresponding second feature extraction network, and the third classification network is used to receive the result output by the corresponding third feature extraction network.

[0059] That is, it can be considered that the twin network to be trained provided by the present application includes three sub-twin networks, the first sub-twin network includes the first feature extraction network and the first classification network, the second sub-twin network includes the second feature extraction network and the second classification network, and the third sub-twin network includes the third feature extraction network and the third classification network. Among them, the first sub-twin network is used to receive and process an image in a sample group, the second sub-twin network is used to receive another image in the same sample group, and the third sub-twin network is used to receive the last image in the same sample group.

[0060] For example, the first sample image in the sample group is input into the first feature extraction network, the first feature extraction network outputs the first sample image feature, and then the first sample image feature is input into the first classification network, and the first predicted behavior of the sample object in the first sample image is obtained based on the corresponding first classification network; the second sample image in the sample group is input into the second feature extraction network, the second feature extraction network outputs the second sample image feature, and then the second sample image feature is input into the second classification network, and the second predicted behavior of the sample object in the second sample image is obtained based on the corresponding second classification network; the third sample image in the sample group is input into the third feature extraction network, the third feature extraction network outputs the third sample image feature, and then the third sample image feature is input into the third classification network, and the third predicted behavior of the sample object in the third sample image is obtained based on the corresponding third classification network. For example, in the case where the sample images in the sample group have smoking or phone calls, the sample group is input into the twin network to be trained provided in the present application, and the first prediction result output by the classification network may be "there is smoking behavior or phone calls", the second prediction result may be "there is smoking behavior", and the third prediction result may be "there is phone calls".

[0061] In an exemplary embodiment, a second loss value is obtained based on the difference between a first predicted behavior and an actual behavior corresponding to a sample group, a third loss value is obtained based on the difference between the second predicted behavior and the actual behavior, and a fourth loss value is obtained based on the difference between the third predicted behavior and the actual behavior, including: determining a second loss value between a predicted behavior type value of the first predicted behavior and an actual behavior type value of the actual behavior of the sample group based on a first loss function (e.g., a cross-entropy loss function); determining a third loss value between a predicted behavior type value of the second predicted behavior and an actual behavior type value of the actual behavior of the sample group based on a second loss function (e.g., a cross-entropy loss function); and determining a fourth loss value between a predicted behavior type value of the third predicted behavior and an actual behavior type value of the actual behavior of the sample group based on a third loss function (e.g., a cross-entropy loss function).

[0062] Specifically, regarding the calculation of the second loss value, the third loss value, and the fourth loss value, a suitable loss function for image classification can be selected for relevant calculations. For example, the above-mentioned first loss function, the second loss function, and the third loss function may include but are not limited to a mean square error loss function, a mean absolute error loss function, or a cross entropy loss function. Among them, the first loss function, the second loss function, and the third loss function may be the same or different. For example, the loss function used to calculate the second loss value, the third loss value, and the fourth loss value may all select a cross entropy loss function (Cross EntropyLoss), so as to calculate the loss value between the predicted behavior type value and the actual behavior type value of a corresponding set of predicted behaviors based on the cross entropy loss function.

[0063] The calculation formula of the cross entropy loss function is, for example, ; K in the calculation formula of the cross entropy loss function is the total number of behavior types, The probability that sample i of the actual behavior belongs to the jth class is represented by =1 means it belongs to, =0 means not belonging; represents the probability that sample i of the predicted behavior belongs to the jth class, =1 means it belongs to, =0 means not belonging. Based on the first loss function, the second loss value between the predicted behavior type value of the first predicted behavior and the actual behavior type value of the actual behavior of the sample group is determined, that is, the predicted behavior type value of the first predicted behavior is calculated by the cross entropy loss function. , and the actual behavior type value The resulting value is the second loss value.

[0064] In an exemplary embodiment, the first feature extraction network is one of the MobileNet feature extraction network and the YOLO feature extraction network; the second feature extraction network is one of the MobileNet feature extraction network and the YOLO feature extraction network; the third feature extraction network is one of the MobileNet feature extraction network and the YOLO feature extraction network. Specifically, the twin networks to be trained provided in the present application include the first feature extraction network, the second feature extraction network, and the third feature extraction network. These three networks can all choose to use convolutional neural networks. The specific type of convolutional neural network can be, for example, any one of the lightweight neural network MobileNet and the target detection network YOLO.

[0065] It should also be added that for the first feature extraction network, the second feature extraction network and the third feature extraction network included in the twin network to be trained, the same convolutional neural network can be selected, or at least two convolutional neural networks can be selected. That is, the type of any one of the first feature extraction network, the second feature extraction network and the third feature extraction network can be selected from MobileNet-V3 and YOLO-V8. The MobileNet-V3 and YOLO-V8 provided here are optional versions corresponding to the convolutional neural network MobileNet and the convolutional neural network YOLO, respectively, but this application does not limit the specific version selection and can be adjusted according to needs.

[0066] In an exemplary embodiment, the multiple sample groups include a positive sample group and a negative sample group; the first sample image, the second sample image, and the third sample image (three sample images) included in the positive sample group are all images with behaviors to be identified (first behaviors to be identified and / or second behaviors to be identified); the first sample image, the second sample image, and the third sample image (three sample images) included in the negative sample group are all images without behaviors to be identified (first behaviors to be identified and second behaviors to be identified).

[0067] Specifically, the sample group collected by the present application for implementing the training of the twin network to be trained includes both a positive sample group and a negative sample group, wherein the behavior of the sample objects in the image in the positive sample group includes at least one of the preset first behavior to be identified and the second behavior to be identified. For example, when the preset behavior to be identified is smoking behavior, the sample objects (such as the driver) in the first sample image, the second sample image, and the third sample image of the same sample group all have smoking behavior; for example, when the preset behavior to be identified is making a phone call, the sample objects (such as the driver) in the first sample image, the second sample image, and the third sample image of the same sample group all have making a phone call; in addition, there is another possibility that, for example, when the preset behavior to be identified is "smoking behavior or making a phone call", the first sample image of the same sample group has "smoking behavior or making a phone call", the second sample image corresponding to the first sample image including smoking behavior has "smoking behavior", and the third sample image corresponding to the first sample image including making a phone call has "making a phone call". Among the behaviors of the sample objects in the images in the negative sample group, there is no preset behavior to be identified. For example, when the preset behavior to be identified is smoking behavior, the sample objects (such as the driver) in the first sample image, the second sample image, and the third sample image of the same sample group do not have smoking behavior.

[0068] By selecting a training sample group for training the twin network, including both a positive sample group and a negative sample group, it is helpful to improve the recognition accuracy of the finally trained twin network for the behavior to be recognized.

[0069] In an exemplary embodiment, the acquisition of multiple sample groups in step S201 can be specifically implemented by executing steps S211 to S214 (not shown), wherein: S211, acquiring multiple sample images including sample objects, and determining a positive sample image to be selected and a negative sample image to be selected from the multiple sample images; S212, for the same behavior to be identified, acquiring a first similarity between the behavior in each positive sample image to be selected and the behavior to be identified, and screening a positive sample image from the multiple positive sample images to be selected according to a comparison result of the first similarity and a first similarity threshold; S213, acquiring a positive sample image for the same The method further comprises the following steps: determining a second similarity between the behavior in each candidate negative sample image of the behavior and the behavior to be identified, and screening the negative sample image from the multiple candidate negative sample images according to the comparison result of the second similarity and the second similarity threshold; S214, after completing the image screening, respectively obtaining the second sample image corresponding to the first target area and the third sample image corresponding to the second target area from each screened positive sample image and each negative sample image; forming a positive sample group based on the positive sample image and the corresponding second sample image and the third sample image, and forming a negative sample group based on the negative sample image and the corresponding second sample image and the third sample image.

[0070] Specifically, a plurality of sample images including sample objects are first obtained, and then these sample images are divided into positive sample images to be selected and negative sample images to be selected; then, for the same behavior to be identified, a first similarity between the behavior of the sample object in each positive sample image to be selected and the behavior to be identified is calculated, that is, the similarity probability between the behavior of the sample object in the positive sample image to be selected and the behavior to be identified, and then, according to the comparison result of the first similarity and the first similarity threshold, the positive sample images to be selected whose first similarity is greater than or equal to the first similarity threshold are screened out as the screened positive sample images; at the same time, for the same behavior to be identified, a second similarity between the behavior of the sample object in each negative sample image to be selected and the behavior to be identified is calculated, that is, the similarity probability between the behavior of the sample object in the negative sample image to be selected and the behavior to be identified, and then, according to the comparison result of the second similarity and the second similarity threshold, the negative sample images to be selected whose second similarity is less than the second similarity threshold are screened out as the screened negative sample images.

[0071] It should be added that the application does not limit the type of specific calculation formula used for the behavior of the sample object in the sample image and the similarity probability between the behavior to be identified, and the appropriate similarity calculation formula can be selected according to the needs. It should also be added that when the sample group is selected to include the cabin image of the sample object, the image used to train the twin network to be trained must include the vehicle cabin image, but it is not limited to all images are vehicle cabin images, and some other images can also be included.

[0072] In addition, it should be added that for the twin network applied to the same application scenario, when training the twin network, the sample images of the sample group used can be collected in the corresponding same application scenario, or can be collected in multiple application scenarios. This application does not make specific limitations on this.

[0073] With respect to obtaining a sample group in step S201, the present application also provides an optional implementation method as follows:

[0074] Acquire a first sample image of a sample object in a vehicle cabin; input the first sample image into a face detection model, and when it is detected that the sample object is contained in the first sample image, output face information corresponding to the sample object; based on the face information, intercept a face image corresponding to the sample object from the first sample image; input the face image into a face key point detection model to obtain the face key point coordinates of the sample object; determine a first target area and a second target area based on the face key point coordinates, and acquire a second sample image corresponding to the first target area and a third sample image corresponding to the second target area from the face image; form a sample group based on the first sample image, the second sample image and the third sample image of the sample object.

[0075] In addition, the applicant has found through research that in today's life, with the increasing demand for vehicle use, traffic accidents occur from time to time. Many traffic accidents are related to the driver's bad driving behavior, and some traffic accidents are caused by the bad riding behavior of the passengers in the car. In the existing vehicle use environment, the driver often does not pay attention to his driving behavior during driving, and the passengers in the car do not pay attention to their riding behavior in the car. For example, there are common bad behaviors of drivers and passengers such as smoking and making phone calls in the car. Among them, similar behaviors such as the driver making phone calls while driving can easily distract the driver's driving concentration, and the smoking behavior of the passengers during driving can easily cause quarrels between the passengers and the driver, which will also distract the driver's driving concentration.

[0076] Although the use of vehicles facilitates people's travel, the safety of drivers and passengers is of paramount importance. In order to improve the safety of drivers and passengers in the car, real-time monitoring and intelligent evaluation of the driver's driving behavior and the passengers' riding behavior will help to detect irregular or bad behaviors of the car members as early as possible, and then remind the car members to change their relevant behavioral habits, so as to effectively reduce the problem of the driver's attention being distracted during driving, thereby helping to reduce the occurrence of traffic accidents and protect the lives of drivers and passengers.

[0077] However, in the relevant existing technologies, the accuracy of intelligent detection of bad behaviors of vehicle occupants is not ideal. Therefore, there is an urgent need to provide a behavior recognition model that can be used to accurately identify the behaviors of vehicle occupants, so as to accurately identify the bad behaviors of vehicle occupants in a shorter time.

[0078] In order to accurately identify the bad behavior of the occupants in the vehicle in a relatively short time, the present application provides an optional embodiment for the training method of the twin network provided in the present application, and provides a training method of the twin network applied to the object behavior recognition in the vehicle cabin. The specific training process is shown in the following steps S1 to S6, wherein:

[0079] S1, obtaining multiple sample groups; wherein each sample group includes a full-image smoking or phone call sample image (first sample image), a mouth ROI smoking sample image (second sample image), and an eye ROI phone call sample image (third sample image), wherein the mouth ROI smoking sample image is Figure 5 In the figure, D1 represents the eye ROI smoking sample image. Figure 5 In the figure, D2 represents the whole smoking or phone call sample image. Figure 5 For the same sample group, the mouth ROI smoking sample image D1 is obtained by intercepting the area related to the smoking behavior to be identified in the full-image smoking or phone call sample image D3, and the eye ROI phone call sample image D2 is obtained by intercepting the area related to the phone call behavior to be identified in the full-image smoking or phone call sample image D3.

[0080] S2, extract the ROI features of the data set D1 through the convolutional neural network 1, and obtain the feature vector G w (D1), which is the second sample image feature mentioned above; the ROI feature of the data set D2 is extracted through the convolutional neural network 2 to obtain the feature vector G w (D2), which is the third sample image feature mentioned above; the full image feature of the data set D3 is extracted through the convolutional neural network 3 to obtain the feature vector G w(D3), that is, the first sample image feature mentioned above; wherein, convolutional neural network 1 can adopt lightweight neural network MobileNet-V3 or target detection network YOLO-V8 based on demand, convolutional neural network 2 can adopt lightweight neural network MobileNet-V3 or target detection network YOLO-V8 based on demand, and similarly, convolutional neural network 3 can also adopt lightweight neural network MobileNet-V3 or target detection network YOLO-V8 based on demand. At the same time, parameter sharing can be achieved among the three networks.

[0081] S3, by combining distance similarity and cosine similarity, the feature vector G is calculated w (D1) and the eigenvector G w The similarity measure of (D3), that is, the similarity of the first sample image features. The specific calculation formula is shown in formula (1).

[0082] (1)

[0083] Among them, dis(G w (D1),G w (D3)) represents G w (D1) and G w (D3) distance similarity between; cos(G w (D1),G w (D3)) represents G w (D1) and G w (D3) between the cosine similarity; where G w (D3) is, for example, the first feature vector, which represents the first sample image feature, G w (D1) is, for example, the second feature vector, which represents the second sample image feature; dis() represents the distance similarity; and cos() represents the cosine similarity.

[0084] The feature vector G is calculated by combining distance similarity and cosine similarity. w (D2) and the eigenvector G w The similarity measure of (D3), that is, the similarity of the second sample image features. The specific calculation formula is shown in formula (2).

[0085] (2)

[0086] Among them, dis(G w (D2),G w (D3)) represents G w (D2) and G w(D3) distance similarity between; cos(G w (D2),G w (D3)) represents G w (D2) and G w (D3) between the cosine similarity; where G w (D2) is the third eigenvector, which represents the third sample image feature.

[0087] Based on the sum of the first sample image feature similarity and the second sample image feature similarity, the total sample image feature similarity is determined. The specific calculation formula is shown in formula (3).

[0088] (3)

[0089] Among them, S1(G w (D1),G w (D3) represents the similarity of the first sample image features, S2(G w (D2),G w (D3) represents the second sample image feature similarity, and S represents the total sample image feature similarity.

[0090] S4, the feature vector G w (D1), eigenvector G w (D2), eigenvector G w (D3) The similarity passes through the fully connected layer and then the Sigmoid layer, and is converted into the numerical sample relative probability RPS in the interval [0,1], that is, the similarity probability of judging whether the sample is smoking or making a phone call. When RPS>0.5, it means that the model judges that the sample is smoking or making a phone call, indicating that the judgment of the ROI image classification model (second classification network) on the positive sample image of smoking in the mouth ROI is more consistent with the judgment of the global image classification task (first classification network) on the positive sample image of smoking in the whole image, and indicating that the judgment of the ROI image classification model (third classification network) on the positive sample image of making a phone call in the eye ROI is more consistent with the judgment of the global image classification task (first classification network) on the positive sample image of making a phone call in the whole image. Among them, when the twin network (model) is trained, the model will tend to judge the positive sample of smoking and the positive sample of making a phone call.

[0091] Specifically, based on the feature vector G w (D1) and the eigenvector G w (D3), we can get the corresponding numerical sample relative probability RPS1, that is, the similarity probability of judging that the sample is smoking. When RPS1>0.5, it means that the model judges that the sample is smoking. Based on the feature vector G w (D2) and the eigenvector G w(D3), we can get the corresponding numerical sample relative probability RPS2, that is, the similarity probability of judging that the sample is a phone call. When RPS1>0.5, it means that the model judges that the samples are all phone calls.

[0092] That is, if the probability of similarity of sample image features is higher, it means that the recognition result of the first classification network in the twin network for the target object's to-be-identified behavior in the first sample image (full image) is more consistent with the recognition result of the second classification network for the target object's to-be-identified behavior in the second sample image (partial image).

[0093] S5, construct the cross entropy loss function Lrps based on RPS, and the relevant construction formula can be shown in formula (4).

[0094] (4)

[0095] Among them, L in formula (4) rps It represents the first loss value, that is, the loss value between the first sample image and the second sample image; RPS represents the probability of similarity of sample image features, that is, the relative probability of the feature vector of the sample image above being converted into a numerical sample within a preset interval; λ represents the loss weight (preset loss weight); It represents the label of the sample group, that is, the actual behavior type of the sample group. If the sample group is a positive sample group, then is 1, if the sample group is a negative sample group, then is 0; It represents the regularization function, which can enhance the generalization ability of the model.

[0096] Among them, when training the model (twin network), the similarity metric and classification task can assist each other in learning, so that the model can not only distinguish whether two samples are similar, but also identify the category of a single sample; in addition, since the ROI image classification task and the full-image image classification task have different noise patterns, learning the three image classification tasks at the same time can average the noise, share their respective overfitting risks, and obtain a more generalized representation.

[0097] S6, the model includes three classification tasks (two are ROI image classification tasks, one is the whole image classification task) and one similarity measurement task, so the total loss function consists of four parts, that is, the total loss value consists of the first loss value, the second loss value, the third loss value and the third loss value. The calculation method of the total loss value is shown in formula (5).

[0098] (5)

[0099] Among them, L in formula (5) represents the total loss value of the sample group; λ represents the loss weight; G w(D1) is the second eigenvector, which represents the second sample image feature; G w (D2) is the third eigenvector, which represents the third sample image feature; L rps It represents the first loss value, that is, the loss value between the first sample image and the second sample image; L cls (G w (D1)) represents the second loss value, that is, the loss value between the first predicted behavior of the sample object and the actual behavior of the sample object; L cls (G w (D2)) represents the third loss value, that is, the loss value between the second predicted behavior and the actual behavior of the sample object; L cls (G w (D3)) represents the third loss value, that is, the loss value between the third predicted behavior and the actual behavior of the sample object.

[0100] In the training method of the twin network applied to user object behavior recognition shown in steps S1-S6, the first behavior to be recognized is set as smoking behavior, and the second behavior to be recognized is set as calling behavior, but this is only an optional embodiment provided by the present application; the behavior to be recognized can also be set to other behaviors according to needs. Among them, the first sample image, the second sample image, and the third sample image in the same sample group can all be infrared images or RGB images, and multiple sample groups can include both infrared images and RGB images.

[0101] In addition, the present application also provides an iterative process of a deep learning model of infrared images based on a twin neural network. When it is necessary to train the twin network to recognize smoking behavior and phone calling behavior, it may include the following contents: select 3,000 positive sample infrared images of the cockpit mouth ROI smoking and 3,000 negative sample infrared images of the cockpit mouth ROI non-smoking as the training set D_1; select 3,000 positive sample infrared images of the cockpit eye ROI calling and 3,000 negative sample infrared images of the cockpit eye ROI non-calling as the training set D_2; select 3,000 positive sample infrared images of the entire cockpit smoking, 3,000 positive sample infrared images of the entire cockpit calling and 3,000 negative sample infrared images of the entire cockpit non-smoking and non-calling as the training set D_3; use D_1, D_2 and D_3 as the initial training sets to train a multi-task twin neural network deep learning initial model based on the fusion of ROI image classification and infrared full image target detection.

[0102] Furthermore, a multi-task twin neural network deep learning initial model based on ROI image classification and infrared full-image target detection fusion can be used to screen a data set including 1 million images, specifically including the following contents: when screening ROI smoking positive samples, use the multi-task twin neural network deep learning initial model based on ROI image classification and infrared full-image target detection fusion to output samples with a probability score of less than 0.1 for the mouth ROI smoking positive samples, and add them to the positive sample set in the data set D_1; when screening ROI calling positive samples, use the multi-task twin neural network deep learning initial model based on ROI image classification and infrared full-image target detection fusion to output samples with a probability score of less than 0.1 for the eye ROI calling positive samples, and add them to the positive sample set in the data set D_2; when screening the full-image smoking positive samples, use the multi-task twin neural network deep learning initial model based on ROI image classification and infrared full-image target detection fusion to output samples with a probability score of less than 0.1 for the full-image smoking positive samples, and add them to the positive sample set in the data set D_3; when screening the full-image calling positive samples, use the multi-task twin neural network deep learning initial model based on ROI image classification and infrared full-image target detection fusion, The samples with a probability score of less than 0.1 for positive samples of the full image making phone calls are added to the positive sample set in the data set D_3; when screening the mouth ROI non-smoking negative samples, the multi-task twin neural network deep learning initial model based on ROI image classification and infrared full image target detection fusion is used to output the samples with a probability score greater than 0.9 for negative samples of the mouth ROI non-smoking and added to the negative sample set in the data set D_1; when screening the eye ROI non-making phone calls negative samples, the multi-task twin neural network deep learning initial model based on ROI image classification and infrared full image target detection fusion is used to output the eye ROI non-making phone calls negative samples. Samples with a probability score of calling negative samples greater than 0.9 are added to the negative sample set in the dataset D_2; when screening the negative samples of non-smoking and non-calling in the whole image, the multi-task twin neural network deep learning initial model based on ROI image classification and infrared full image target detection fusion is used to output the samples with a probability score of non-smoking and non-calling negative samples in the whole image greater than 0.9 and add them to the negative sample set in the dataset D_3; before adding the above-screened ROI and full image positive and negative samples to the initial training set, it is necessary to browse the samples to be added to check whether there are negative samples mixed in the positive samples and whether there are positive samples mixed in the negative samples. When the model outputs the positive sample threshold setting of 0.1 and the negative sample threshold setting of 0.9 to screen the sample set, and the number of positive and negative samples screened out is less than 1000, the positive sample threshold is set to 0.2 and the negative sample threshold is set to 0.8. This cycle is repeated, and the positive samples are screened with thresholds of 0.1, 0.2, 0.3, and 0.4, and the negative samples are screened with thresholds of 0.9, 0.8, 0.7, and 0.6. Before updating the training set, you need to browse the positive and negative samples to be added to see if there are any negative samples mixed in the positive samples or any positive samples mixed in the negative samples.By continuously adding positive and negative samples to the initial training set in the above way, a multi-task twin neural network deep learning model based on the fusion of ROI image classification and infrared full-image target detection is obtained.

[0103] In the prior art, for example, when using the cockpit perception system in a vehicle to detect whether the driver among the vehicle members has smoking behavior or making phone calls, the relevant detection process is as follows: face detection is performed on the image frame sequence of the main driver's seat. If a face is detected, face key point regression is performed in the face ROI area; based on the key points, the ROI area near the mouth is intercepted and input into the smoking image classification model to judge the smoking behavior of the main driver's seat, and the ROI area near the eyes is intercepted and input into the phone call image classification model to judge the phone call behavior of the main driver's seat; the smoking image classification model gives a judgment on whether the main driver is smoking, and the phone call classification model gives a judgment on whether the main driver is making a phone call. In this process, making a phone call and smoking are the same classification model, but two such classification models are required for deployment, which invisibly wastes the precious computing power resources of the platform.

[0104] Based on this, the present application provides an optional implementation method, which is to input the main driver's seat image frame sequence into a face detection model. If a face is detected, face key point regression is performed in the face ROI area; based on the key points, the ROI area near the mouth and the ROI area near the eyes are simultaneously captured and input into a multi-task image classification model to judge the smoking and phone call behaviors in the main driver's seat.

[0105] In the process of training the multi-task infrared image classification model, this method needs to use the positive samples of smoking in the ROI area near the mouth, the negative samples of making phone calls in the ROI area near the eyes, and the negative samples (including negative samples of smoking in the ROI area near the mouth and negative samples of the ROI area near the eyes) as training sets. The multi-task infrared image model can output the judgment of the phone call and smoking tasks at the same time, saving the GPU computing power on the board, thereby providing support for the realization of more DMS perception tasks. It can be seen that this application uses a multi-task approach to combine multiple functional tasks into one model to reduce the amount of model data.

[0106] Moreover, in the existing detection process, only the ROI area is used to judge smoking or making phone calls, but the global features are not correlated. For example, when smoking in the driver's seat, there will be a behavioral feature of arm movement, that is, the arm is different from the normal driving state, but the ROI infrared image only involves the image of the mouth area, missing the arm feature, which will make it impossible to accurately judge some short cigarette butts.

[0107] Based on this, the present application provides an optional implementation method, which can combine the full-image infrared image features to assist the ROI infrared image to determine whether there is smoking or making phone calls in the driver's seat.

[0108] When the twin network is trained in the present application, the sample images input into the twin network to be trained are two ROI images and a global image. When the ROI image is a mouth area image, the corresponding global image can include the driver's arm area. When the ROI image is an eye area image, the corresponding global image can include the driver's arm area. This is because some drivers hold the phone to the eye area, that is, the temple area, when making a call. Therefore, based on the twin network training method provided in the present application, the trained twin network for user object behavior recognition can be used to determine whether the main driver's behavior involves smoking or making a phone call.

[0109] Similarly, the above-mentioned judgment method for whether the driver is smoking or making a phone call can also be used to judge whether a member in the vehicle is smoking or making a phone call. That is, it can also provide support for more OMS perception tasks.

[0110] Among them, DMS stands for Driver Monitoring System, which monitors the driver and can monitor the driver's fatigue and dangerous driving behavior. OMS stands for Occupancy Monitoring System, which monitors the passengers and can monitor the passengers' age, status, emotions and behavior.

[0111] Based on this, the present application can also provide a method for object behavior recognition in a vehicle cabin using a twin network trained by the present application, specifically including: obtaining a first image of a target object (such as a driver or a member) in the cabin, and determining a second image and a third object based on the first image; the second image is a local image corresponding to a first area in the first image, and the first area is an area related to a first behavior to be recognized of the target object; the third image is a local image corresponding to a second area in the first image, and the second area is an area related to a second behavior to be recognized of the target object; the first image, the second image, and the third image are input into the twin network to obtain the similarity probability between the first image feature of the first image, the second image feature of the second image, and the third image feature of the third image; based on the similarity probability, determining a behavior recognition result for the target object; wherein the twin network is a twin network trained based on the twin network training method provided in the present application.

[0112] Among them, based on the similarity probability, the recognition result of the behavior to be identified for the target object is determined, specifically including: when it is determined that the similarity probability is greater than a preset threshold, the behavior recognition result for the target object is determined to be the occurrence of the first behavior to be identified or the second behavior to be identified; when it is determined that the similarity probability is less than or equal to the preset threshold, the behavior recognition result for the target object is determined to be the non-occurrence of the first behavior to be identified or the second behavior to be identified.

[0113] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0114] Based on the same inventive concept, the embodiment of the present application also provides a twin network training device for implementing the twin network training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more twin network training devices provided below can be referred to the limitations of the twin network training method above, and will not be repeated here.

[0115] In an exemplary embodiment, Figure 6 As shown, a training device 300 for a twin network is provided, comprising: a sample acquisition module 81, a sample processing module 82 and a parameter adjustment module 83, wherein:

[0116] A sample acquisition module 81 is used to acquire multiple sample groups; each sample group includes a first sample image, a second sample image and a third sample image of the sample object; the second sample image is a local image of the first target area in the first sample image, and the third sample image is a local image of the second target area in the first sample image, the first target area is an area related to the first behavior to be identified, and the second target area is an area related to the second behavior to be identified; a sample processing module 82 is used to input the sample group into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image in the sample group based on the feature extraction network in the twin network to be trained; and input the first sample image feature, the second sample image feature and the third sample image feature into the classification network in the twin network to be trained, and the classification network is based on the first sample image feature, the second sample image feature and the third sample image feature. Image features, respectively obtain the first predicted behavior, the second predicted behavior and the third predicted behavior; determine the first loss value based on the distance similarity and cosine similarity between the second sample image features and the third sample image features and the first sample image features respectively; obtain the second loss value based on the difference between the first predicted behavior and the actual behavior corresponding to the sample group, obtain the third loss value based on the difference between the second predicted behavior and the actual behavior, and obtain the fourth loss value based on the difference between the third predicted behavior and the actual behavior; a parameter adjustment module 83 is used to determine the total loss value corresponding to each sample group based on the first loss value, the second loss value, the third loss value and the fourth loss value, and update the network parameters of the twin network to be trained when the total loss value does not meet the training stop condition, and train the twin network to be trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognition of user objects.

[0117] In one embodiment, when the sample processing module 82 is used to determine the first loss value based on the distance similarity and cosine similarity between the second sample image feature and the third sample image feature and the first sample image feature respectively, it is specifically used to: determine the first sample image feature similarity based on the product of the first distance similarity and the first cosine similarity between the first sample image feature and the second sample image feature; determine the second sample image feature similarity based on the product of the second distance similarity and the second cosine similarity between the first sample image feature and the third sample image feature; determine the total sample image feature similarity based on the sum of the first sample image feature similarity and the second sample image feature similarity; convert the total sample image feature similarity into a sample image feature similarity probability within a preset value range; determine the first loss value based on the sample image feature similarity probability, the actual behavior type value of the sample group and the preset loss weight.

[0118] In one embodiment, the sample processing module 82 is used to input the first sample image feature, the second sample image feature and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first prediction behavior, the second prediction behavior and the third prediction behavior respectively based on the first sample image feature, the second sample image feature and the third sample image feature. It is specifically used to: input the first sample image feature into the first classification network in the twin network to be trained to obtain the first prediction behavior of the sample object; input the second sample image feature into the second classification network in the twin network to be trained to obtain the second prediction behavior of the sample object; input the third sample image feature into the third classification network in the twin network to be trained to obtain the third prediction behavior of the sample object.

[0119] In one embodiment, the sample processing module 82 is used to input the first sample image feature, the second sample image feature and the third sample image feature into the classification network in the twin network to be trained, and the classification network obtains the first prediction behavior, the second prediction behavior and the third prediction behavior respectively based on the first sample image feature, the second sample image feature and the third sample image feature. It is specifically used to: input the first sample image feature into the first classification network in the twin network to be trained to obtain the first prediction behavior; input the second sample image feature into the second classification network in the twin network to be trained to obtain the second prediction behavior; input the third sample image feature into the third classification network in the twin network to be trained to obtain the third prediction behavior.

[0120] In one embodiment, the sample processing module 82 is used to input the sample group into the twin network to be trained, and obtain the first sample image feature corresponding to the first sample image in the sample group, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image based on the feature extraction network in the twin network to be trained. It is specifically used to: input the first sample image in the sample group into the first feature extraction network in the twin network to be trained to obtain the corresponding first sample image feature; input the second sample image in the sample group into the second feature extraction network in the twin network to be trained to obtain the corresponding second sample image feature; input the third sample image in the sample group into the third feature extraction network in the twin network to be trained to obtain the corresponding third sample image feature.

[0121] In one embodiment, any one of the first feature extraction network, the second feature extraction network, and the third feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network.

[0122] In one embodiment, the sample groups include a positive sample group and a negative sample group; the three sample images included in the positive sample group are images with behaviors to be identified; the three sample images included in the negative sample group are images without behaviors to be identified.

[0123] In one embodiment, when the sample acquisition module 81 is used to acquire multiple sample groups, it is specifically used to: acquire multiple sample images including sample objects, and determine the positive sample images to be selected and the negative sample images to be selected from the multiple sample images; for the same behavior to be identified, acquire the first similarity between the behavior in each of the positive sample images to be selected and the behavior to be identified, and filter the positive sample images from the multiple positive sample images to be selected based on the comparison result of the first similarity and the first similarity threshold; acquire the second similarity between the behavior in each of the negative sample images to be selected and the behavior to be identified for the same behavior, and filter the negative sample images from the multiple negative sample images to be selected based on the comparison result of the second similarity and the second similarity threshold; after completing the image screening, respectively acquire the second sample image corresponding to the first target area and the third sample image corresponding to the second target area from the filtered positive sample images and negative sample images; form a positive sample group based on the positive sample image and the corresponding second sample image and third sample image, and form a negative sample group based on the negative sample image and the corresponding second sample image and third sample image.

[0124] Each module in the above-mentioned twin network training device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a training method for a twin network is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0126] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the relevant steps of the training method of the twin network provided in the present application are implemented.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the relevant steps of the training method of the twin network provided in the present application are implemented.

[0129] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the relevant steps of the training method of the twin network provided in the present application.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations. It should also be added that the images of user objects involved in this application are also images authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant images must comply with relevant regulations.

[0131] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0132] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0133] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a twin network, characterized in that: The method comprises: Acquire multiple sample groups; each of the sample groups includes a first sample image, a second sample image, and a third sample image of a sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified; The sample group is input together into the twin network to be trained, and based on the feature extraction network in the twin network to be trained, a first sample image feature corresponding to the first sample image, a second sample image feature corresponding to the second sample image, and a third sample image feature corresponding to the third sample image in the sample group are obtained; and the first sample image feature, the second sample image feature, and the third sample image feature are respectively input into the classification network in the twin network to be trained to obtain a first prediction behavior, a second prediction behavior, and a third prediction behavior respectively; wherein the feature extraction network includes a first feature extraction network for receiving the first sample image and outputting the first sample image feature, a second feature extraction network for receiving the second sample image and outputting the second sample image feature, and a third feature extraction network for receiving the third sample image and outputting the third sample image feature; Determine the similarity of the first sample image feature based on the product of the first distance similarity and the first cosine similarity between the first sample image feature and the second sample image feature; determine the similarity of the second sample image feature based on the product of the second distance similarity and the second cosine similarity between the first sample image feature and the third sample image feature; determine the total sample image feature similarity based on the sum of the first sample image feature similarity and the second sample image feature similarity; determine the first loss value based on the total sample image feature similarity, the actual behavior type value of the sample group and the preset loss weight; obtain the second loss value based on the difference between the first predicted behavior and the actual behavior corresponding to the sample group, obtain the third loss value based on the difference between the second predicted behavior and the actual behavior, and obtain the fourth loss value based on the difference between the third predicted behavior and the actual behavior; wherein the cosine similarity is characterized based on the cosine value of the angle between two feature vectors; the feature vector includes feature vectors corresponding to the first sample image feature, the second sample image feature, and the third sample image feature, respectively; Based on the first loss value, the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to each of the sample groups is determined, and when the total loss value does not meet the condition for stopping training, the network parameters of the twin network to be trained are updated, and the twin network to be trained is trained again until the condition for stopping training is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognitions of user objects.

2. The method according to claim 1, characterized in that The first sample image feature, the second sample image feature and the third sample image feature are respectively input into the classification network in the twin network to be trained to obtain a first prediction behavior, a second prediction behavior and a third prediction behavior, respectively, including: Inputting the first sample image feature into a first classification network in the twin network to be trained to obtain the first predicted behavior of the sample object; Inputting the second sample image feature into the second classification network in the twin network to be trained to obtain the second predicted behavior of the sample object; The third sample image feature is input into the third classification network in the twin network to be trained to obtain the third predicted behavior of the sample object.

3. The method according to claim 1, characterized in that: The step of inputting the sample group into the twin network to be trained, and obtaining a first sample image feature corresponding to the first sample image in the sample group, a second sample image feature corresponding to the second sample image, and a third sample image feature corresponding to the third sample image based on a feature extraction network in the twin network to be trained, comprises: Inputting the first sample image in the sample group into the first feature extraction network in the twin network to be trained to obtain corresponding first sample image features; Inputting the second sample image in the sample group into the second feature extraction network in the twin network to be trained to obtain corresponding second sample image features; The third sample image in the sample group is input into the third feature extraction network in the twin network to be trained to obtain the corresponding third sample image feature.

4. The method according to claim 3, characterized in that The first feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network; The second feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network; The third feature extraction network is one of a MobileNet feature extraction network and a YOLO feature extraction network.

5. The method according to claim 1, characterized in that The sample groups include a positive sample group and a negative sample group; The three sample images included in the positive sample group are all images containing behaviors to be identified; The three sample images included in the negative sample group are all images without the behavior to be identified.

6. The method according to claim 5, characterized in that The obtaining of multiple sample groups comprises: Acquire a plurality of sample images including sample objects, and determine a positive sample image to be selected and a negative sample image to be selected from the plurality of sample images; For the same behavior to be identified, obtaining a first similarity between the behavior in each of the to-be-selected positive sample images and the behavior to be identified, and screening a positive sample image from the plurality of to-be-selected positive sample images according to a comparison result of the first similarity and a first similarity threshold; Obtaining a second similarity between the behavior in each of the negative sample images to be selected and the behavior to be identified for the same behavior, and screening a negative sample image from the plurality of negative sample images to be selected according to a comparison result between the second similarity and a second similarity threshold; After completing the image screening, the second sample image corresponding to the first target area and the third sample image corresponding to the second target area are respectively obtained from the screened positive sample images and the screened negative sample images; a positive sample group is formed based on the positive sample image and the corresponding second sample image and the third sample image, and a negative sample group is formed based on the negative sample image and the corresponding second sample image and the third sample image.

7. A training device for a twin network, characterized in that: The device comprises: A sample acquisition module, used to acquire multiple sample groups; each of the sample groups includes a first sample image, a second sample image, and a third sample image of a sample object; the second sample image is a local image of a first target area in the first sample image, the third sample image is a local image of a second target area in the first sample image, the first target area is an area related to a first behavior to be identified, and the second target area is an area related to a second behavior to be identified; A sample processing module is used to input the sample group into the twin network to be trained together, and obtain the first sample image feature corresponding to the first sample image in the sample group, the second sample image feature corresponding to the second sample image, and the third sample image feature corresponding to the third sample image based on the feature extraction network in the twin network to be trained; and input the first sample image feature, the second sample image feature, and the third sample image feature into the classification network in the twin network to be trained, respectively, to obtain the first prediction behavior, the second prediction behavior, and the third prediction behavior respectively; determine the first sample image feature similarity based on the product of the first distance similarity and the first cosine similarity between the first sample image feature and the second sample image feature; determine the second sample image feature similarity based on the product of the second distance similarity and the second cosine similarity between the first sample image feature and the third sample image feature; determine the total sample based on the sum of the first sample image feature similarity and the second sample image feature similarity. Image feature similarity; based on the total sample image feature similarity, the actual behavior type value of the sample group and the preset loss weight, determine the first loss value; based on the difference between the first predicted behavior and the actual behavior corresponding to the sample group, obtain the second loss value, based on the difference between the second predicted behavior and the actual behavior, obtain the third loss value, based on the difference between the third predicted behavior and the actual behavior, obtain the fourth loss value; wherein the cosine similarity is characterized based on the cosine value of the angle between two feature vectors; the feature vector includes feature vectors corresponding to the first sample image feature, the second sample image feature, and the third sample image feature respectively; the feature extraction network includes a first feature extraction network for receiving the first sample image and outputting the first sample image feature, a second feature extraction network for receiving the second sample image and outputting the second sample image feature, and a third feature extraction network for receiving the third sample image and outputting the third sample image feature; A parameter adjustment module is used to determine the total loss value corresponding to each of the sample groups based on the first loss value, the second loss value, the third loss value and the fourth loss value, and when the total loss value does not meet the training stop condition, update the network parameters of the twin network to be trained, and train the twin network to be trained again until the training stop condition is met, so as to obtain a trained twin network, and the trained twin network is applied to at least two behavior recognitions of user objects.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images