A correlation detection model training method and device, a storage medium and an electronic device

Abstract

The specification discloses a correlation detection model training method and device, a storage medium and an electronic device, wherein the method comprises: obtaining a training sample set, intercepting a human hand selection box of the training sample to obtain a sub-training sample, inputting the sub-training sample into a gesture recognition model, obtaining an interaction gesture probability value corresponding to the training sample, obtaining an identification loss function of the gesture recognition model, updating parameters of the gesture recognition model based on the identification loss function, until the model converges to obtain a trained gesture recognition model, inputting the training sample into a correlation detection model, obtaining a feature map of the training sample, fusing the feature map with a human face selection map and a human hand selection map of the training sample to obtain a fused feature map of the training sample, obtaining an interaction probability value corresponding to the training sample based on the fused feature map, obtaining a detection loss function of the correlation detection model, and updating parameters of the correlation detection model based on the detection loss function, until the model converges to obtain a trained correlation detection model.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium and electronic device for training an association detection model. Background Technology

[0002] Nowadays, with the popularization of electronic payment, people have gradually become accustomed to using electronic payment for shopping in their daily lives. In addition to scanning electronic information such as QR codes, electronic payment methods also include payment through biometric identification, such as facial recognition and fingerprint recognition. Summary of the Invention

[0003] This specification provides a method, apparatus, storage medium, and electronic device for training an association detection model. Based on the association detection of faces and hands, the interaction probability value corresponding to the training sample can be determined based on faces and hands of the same target, thereby improving the reliability and accuracy of the interaction probability value and thus improving the accuracy of correctly executing the interaction.

[0004] In a first aspect, embodiments of this application provide a method for training an association detection model, the method comprising:

[0005] Obtain a training sample set, which includes at least one training sample, wherein the training sample is image data that identifies face selection boxes, hand selection boxes, and gesture labels;

[0006] Extract the hand selection box of each training sample in the training sample set to obtain the sub-training sample corresponding to each training sample. Input each sub-training sample into the gesture recognition model. Based on the sub-training sample and the gesture label, obtain the interaction gesture probability value corresponding to the training sample.

[0007] Obtain the recognition loss function of the gesture recognition model, perform supervised training on the gesture recognition model based on the recognition loss function and the probability value of the interaction gesture, and iteratively update the model parameters of the gesture recognition model until the gesture recognition model converges, thus obtaining the trained gesture recognition model.

[0008] Each training sample is input into the association detection model to obtain a feature map corresponding to each training sample. The feature map is then fused with the face selection map and hand selection map corresponding to each training sample to obtain a fused feature map corresponding to each training sample. Based on the fused feature map, the interaction probability value corresponding to the training sample is obtained. The interaction probability value is used to indicate the interaction intention prediction result corresponding to the training sample. The face selection map and hand selection map are image data of the face and hand selected in the training sample, respectively.

[0009] Obtain the detection loss function of the association detection model, perform supervised training on the association detection model based on the detection loss function and the interaction probability value, and iteratively update the model parameters of the association detection model until the association detection model converges, thus obtaining the trained association detection model.

[0010] Secondly, embodiments of this application provide an interaction detection method, the method comprising:

[0011] Image data is collected, and the target face image and target hand image corresponding to the image data are obtained based on the pre-trained face detection model and hand detection model.

[0012] The target hand image is input into the trained gesture recognition model obtained by the above-mentioned association detection model training method, and the probability value of the interactive gesture corresponding to the target hand image is output.

[0013] A preset gesture probability value is obtained. When the interaction gesture probability value is greater than or equal to the preset gesture probability value, the target face image and the target hand image are input into the trained association detection model obtained by the above-mentioned association detection model training method to obtain the interaction probability value corresponding to the image data, and the interaction transaction is executed based on the interaction probability value.

[0014] Thirdly, embodiments of this application provide an association detection model training apparatus, comprising:

[0015] A sample acquisition unit is used to acquire a training sample set, the training sample set including at least one training sample, the training sample being image data that identifies face selection boxes, hand selection boxes and gesture labels;

[0016] The gesture recognition training unit is used to extract the human hand selection box of each training sample in the training sample set, obtain the sub-training sample corresponding to each training sample, input each sub-training sample into the gesture recognition model, and obtain the interaction gesture probability value corresponding to the training sample based on the sub-training sample and the gesture label.

[0017] The gesture recognition model is completed and the recognition loss function of the gesture recognition model is obtained. Based on the recognition loss function and the probability value of the interaction gesture, the gesture recognition model is supervised and the model parameters of the gesture recognition model are iteratively updated until the gesture recognition model converges, and the trained gesture recognition model is obtained.

[0018] The association detection training unit is used to input each training sample into the association detection model, obtain the feature map corresponding to each training sample, fuse the feature map with the face selection map and hand selection map corresponding to each training sample to obtain the fused feature map corresponding to each training sample, and obtain the interaction probability value corresponding to the training sample based on the fused feature map. The interaction probability value is used to indicate the interaction intention prediction result corresponding to the training sample. The face selection map and hand selection map are image data of the face and hand selected in the training sample, respectively.

[0019] The association detection model completion unit is used to obtain the detection loss function of the association detection model, perform supervised training on the association detection model based on the detection loss function and the interaction probability value, and iteratively update the model parameters of the association detection model until the association detection model converges, thus obtaining the trained association detection model.

[0020] Fourthly, embodiments of this application provide an interactive detection device, comprising:

[0021] The image acquisition unit is used to collect image data and obtain the target face image and target hand image corresponding to the image data based on the pre-trained face detection model and hand detection model.

[0022] The gesture recognition unit is used to input the target human hand image into the trained gesture recognition model obtained by the above-mentioned association detection model training method, and output the probability value of the interactive gesture corresponding to the target human hand image.

[0023] The association detection unit is used to obtain a preset gesture probability value. When the interaction gesture probability value is greater than or equal to the preset gesture probability value, the target face image and the target hand image are input into the trained association detection model obtained by the above-mentioned association detection model training method to obtain the interaction probability value corresponding to the image data, and the interaction transaction is executed based on the interaction probability value.

[0024] Fifthly, embodiments of this application provide a computer program product that stores at least one instruction, the at least one instruction being adapted to be loaded by a processor and executed in accordance with the above-described method steps.

[0025] Sixthly, embodiments of this application provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the method described above.

[0026] In a seventh aspect, embodiments of this application provide an electronic device, including: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method described above.

[0027] In this embodiment, by acquiring training samples, extracting hand selection boxes from the training samples to obtain sub-training samples, and obtaining corresponding interaction gesture probability values ​​based on the sub-training samples, the effectiveness of determining whether the interaction gestures of the training samples have interactive intent is achieved. The recognition loss function of the gesture recognition model is obtained, and the parameters of the gesture recognition model are updated based on the recognition loss function to obtain a converged and trained gesture recognition model. The training samples are input into the association detection model to obtain the interaction probability values ​​corresponding to the training samples, and the parameters of the association detection model are updated according to the detection loss function of the association detection model to obtain a converged and trained association detection model. This achieves association detection of faces and hands, and determines that the interaction probability values ​​corresponding to the training samples are obtained based on faces and hands of the same target, improving the reliability and accuracy of the interaction probability values, thereby improving the accuracy of correctly executing interactions. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A system architecture diagram of an association detection model training method provided in this application embodiment;

[0030] Figure 2 A flowchart illustrating a method for training an association detection model provided in an embodiment of this application;

[0031] Figure 3 A schematic diagram illustrating an example of a face selection box provided in an embodiment of this application;

[0032] Figure 4 A schematic diagram illustrating an example of a hand selection box provided in an embodiment of this application;

[0033] Figure 5 This is an example diagram illustrating how to obtain gesture recognition probability in an embodiment of this application.

[0034] Figure 6 A highlighted example diagram provided for an embodiment of this application;

[0035] Figure 7 A flowchart illustrating a method for training an association detection model provided in an embodiment of this application;

[0036] Figure 8 A flowchart illustrating an interactive detection method provided in an embodiment of this application;

[0037] Figure 9 This is a schematic diagram of the structure of an association detection model training device provided in an embodiment of this application;

[0038] Figure 10 This is a schematic diagram of the structure of a gesture recognition training unit provided in an embodiment of this application;

[0039] Figure 11 This is a schematic diagram of the structure of a gesture recognition model completion unit provided in an embodiment of this application;

[0040] Figure 12 This is a schematic diagram of the structure of a training unit for an association detection model provided in an embodiment of this application;

[0041] Figure 13 This is a schematic diagram of the structure of an association detection model completion unit provided in an embodiment of this application;

[0042] Figure 14 This is a schematic diagram of the structure of an interactive detection device provided in an embodiment of this application;

[0043] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0044] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] To make the features and advantages of this specification more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0046] In existing technologies, when completing payments through biometric identification, such as facial recognition, the collected images may include more than one face image, leading to false scans and other issues that affect users' ability to successfully complete facial recognition payments and reduce the actual practicality of the convenience brought by facial recognition payments.

[0047] Based on this, this application provides a method for training an association detection model. This method involves acquiring training samples, extracting hand selection boxes from the training samples to obtain sub-training samples, and obtaining corresponding interaction gesture probability values ​​based on the sub-training samples to determine whether the interaction gestures of the training samples have an interactive intent. The method then obtains the recognition loss function of the gesture recognition model and updates the parameters of the gesture recognition model based on the recognition loss function to obtain a converged and trained gesture recognition model. Finally, the training samples are input into the association detection model to obtain the interaction probability values ​​corresponding to the training samples. The parameters of the association detection model are then updated according to the detection loss function of the association detection model to obtain a converged and trained association detection model. This achieves association detection of faces and hands, determining that the interaction probability values ​​corresponding to the training samples are based on faces and hands of the same target, improving the reliability and accuracy of the interaction probability values, and thus improving the accuracy of correctly executing interactions.

[0048] Please see Figure 1 This application provides a system architecture diagram for training an association detection model. For example... Figure 1 As shown, the association detection model training method provided in this application embodiment can be applied to a terminal to realize the process of training an association detection model for a terminal application in the terminal. The system structure provided in this application embodiment mainly includes an association detection model training server 10 and a training sample acquisition device 20. The association detection model training server 10 can be a large integrated server used by an enterprise, or a microcomputer, such as a personal computer; the training sample acquisition device 20 can be a device equipped with a camera capable of acquiring image data, such as a camera on an access control system, or a smartphone equipped with a camera.

[0049] In this embodiment, the association detection model training server 10 obtains the training sample set sent by the training sample acquisition device 20, extracts the hand selection box from the training samples to obtain sub-training samples, inputs the sub-training samples into the gesture recognition model, obtains the interaction gesture probability value corresponding to the training samples, and calculates the recognition loss function of the gesture recognition model. Based on the recognition loss function, the trained gesture recognition model is determined. The face selection image and hand selection image corresponding to the training samples, as well as the training samples, are input into the association detection model to obtain the interaction probability value corresponding to the training samples. The detection loss function of the association detection model is calculated, and based on the detection loss function, the trained association model is determined.

[0050] In this embodiment, by acquiring training samples, extracting hand selection boxes from the training samples to obtain sub-training samples, and obtaining corresponding interaction gesture probability values ​​based on the sub-training samples, the effectiveness of determining whether the interaction gestures of the training samples have interactive intent is achieved. The recognition loss function of the gesture recognition model is obtained, and the parameters of the gesture recognition model are updated based on the recognition loss function to obtain a converged and trained gesture recognition model. The training samples are input into the association detection model to obtain the interaction probability values ​​corresponding to the training samples, and the parameters of the association detection model are updated according to the detection loss function of the association detection model to obtain a converged and trained association detection model. This achieves association detection of faces and hands, and determines that the interaction probability values ​​corresponding to the training samples are obtained based on faces and hands of the same target, improving the reliability and accuracy of the interaction probability values, thereby improving the accuracy of correctly executing interactions.

[0051] based on Figure 1 The system architecture shown below will be combined with... Figures 2-6 This paper provides a detailed description of the association detection model training method provided in the embodiments of this application.

[0052] Please see Figure 2 This is a flowchart illustrating a method for training an association detection model, as provided in this application embodiment. Figure 2 As shown, the method may include the following steps S102-S110.

[0053] S102, Obtain the training sample set;

[0054] In one embodiment, the training sample set can be a set of image data including at least one training sample, which is image data identifying face selection boxes, hand selection boxes, and gesture labels.

[0055] Furthermore, the face selection box can be a selection box used to mark the location of face regions in the training samples. A feasible marking method is to determine the coordinates of the upper left and lower right corners of the face region, and then determine the face selection box based on these two coordinates. For example, as... Figure 3 As shown, Figure 3 The top-left corner coordinates of the face in the training sample are (x1, y1), and the bottom-right corner coordinates are (x2, y2). A face selection box is built based on the coordinates.

[0056] Furthermore, the hand selection box can be used to mark the location of the hand region in the training samples. One feasible marking method is to determine the coordinates of the upper left and lower right corners of the hand region, and then determine the hand selection box based on these two coordinates. For example, ... Figure 4 As shown, Figure 4The top-left corner of the face in the training sample is (x3, y3), and the bottom-right corner is (x4, y4). A hand selection box is built based on the coordinates.

[0057] Furthermore, the gesture label can be used to mark whether the gesture displayed by the hand in the hand selection box is a preset interactive gesture. For example, "1" and "0" can be used for marking, with "1" used to mark the gesture as a preset interactive gesture and "0" used to mark the gesture as a non-preset interactive gesture.

[0058] Furthermore, interactive gestures can be gestures used to indicate an interactive intention. The specific gestures can be set according to the actual situation, such as "thumbs up".

[0059] It should be noted that the training samples may contain more than one face and / or hand. Therefore, a pre-trained recognition model can be used to identify the target face and hand in the training samples and obtain their corresponding coordinates. One feasible method for selecting the target face and hand is to choose the object occupying the largest area in the training samples as the target face and hand. The specific selection method can be set according to the actual situation.

[0060] S104, extract the hand selection box of each training sample in the training sample set to obtain the sub-training sample corresponding to each training sample, input each sub-training sample into the gesture recognition model, and obtain the interaction gesture probability value corresponding to the training sample based on the sub-training sample and the gesture label.

[0061] In one embodiment, after obtaining the hand selection box of each training sample in the training sample set, the hand selection box is cropped in the training sample and used as the sub-training sample corresponding to the training sample. The sub-training sample and the gesture label corresponding to the training sample are input into the gesture recognition model. Based on the sub-training sample and the gesture label corresponding to the training sample, the probability value of the corresponding interactive gesture in the training sample is obtained.

[0062] Furthermore, a feasible method for cropping is to crop the training samples, retaining only the area where the hand selection box is located, to obtain the sub-training samples corresponding to the training samples.

[0063] Furthermore, after receiving the input sub-training sample and the gesture label corresponding to the training sample, the gesture recognition model identifies the sub-training sample and obtains the probability value of the interactive gesture corresponding to the sub-training sample.

[0064] Furthermore, the interaction gesture probability value can be the probability value of the interaction gesture used to instruct the training sample to perform gesture recognition. It's understandable that since the judgment result of an interaction gesture is either yes or no, the interaction gesture probability value can be the probability value indicating that the gesture corresponding to the training sample is for interaction and the probability value indicating that no interaction is performed. The sum of these two probability values ​​is 1. For example, as... Figure 5 As shown, Figure 5 After the sub-training samples and gesture labels are input into the gesture recognition model, the gesture interaction probability values ​​P1 and P2 corresponding to the training samples are obtained. The sum of P1 and P2 is 1.

[0065] It should be noted that when training training samples based on the gesture recognition model, a feasible method is to extract a training sample and input it into the gesture recognition model. After obtaining the interaction gesture probability value corresponding to the training sample output by the gesture recognition model, another training sample that has not been used for gesture recognition is extracted from the training sample set and input into the gesture recognition model for recognition.

[0066] S106, obtain the recognition loss function of the gesture recognition model, perform supervised training on the gesture recognition model based on the recognition loss function and the probability value of the interaction gesture, and iteratively update the model parameters of the gesture recognition model until the gesture recognition model converges, and obtain the trained gesture recognition model.

[0067] In one embodiment, the recognition loss function of the gesture recognition model, the first number of sub-training samples input to the gesture recognition model, and the first preset training number are obtained. When the first number of samples is greater than or equal to the first preset training number, the gesture recognition model is supervised and trained based on the recognition loss function and the probability value of the interaction gesture, and the model parameters of the gesture recognition model are iteratively updated. When the gesture recognition model converges, the trained gesture recognition model is obtained.

[0068] Furthermore, the first sample number can be the number of sub-training samples input to the gesture recognition model. It is understandable that, since each sub-training sample corresponds to a training sample, the number of sub-training samples used for recognition is equal to the number of training sample data used for recognition.

[0069] Furthermore, the first preset training quantity can be a pre-set quantity value, such as 200. When the first sample quantity is greater than or equal to the first preset training quantity, the change analysis of the recognition loss function is performed based on the recognition loss function obtained after each gesture recognition. Then, the model parameters of the gesture recognition model are adjusted based on the change analysis so that the gesture recognition model tends to converge, and the converged gesture recognition model is obtained, which is the completed gesture recognition model.

[0070] Understandably, the recognition loss function of a gesture recognition model can be used to indicate the quantitative difference between the error caused during the training process of the gesture recognition model and the actual result. When the recognition loss function approaches a preset threshold, the gesture recognition model corresponding to the recognition loss function can be considered to have converged.

[0071] S108, input each of the training samples into the association detection model, obtain the feature map corresponding to each of the training samples, fuse the feature map with the face selection map and hand selection map corresponding to each of the training samples to obtain the fused feature map corresponding to each training sample, and obtain the interaction probability value corresponding to the training sample based on the fused feature map.

[0072] In one embodiment, each training sample is input into the feature extraction module of the association detection model to obtain the feature map corresponding to the training sample. A pre-trained recognition model is used to obtain the face selection map and hand selection map corresponding to the training sample. The face selection map and hand selection map are down-resolutiond to obtain the face blur map corresponding to the face selection map and the hand blur map corresponding to the hand selection map, which maintain the same resolution as the feature map. The face blur map, hand blur map and feature map are simultaneously input into the fusion module of the association detection model to obtain the fused feature map corresponding to the training sample. The fused feature map is input into the prediction module of the association detection model to obtain the interaction probability value corresponding to the fused feature map.

[0073] Furthermore, the association detection model can include a feature extraction module, a fusion module, and a prediction module. The feature extraction module and the prediction module can be two modules obtained by splitting an existing recognition network, and the fusion module can be a convolutional module used to fuse the feature map and the selection map.

[0074] Furthermore, the feature map can be an image obtained by extracting features from the training samples using the feature extraction module in the association detection model. It should be noted that since the feature extraction module is derived from a segmentation within a recognition network, and the segmentation position is determined by the feature resolution generated by the network, the feature map obtained after feature extraction has a reduced resolution compared to the training samples. For example, if the segmentation position is at one-quarter resolution, and the resolution of the training samples is 600*800, then the resolution of the feature map corresponding to those training samples will be 300*400.

[0075] Furthermore, the face selection image and hand selection image are the image data of faces and hands selected from the training samples, respectively. It should be noted that in order for the face and hand selection images to be fused with the feature map, their resolution must be the same as the feature map. This ensures that the three images have the same channel dimension, enabling fusion processing. Therefore, the face and hand selection images are down-resolution processed to obtain blurred face images corresponding to the face selection image and blurred hand images corresponding to the hand selection image, respectively, with the same resolution as the feature map. A feasible down-resolution processing method is to use nearest neighbor sampling to process the face and hand selection images.

[0076] Furthermore, to highlight the location of faces and hands in the training samples, a feasible method is to use, for example... Figure 6 The highlighted method, as shown, involves selecting a bounding box around a face or hand in the training sample, and then using color to distinguish the selected area from the unselected area. Figure 6 The selected area is highlighted in white, while the unselected area is highlighted in black.

[0077] Furthermore, the interaction probability value is used to indicate the prediction result of the interaction intention corresponding to the training sample. Similar to the interaction gesture probability value, the prediction result of the interaction intention is either yes or no. Therefore, the interaction probability value can be the probability value indicating the existence of the association between the face and the hand corresponding to the training sample and the probability value indicating the absence of the association. The sum of the two probability values ​​is 1.

[0078] Understandably, since the purpose of the association detection model is to determine the association between a face and a hand, the hand selection image used by the association detection model is not limited to the same image as the display content corresponding to the sub-training samples in the gesture recognition model. For example, the gesture recognized in the gesture recognition model is "thumbs up," but the gesture in the hand selection image of the association detection model is not limited to "thumbs up," it can also be "V sign," "fist," etc. By not limiting the display content corresponding to the hand selection image and the sub-training samples to the same image, the association detection model can complete the recognition training for multiple gestures during the training process, thus shortening the training time required for the association detection model.

[0079] It should be noted that when training training samples based on the association detection model, a feasible method is to input a training sample into the association detection model, and after obtaining the interaction probability value corresponding to the training sample output by the association detection model, input another training sample in the training sample set that has not undergone association detection into the association detection model for detection.

[0080] S110, obtain the detection loss function of the association detection model, perform supervised training on the association detection model based on the detection loss function and the interaction probability value, and iteratively update the model parameters of the association detection model until the association detection model converges, and obtain the trained association detection model.

[0081] In one embodiment, the detection loss function of the association detection model, the second number of training samples input to the association detection model, and the second preset training number are obtained. When the second number of samples is greater than or equal to the second preset training number, the association detection model is supervised and trained based on the detection loss function and the interaction probability value, and the model parameters of the association detection model are iteratively updated. When the association detection model converges, the trained association detection model is obtained.

[0082] Furthermore, the second sample number can be the number of training samples input into the association detection model.

[0083] Furthermore, the second preset training quantity can be a pre-set quantity value, such as 200. When the second sample quantity is greater than or equal to the second preset training quantity, the change analysis of the detection loss function is performed based on the detection loss function obtained after each association detection. Then, the model parameters of the association detection model are adjusted based on the change analysis so that the association detection model tends to converge, and the converged association detection model is obtained, that is, the trained association detection model is obtained.

[0084] Understandably, the detection loss function of the association detection model can be used to indicate the quantitative difference between the error caused during the training process of the association detection model and the actual result. When the detection loss function approaches the preset threshold, the association detection model corresponding to the detection loss function can be considered to have converged.

[0085] It should be noted that the training processes for the association detection model and the gesture recognition model can be carried out separately and do not conflict with each other.

[0086] In this embodiment, by acquiring training samples, extracting hand selection boxes from the training samples to obtain sub-training samples, and obtaining corresponding interaction gesture probability values ​​based on the sub-training samples, the effectiveness of determining whether the interaction gestures of the training samples have interactive intent is achieved. The recognition loss function of the gesture recognition model is obtained, and the parameters of the gesture recognition model are updated based on the recognition loss function to obtain a converged and trained gesture recognition model. The training samples are input into the association detection model to obtain the interaction probability values ​​corresponding to the training samples, and the parameters of the association detection model are updated according to the detection loss function of the association detection model to obtain a converged and trained association detection model. This achieves association detection of faces and hands, and determines that the interaction probability values ​​corresponding to the training samples are obtained based on faces and hands of the same target, improving the reliability and accuracy of the interaction probability values, thereby improving the accuracy of correctly executing interactions.

[0087] Please see Figure 7 This is a flowchart illustrating a method for training an association detection model, as provided in this application embodiment. Figure 7 As shown, the method may include the following steps S202-S218.

[0088] S202, Obtain the training sample set;

[0089] In one embodiment, the training sample set can be a set of image data including at least one training sample, which is image data identifying face selection boxes, hand selection boxes, and gesture labels.

[0090] Furthermore, the face selection box can be a selection box used to mark the location of face regions in the training samples. A feasible marking method is to determine the coordinates of the upper left and lower right corners of the face region, and then determine the face selection box based on these two coordinates. For example, as... Figure 3 As shown, Figure 3 The top-left corner coordinates of the face in the training sample are (x1, y1), and the bottom-right corner coordinates are (x2, y2). A face selection box is built based on the coordinates.

[0091] Furthermore, the hand selection box can be used to mark the location of the hand region in the training samples. One feasible marking method is to determine the coordinates of the upper left and lower right corners of the hand region, and then determine the hand selection box based on these two coordinates. For example, ... Figure 4 As shown, Figure 4 The top-left corner of the face in the training sample is (x3, y3), and the bottom-right corner is (x4, y4). A hand selection box is built based on the coordinates.

[0092] Furthermore, the gesture label can be used to mark whether the gesture displayed by the hand in the hand selection box is a preset interactive gesture. For example, "1" and "0" can be used for marking, with "1" used to mark the gesture as a preset interactive gesture and "0" used to mark the gesture as a non-preset interactive gesture.

[0093] Furthermore, interactive gestures can be gestures used to indicate an interactive intention. The specific gestures can be set according to the actual situation, such as "thumbs up".

[0094] It should be noted that the training samples may contain more than one face and / or hand. Therefore, a pre-trained recognition model can be used to identify the target face and hand in the training samples and obtain their corresponding coordinates. One feasible method for selecting the target face and hand is to choose the object occupying the largest area in the training samples as the target face and hand. The specific selection method can be set according to the actual situation.

[0095] S204, extract the hand selection box corresponding to each training sample in the training sample set, and use each hand selection box as a sub-training sample corresponding to each training sample.

[0096] In one embodiment, after obtaining the hand selection box of each training sample in the training sample set, the hand selection box is cropped in the training sample, and the cropped hand selection box is used as the sub-training sample corresponding to the training sample.

[0097] Furthermore, a feasible method for cropping is to crop the training samples, retaining only the area where the hand selection box is located, to obtain the sub-training samples corresponding to the training samples. It is understandable that since the sub-training samples obtained by cropping are the hand selection boxes corresponding to the target hand in the training samples, each sub-training sample corresponds to one training sample and its corresponding gesture label.

[0098] S206, Input the sub-training samples into the gesture recognition model, and obtain the interaction gesture probability value corresponding to each sub-training sample based on the sub-training sample and the gesture label corresponding to the sub-training sample;

[0099] In one embodiment, after obtaining the sub-training sample, the sub-training sample and the gesture label corresponding to the training sample are input into the gesture recognition model. Based on the sub-training sample and the gesture label corresponding to the training sample, the probability value of the corresponding interactive gesture in the training sample is obtained.

[0100] Furthermore, after receiving the input sub-training sample and the gesture label corresponding to the training sample, the gesture recognition model identifies the sub-training sample and obtains the probability value of the interactive gesture corresponding to the sub-training sample.

[0101] Furthermore, the interaction gesture probability value can be the recognition result used to indicate that the training sample is an interaction gesture, that is, the interaction gesture probability value that instructs the training sample to perform gesture recognition. It can be understood that since the judgment result of an interaction gesture is either yes or no, the interaction gesture probability value can be the probability value indicating that the gesture corresponding to the training sample is for interaction and the probability value indicating that no interaction is performed. The sum of the two probability values ​​is 1. For example, as... Figure 5 As shown, Figure 5 After the sub-training samples and gesture labels are input into the gesture recognition model, the gesture interaction probability values ​​P1 and P2 corresponding to the training samples are obtained. The sum of P1 and P2 is 1.

[0102] It should be noted that when training training samples based on the gesture recognition model, a feasible method is to extract a training sample and input it into the gesture recognition model. After obtaining the interaction gesture probability value corresponding to the training sample output by the gesture recognition model, another training sample that has not been used for gesture recognition is extracted from the training sample set and input into the gesture recognition model for recognition.

[0103] S208, obtain the recognition loss function of the gesture recognition model, perform supervised training on the gesture recognition model based on the recognition loss function and the probability value of the interaction gesture, and iteratively update the model parameters of the gesture recognition model until the gesture recognition model converges, and obtain the trained gesture recognition model;

[0104] In one embodiment, the recognition loss function of the gesture recognition model, the first number of sub-training samples input to the gesture recognition model, and the first preset training number are obtained. When the first number of samples is greater than or equal to the first preset training number, the gesture recognition model is supervised and trained based on the recognition loss function and the probability value of the interaction gesture, and the model parameters of the gesture recognition model are iteratively updated. When the gesture recognition model converges, the trained gesture recognition model is obtained.

[0105] Furthermore, the first sample number can be the number of sub-training samples input to the gesture recognition model. It is understandable that, since each sub-training sample corresponds to a training sample, the number of sub-training samples used for recognition is equal to the number of training sample data used for recognition.

[0106] Furthermore, the first preset training quantity can be a pre-set quantity value, such as 200. When the first sample quantity is greater than or equal to the first preset training quantity, the change analysis of the recognition loss function is performed based on the recognition loss function obtained after each gesture recognition. Then, the model parameters of the gesture recognition model are adjusted based on the change analysis so that the gesture recognition model tends to converge, and the converged gesture recognition model is obtained, which is the completed gesture recognition model.

[0107] Understandably, the recognition loss function of a gesture recognition model can be used to indicate the quantitative difference between the error caused during the training process of the gesture recognition model and the actual result. When the recognition loss function approaches a preset threshold, the gesture recognition model corresponding to the recognition loss function can be considered to have converged.

[0108] S210, each training sample is input into the feature extraction module of the association detection model to obtain the feature map corresponding to the training sample;

[0109] In one embodiment, each training sample is input into the feature extraction module in the association detection model to obtain the feature map corresponding to the training sample after the feature extraction module extracts features from the training sample.

[0110] Furthermore, the association detection model can include a feature extraction module, a fusion module, and a prediction module. The feature extraction module and the prediction module can be two modules obtained by splitting an existing recognition network, and the fusion module can be a convolutional module used to fuse the feature map and the selection map.

[0111] Furthermore, the feature map can be an image obtained by extracting features from the training samples using the feature extraction module in the association detection model. It should be noted that since the feature extraction module is derived from a segmentation within a recognition network, and the segmentation position is determined by the feature resolution generated by the network, the feature map obtained after feature extraction has a reduced resolution compared to the training samples. For example, if the segmentation position is at one-quarter resolution, and the resolution of the training samples is 600*800, then the resolution of the feature map corresponding to those training samples will be 300*400.

[0112] S212, obtain the face selection image and hand selection image corresponding to the training sample, and reduce the resolution of the face selection image and the hand selection image to obtain the face blur image corresponding to the face selection image and the hand blur image corresponding to the hand selection image;

[0113] In one embodiment, a pre-trained recognition model is used to obtain a face selection image and a hand selection image corresponding to the training samples. The face selection image and the hand selection image are then down-resolutiond to obtain a blurred face image corresponding to the face selection image and a blurred hand image corresponding to the hand selection image, which maintain the same resolution as the feature map.

[0114] Furthermore, the face selection image and hand selection image are the image data of faces and hands selected from the training samples, respectively. It should be noted that in order for the face and hand selection images to be fused with the feature map, their resolution must be the same as the feature map. This ensures that the three images have the same channel dimension, enabling fusion processing. Therefore, the face and hand selection images are down-resolution processed to obtain blurred face images corresponding to the face selection image and blurred hand images corresponding to the hand selection image, respectively, with the same resolution as the feature map. A feasible down-resolution processing method is to use nearest neighbor sampling to process the face and hand selection images.

[0115] Furthermore, to highlight the location of faces and hands in the training samples, a feasible method is to use, for example... Figure 6 The highlighted method, as shown, involves selecting a bounding box around a face or hand in the training sample, and then using color to distinguish the selected area from the unselected area. Figure 6 The selected area is highlighted in white, while the unselected area is highlighted in black.

[0116] S214, simultaneously input the blurred face image, the blurred hand image, and the feature map into the fusion module of the association detection model to obtain the fused feature map;

[0117] In one embodiment, by simultaneously inputting the blurred face image, blurred hand image, and feature map into the fusion module of the association detection model, the face region and hand region in the feature map are highlighted to obtain the fused feature map corresponding to the training sample.

[0118] It should be noted that when performing feature fusion on the blurred face image, blurred hand image, and feature map, the blurred face image and blurred hand image are obtained in advance by reducing the image resolution, which ensures that the channel dimensions of the blurred face image, blurred hand image, and feature map are the same, thus enabling feature fusion.

[0119] S216, The fused feature map is input into the prediction module of the association detection model to obtain the interaction probability value corresponding to the fused feature map;

[0120] In one embodiment, the obtained fused feature map is input into the prediction module in the association detection model to obtain the interaction probability value corresponding to the fused feature map after the prediction module detects the association between the face and the hand in the fused feature map.

[0121] Furthermore, the interaction probability value is used to indicate the prediction result of the interaction intention corresponding to the training sample. Similar to the interaction gesture probability value, the prediction result of the interaction intention is either yes or no. Therefore, the interaction probability value can be the probability value indicating the existence of the association between the face and the hand corresponding to the training sample and the probability value indicating the absence of the association. The sum of the two probability values ​​is 1.

[0122] Understandably, since the purpose of the association detection model is to determine the association between a face and a hand, the hand selection image used by the association detection model is not limited to the same image as the display content corresponding to the sub-training samples in the gesture recognition model. For example, the gesture recognized in the gesture recognition model is "thumbs up," but the gesture in the hand selection image of the association detection model is not limited to "thumbs up," it can also be "V sign," "fist," etc. By not limiting the display content corresponding to the hand selection image and the sub-training samples to the same image, the association detection model can complete the recognition training for multiple gestures during the training process, thus shortening the training time required for the association detection model.

[0123] It should be noted that when training training samples based on the association detection model, a feasible method is to input a training sample into the association detection model, and after obtaining the interaction probability value corresponding to the training sample output by the association detection model, input another training sample in the training sample set that has not undergone association detection into the association detection model for detection.

[0124] S218, obtain the detection loss function of the association detection model, perform supervised training on the association detection model based on the detection loss function and the interaction probability value, and iteratively update the model parameters of the association detection model until the association detection model converges, and obtain the trained association detection model.

[0125] In one embodiment, the detection loss function of the association detection model, the second number of training samples input to the association detection model, and the second preset training number are obtained. When the second number of samples is greater than or equal to the second preset training number, the association detection model is supervised and trained based on the detection loss function and the interaction probability value, and the model parameters of the association detection model are iteratively updated. When the association detection model converges, the trained association detection model is obtained.

[0126] Furthermore, the second sample number can be the number of training samples input into the association detection model.

[0127] Furthermore, the second preset training quantity can be a pre-set quantity value, such as 200. When the second sample quantity is greater than or equal to the second preset training quantity, the change analysis of the detection loss function is performed based on the detection loss function obtained after each association detection. Then, the model parameters of the association detection model are adjusted based on the change analysis so that the association detection model tends to converge, and the converged association detection model is obtained, that is, the trained association detection model is obtained.

[0128] Understandably, the detection loss function of the association detection model can be used to indicate the quantitative difference between the error caused during the training process of the association detection model and the actual result. When the detection loss function approaches the preset threshold, the association detection model corresponding to the detection loss function can be considered to have converged.

[0129] In this embodiment, by obtaining training samples, a sub-training sample is obtained by extracting the hand selection box from the training samples. Based on the sub-training sample, the corresponding interaction gesture probability value is obtained to determine whether the interaction gesture of the training sample has an interactive intent. The recognition loss function of the gesture recognition model is obtained, and the parameters of the gesture recognition model are updated based on the recognition loss function to obtain the gesture recognition model after convergence. The training samples are input into the association detection model, and the face and hand in the training samples are associated with each other using the feature extraction and fusion method to obtain the interaction probability value corresponding to the training sample. The parameters of the association detection model are updated according to the detection loss function of the association detection model to obtain the association detection model after convergence. This achieves the association detection of face and hand, and determines that the interaction probability value corresponding to the training sample is obtained based on the face and hand of the same target, improving the reliability and accuracy of the interaction probability value, thereby improving the accuracy of correctly executing the interaction.

[0130] Please see Figure 8 This is a flowchart illustrating a method for training an association detection model, as provided in this application embodiment. Figure 8 As shown, the method may include the following steps S302-S306.

[0131] S302, Collect image data, and obtain the target face image and target hand image corresponding to the image data based on the pre-trained face detection model and hand detection model;

[0132] In one embodiment, a camera on a terminal device is used to collect image data in real time. Each frame of image data is then detected using a pre-trained face detection model and a hand detection model to determine the target face and target hand in the image data, thereby obtaining the target face image and target hand image corresponding to the image data.

[0133] Furthermore, the target face image can be an image with labeled face selection boxes after the target face in the image data has been labeled. The face selection boxes can be selection boxes used to label the location of face regions in the training samples. One feasible labeling method is to determine the coordinates of the upper left and lower right corners of the face region, and then determine the face selection box based on these two coordinates. For example, as... Figure 3 As shown, Figure 3 The top-left corner coordinates of the face in the training sample are (x1, y1), and the bottom-right corner coordinates are (x2, y2). A face selection box is built based on the coordinates.

[0134] Furthermore, the target hand image can be an image with hand selection boxes marked after labeling the target hand in the image data. The hand selection boxes can be selection boxes used to mark the location of the hand region in the training samples. A feasible labeling method is to determine the coordinates of the upper left and lower right corners of the hand region, and then determine the hand selection box based on these coordinates. For example, as... Figure 4 As shown, Figure 4 The top-left corner of the face in the training sample is (x3, y3), and the bottom-right corner is (x4, y4). A hand selection box is built based on the coordinates.

[0135] It should be noted that image data may contain more than one face and / or hand. Therefore, a pre-trained recognition model can be used to identify the target face and hand in the training sample and obtain their corresponding coordinates. One feasible method for selecting the target face and hand is to choose the object occupying the largest area in the training sample as the target face and hand. The specific selection method can be set according to the actual situation.

[0136] S304, input the target human hand image into the trained gesture recognition model obtained by the above-mentioned association detection model training method, and output the interaction gesture probability value corresponding to the target human hand image;

[0137] In one embodiment, the target hand image is input into the trained gesture recognition model obtained by the association detection model training method of the first two embodiments, the gesture recognition of the target hand image is performed, and the probability value of the interactive gesture corresponding to the target hand image is output.

[0138] Furthermore, the interaction gesture probability value can be the interaction gesture probability value used to instruct the training sample to perform gesture recognition. It can be understood that since the judgment result of the interaction gesture is either yes or no, the interaction gesture probability value can be the probability value of instructing the corresponding gesture of the training sample to perform interaction and the probability value of instructing not to perform interaction. The sum of the two probability values ​​is 1.

[0139] S306, obtain a preset gesture probability value. When the interaction gesture probability value is greater than or equal to the preset gesture probability value, input the target face image and the target hand image into the trained association detection model obtained by the above-mentioned association detection model training method to obtain the interaction probability value corresponding to the image data, and execute the interaction transaction based on the interaction probability value.

[0140] In one embodiment, a preset gesture probability value is obtained. When the interaction gesture probability value corresponding to the target hand image in the image data is greater than or equal to the preset gesture probability value, it can be considered that the target hand in the image data indicates that interaction is required. Then, the target face image and the target hand image are further input into the association detection model trained by the above association detection model training method to obtain the interaction probability value corresponding to the image data. Based on the interaction probability value, it is determined that the face and hand in the image data are associated, and the interaction transaction is executed.

[0141] Furthermore, the specific execution method for obtaining the interaction probability value of image data using the association detection model can be found in step S108, which will not be elaborated here.

[0142] Furthermore, if the probability value of the interactive gesture is less than the preset probability value, the association detection of the image data will not be performed, and the next frame of image data in the collected image data will be selected for gesture recognition.

[0143] Furthermore, a preset interaction probability value is obtained. If the interaction probability value is less than the preset interaction probability value, the interaction probability value indicates that the correlation between the face and the hand in the image data is not strong, and no interaction transaction is performed.

[0144] Furthermore, interactive transactions can include transactions such as facial recognition payment.

[0145] It should be noted that the image data processing, recognition, and detection processes can be performed by a terminal device equipped with a camera, or by the terminal device equipped with a camera transmitting the acquired image data to a server that stores a trained gesture recognition model and an association detection model, and receiving the interaction gesture probability value and interaction probability value returned by the server.

[0146] In this embodiment, image data is acquired, and after target face and target face selection are performed on the image data, gesture recognition is carried out. When the obtained interaction gesture probability value indicates that interaction is to be performed, the target face and target hand are associated with each other. Based on the obtained interaction probability value, the interaction transaction is executed. This is so that the execution of the interaction transaction is determined based on the target hand and target face in the acquired image data. In this way, the probability of erroneous execution of the interaction transaction is reduced based on the association detection of the target face and target hand, thereby improving the security and reliability of the execution of the interaction transaction and ensuring the accuracy of the correct execution of the interaction transaction.

[0147] based on Figure 1 The system architecture shown below will be combined with... Figures 9-13 This application provides a detailed description of the association detection model training device provided in its embodiments. It should be noted that... Figures 9-13 The association detection model training device in this specification is used to execute the code described herein. Figures 2-7 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts relevant to the embodiments of this application. For specific technical details not disclosed, please refer to this specification. Figures 2-7 The example shown.

[0148] Please see Figure 9 This is a schematic diagram of the structure of an association detection model training device provided in an embodiment of this application. Figure 9 As shown, the association detection model training device 1 in this application embodiment may include: a sample acquisition unit 11, a gesture recognition training unit 12, a gesture recognition model completion unit 13, an association detection training unit 14, and an association detection model completion unit 15.

[0149] The sample acquisition unit 11 is used to acquire a training sample set, the training sample set including at least one training sample, the training sample being image data that identifies a face selection box, a hand selection box and a gesture label;

[0150] The gesture recognition training unit 12 is used to extract the human hand selection box of each training sample in the training sample set, obtain the sub-training sample corresponding to each training sample, input each sub-training sample into the gesture recognition model, and obtain the interaction gesture probability value corresponding to the training sample based on the sub-training sample and the gesture label.

[0151] The gesture recognition model completion unit 13 is used to obtain the recognition loss function of the gesture recognition model, perform supervised training on the gesture recognition model based on the recognition loss function and the probability value of the interaction gesture, and iteratively update the model parameters of the gesture recognition model until the gesture recognition model converges, thereby obtaining the trained gesture recognition model.

[0152] The association detection training unit 14 is used to input each of the training samples into the association detection model, obtain the feature map corresponding to each of the training samples, fuse the feature map with the face selection map and hand selection map corresponding to each of the training samples to obtain the fused feature map corresponding to each training sample, and obtain the interaction probability value corresponding to the training sample based on the fused feature map. The interaction probability value is used to indicate the interaction intention prediction result corresponding to the training sample. The face selection map and hand selection map are image data of the face and hand selected in the training sample, respectively.

[0153] The association detection model completion unit 15 is used to obtain the detection loss function of the association detection model, perform supervised training on the association detection model based on the detection loss function and the interaction probability value, and iteratively update the model parameters of the association detection model until the association detection model converges, thereby obtaining the trained association detection model.

[0154] Optional, such as Figure 10 As shown, the gesture recognition training unit 12 includes:

[0155] Sub-training sample acquisition sub-unit 121 is used to extract the hand selection box corresponding to each training sample in the training sample set, and use each hand selection box as a sub-training sample corresponding to each training sample.

[0156] The gesture probability value acquisition subunit 122 is used to input the sub-training sample into the gesture recognition model, and to acquire the interaction gesture probability value corresponding to each sub-training sample based on the sub-training sample and the gesture label corresponding to the sub-training sample. The interaction gesture probability value is used to indicate the recognition result of the training sample as an interaction gesture.

[0157] Optional, such as Figure 11 As shown, the gesture recognition model completion unit 13 includes:

[0158] The first quantity acquisition subunit 131 is used to acquire the recognition loss function of the gesture recognition model, the first sample quantity of the sub-training samples input to the gesture recognition model, and the first preset training quantity.

[0159] The parameter update subunit 132 is used to perform supervised training on the gesture recognition model and iteratively update the model parameters of the gesture recognition model based on the recognition loss function and the interaction gesture probability value when the first sample number is greater than or equal to the first preset training number.

[0160] The gesture recognition model completion subunit 133 is used to obtain the trained gesture recognition model when the gesture recognition model converges.

[0161] Optional, such as Figure 12 As shown, the association detection training unit 14 includes:

[0162] The feature map acquisition subunit 141 is used to input each training sample into the feature extraction module in the association detection model to obtain the feature map corresponding to the training sample.

[0163] The blur image acquisition subunit 142 is used to acquire the face selection image and hand selection image corresponding to the training sample, and reduce the resolution of the face selection image and the hand selection image to obtain the face blur image corresponding to the face selection image and the hand blur image corresponding to the hand selection image.

[0164] The fusion subunit 143 is used to simultaneously input the blurred face image, the blurred hand image, and the feature map into the fusion module of the association detection model to obtain a fused feature map.

[0165] The prediction subunit 144 is used to input the fused feature map into the prediction module in the association detection model to obtain the interaction probability value corresponding to the fused feature map.

[0166] Optional, such as Figure 13 As shown, the association detection model completion unit 15 includes:

[0167] The second quantity acquisition subunit 151 is used to acquire the detection loss function of the association detection model, the second quantity of the training samples input to the association detection model, and the second preset training quantity.

[0168] The parameter update subunit 152 is used to supervise the training of the association detection model and iteratively update the model parameters of the association detection model based on the detection loss function and the interaction probability value when the second sample number is greater than or equal to the second preset training number.

[0169] The association detection model completion subunit 153 is used to obtain the trained association detection model when the association detection model converges.

[0170] Optionally, the association detection model training device 1 further includes:

[0171] The hand selection image used in the association detection model and the display content corresponding to the sub-training samples in the gesture recognition model are not limited to the same image.

[0172] In this embodiment, by acquiring training samples, a sub-training sample is obtained by cropping the hand selection box in the training samples. Based on the sub-training sample, the corresponding interaction gesture probability value is obtained to determine whether the interaction gesture of the training sample has an interaction intention. The recognition loss function of the gesture recognition model is obtained, and the parameters of the gesture recognition model are updated based on the recognition loss function to obtain the gesture recognition model after convergence. The training samples are input into the association detection model, and the face and hand in the training samples are associated with each other using the feature extraction and fusion method to obtain the interaction probability value corresponding to the training sample. The parameters of the association detection model are updated according to the detection loss function of the association detection model to obtain the association detection model after convergence. This achieves the association detection of face and hand, and determines that the interaction probability value corresponding to the training sample is obtained based on the face and hand of the same target, improving the reliability and accuracy of the interaction probability value, thereby improving the accuracy of correctly executing the interaction.

[0173] based on Figure 1 The system architecture shown below will be combined with... Figure 14 This application provides a detailed description of the interactive detection device provided in its embodiments. It should be noted that... Figure 14 The interactive detection device in this specification is used to execute the interactive detection device in this specification. Figure 8 The methods shown in the embodiments are for illustrative purposes only, illustrating the parts relevant to the embodiments of this application. For specific technical details not disclosed, please refer to this application. Figure 8 The example shown.

[0174] Please see Figure 14 The diagram below illustrates the structure of an interactive detection device according to an embodiment of this application. Figure 14 As shown, the interaction detection device 2 in this embodiment may include: an image acquisition unit 21, a gesture recognition unit 22, and an association detection unit 23.

[0175] Image acquisition unit 21 is used to acquire image data and obtain the target face image and target hand image corresponding to the image data based on the pre-trained face detection model and hand detection model.

[0176] The gesture recognition unit 22 is used to input the target human hand image into the trained gesture recognition model obtained by the above-mentioned association detection model training method, and output the probability value of the interactive gesture corresponding to the target human hand image;

[0177] The association detection unit 23 is used to obtain a preset gesture probability value. When the interaction gesture probability value is greater than or equal to the preset gesture probability value, the target face image and the target hand image are input into the trained association detection model obtained by the above-mentioned association detection model training method to obtain the interaction probability value corresponding to the image data, and the interaction transaction is executed based on the interaction probability value.

[0178] In this embodiment, image data is acquired, and after target face and target face selection are performed on the image data, gesture recognition is carried out. When the obtained interaction gesture probability value indicates that interaction is to be performed, the target face and target hand are associated with each other. Based on the obtained interaction probability value, the interaction transaction is executed. This is so that the execution of the interaction transaction is determined based on the target hand and target face in the acquired image data. In this way, the probability of erroneous execution of the interaction transaction is reduced based on the association detection of the target face and target hand, thereby improving the security and reliability of the execution of the interaction transaction and ensuring the accuracy of the correct execution of the interaction transaction.

[0179] This application also provides a computer storage medium that can store multiple program instructions, which are adapted to be loaded and executed by a processor as described above. Figures 1-8 The method steps of the illustrated embodiment can be found in the following documentation for detailed execution. Figures 1-8 The specific details of the illustrated embodiments will not be elaborated here.

[0180] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-8 The specific execution process of the association detection model training method described in the illustrated embodiment can be found in [reference needed]. Figures 1-8 The specific details of the illustrated embodiments will not be elaborated here.

[0181] Please see Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 15As shown, the electronic device 1000 may include: at least one processor 1001, such as a CPU; at least one network interface 1004; an input / output interface 1003; a memory 1005; and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk drive. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 15 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, an input / output interface module, and an application program for training correlation detection models.

[0182] exist Figure 15 In the electronic device 1000 shown, the input / output interface 1003 is mainly used to provide an input interface for the user and to obtain the user input data.

[0183] In one embodiment, processor 1001 can be used to invoke the association detection model training application stored in memory 1005, and specifically perform the following operations:

[0184] Obtain a training sample set, which includes at least one training sample, wherein the training sample is image data that identifies face selection boxes, hand selection boxes, and gesture labels;

[0185] Extract the hand selection box of each training sample in the training sample set to obtain the sub-training sample corresponding to each training sample. Input each sub-training sample into the gesture recognition model. Based on the sub-training sample and the gesture label, obtain the interaction gesture probability value corresponding to the training sample.

[0186] Obtain the recognition loss function of the gesture recognition model, perform supervised training on the gesture recognition model based on the recognition loss function and the probability value of the interaction gesture, and iteratively update the model parameters of the gesture recognition model until the gesture recognition model converges, thus obtaining the trained gesture recognition model.

[0187] Each training sample is input into the association detection model to obtain a feature map corresponding to each training sample. The feature map is then fused with the face selection map and hand selection map corresponding to each training sample to obtain a fused feature map corresponding to each training sample. Based on the fused feature map, the interaction probability value corresponding to the training sample is obtained. The interaction probability value is used to indicate the interaction intention prediction result corresponding to the training sample. The face selection map and hand selection map are image data of the face and hand selected in the training sample, respectively.

[0188] Obtain the detection loss function of the association detection model, perform supervised training on the association detection model based on the detection loss function and the interaction probability value, and iteratively update the model parameters of the association detection model until the association detection model converges, thus obtaining the trained association detection model.

[0189] Optionally, when the processor 1001 extracts the hand selection box of each training sample in the training sample set to obtain a sub-training sample corresponding to each training sample, inputs each sub-training sample into the gesture recognition model, and obtains the interaction gesture probability value corresponding to the training sample based on the sub-training sample and the gesture label, it performs the following operations:

[0190] Extract the hand selection box corresponding to each training sample in the training sample set, and use each hand selection box as a sub-training sample corresponding to each training sample.

[0191] The sub-training samples are input into the gesture recognition model. Based on the sub-training samples and the gesture labels corresponding to the sub-training samples, the probability value of the interactive gestures corresponding to each sub-training sample is obtained. The probability value of the interactive gestures is used to indicate the recognition result of the training sample as an interactive gesture.

[0192] Optionally, when the processor 1001 executes the following steps: obtaining the recognition loss function of the gesture recognition model, performing supervised training on the gesture recognition model based on the recognition loss function and the interaction gesture probability value, and iteratively updating the model parameters of the gesture recognition model until the gesture recognition model converges and a trained gesture recognition model is obtained:

[0193] Obtain the recognition loss function of the gesture recognition model, the first number of sub-training samples input to the gesture recognition model, and the first preset training number;

[0194] When the number of the first samples is greater than or equal to the first preset training number, the gesture recognition model is trained under supervision based on the recognition loss function and the interaction gesture probability value, and the model parameters of the gesture recognition model are iteratively updated.

[0195] When the gesture recognition model converges, the trained gesture recognition model is obtained.

[0196] Optionally, when the processor 1001 executes the process of inputting each training sample into the association detection model, obtaining the feature map corresponding to each training sample, fusing the feature map with the face selection map and hand selection map corresponding to each training sample to obtain the fused feature map corresponding to each training sample, and obtaining the interaction probability value corresponding to the training sample based on the fused feature map, the processor 1001 specifically performs the following operations:

[0197] Each training sample is input into the feature extraction module of the association detection model to obtain the feature map corresponding to the training sample;

[0198] Obtain the face selection image and hand selection image corresponding to the training sample, and reduce the resolution of the face selection image and hand selection image to obtain the face blur image corresponding to the face selection image and the hand blur image corresponding to the hand selection image.

[0199] The blurred face image, the blurred hand image, and the feature map are simultaneously input into the fusion module of the association detection model to obtain a fused feature map.

[0200] The fused feature map is input into the prediction module of the association detection model to obtain the interaction probability value corresponding to the fused feature map.

[0201] Optionally, when the processor 1001 executes the following operations: obtaining the detection loss function of the association detection model, performing supervised training on the association detection model based on the detection loss function and the interaction probability value, and iteratively updating the model parameters of the association detection model until the association detection model converges and a trained association detection model is obtained:

[0202] Obtain the detection loss function of the association detection model, the second number of training samples input to the association detection model, and the second preset training number;

[0203] When the number of the second samples is greater than or equal to the second preset training number, the association detection model is trained under supervision based on the detection loss function and the interaction probability value, and the model parameters of the association detection model are iteratively updated.

[0204] When the association detection model converges, the trained association detection model is obtained.

[0205] Optionally, the processor 1001 also performs the following operations:

[0206] The hand selection image used in the association detection model and the display content corresponding to the sub-training samples in the gesture recognition model are not limited to the same image.

[0207] In this embodiment, by acquiring training samples, a sub-training sample is obtained by cropping the hand selection box in the training samples. Based on the sub-training sample, the corresponding interaction gesture probability value is obtained to determine whether the interaction gesture of the training sample has an interaction intention. The recognition loss function of the gesture recognition model is obtained, and the parameters of the gesture recognition model are updated based on the recognition loss function to obtain the gesture recognition model after convergence. The training samples are input into the association detection model, and the face and hand in the training samples are associated with each other using the feature extraction and fusion method to obtain the interaction probability value corresponding to the training sample. The parameters of the association detection model are updated according to the detection loss function of the association detection model to obtain the association detection model after convergence. This achieves the association detection of face and hand, and determines that the interaction probability value corresponding to the training sample is obtained based on the face and hand of the same target, improving the reliability and accuracy of the interaction probability value, thereby improving the accuracy of correctly executing the interaction.

[0208] Please see Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 16 As shown, the electronic device 2000 may include: at least one processor 2001, such as a CPU; at least one network interface 2004; an input / output interface 2003; a memory 2005; and at least one communication bus 2002. The communication bus 2002 is used to enable communication between these components. The network interface 2004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 2005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 2005 may also be at least one storage device located remotely from the aforementioned processor 2001. Figure 16 As shown, the memory 2005, which is a computer storage medium, may include an operating system, a network communication module, an input / output interface module, and an interactive detection application.

[0209] exist Figure 16 In the electronic device 2000 shown, the input / output interface 2003 is mainly used to provide an input interface for the user and to obtain the user input data.

[0210] In one embodiment, the processor 2001 can be used to invoke the interactive detection application stored in the memory 2005, and specifically perform the following operations:

[0211] Image data is collected, and the target face image and target hand image corresponding to the image data are obtained based on the pre-trained face detection model and hand detection model.

[0212] The target hand image is input into the trained gesture recognition model obtained by using the association detection model training method, and the probability value of the interactive gesture corresponding to the target hand image is output.

[0213] A preset gesture probability value is obtained. When the interaction gesture probability value is greater than or equal to the preset gesture probability value, the target face image and the target hand image are input into the trained association detection model obtained by the association detection model training method to obtain the interaction probability value corresponding to the image data, and the interaction transaction is executed based on the interaction probability value.

[0214] In this embodiment, image data is acquired, and after target face and target face selection are performed on the image data, gesture recognition is carried out. When the obtained interaction gesture probability value indicates that interaction is to be performed, the target face and target hand are associated with each other. Based on the obtained interaction probability value, the interaction transaction is executed. This is so that the execution of the interaction transaction is determined based on the target hand and target face in the acquired image data. In this way, the probability of erroneous execution of the interaction transaction is reduced based on the association detection of the target face and target hand, thereby improving the security and reliability of the execution of the interaction transaction and ensuring the accuracy of the correct execution of the interaction transaction.

[0215] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0216] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. A method for training an association detection model, the method comprising: Obtain a training sample set, which includes at least one training sample, wherein the training sample is image data that identifies face selection boxes, hand selection boxes, and gesture labels; Extract the hand selection box of each training sample in the training sample set to obtain the sub-training sample corresponding to each training sample. Input each sub-training sample into the gesture recognition model. Based on the sub-training sample and the gesture label, obtain the interaction gesture probability value corresponding to the training sample. Obtain the recognition loss function of the gesture recognition model, perform supervised training on the gesture recognition model based on the recognition loss function and the probability value of the interaction gesture, and iteratively update the model parameters of the gesture recognition model until the gesture recognition model converges, thus obtaining the trained gesture recognition model. Each training sample is input into the association detection model to obtain a feature map corresponding to each training sample. The feature map is then fused with the face selection map and hand selection map corresponding to each training sample to obtain a fused feature map corresponding to each training sample. Based on the fused feature map, the interaction probability value corresponding to the training sample is obtained. The interaction probability value is used to indicate the interaction intention prediction result corresponding to the training sample. The face selection map and hand selection map are image data of the face and hand selected in the training sample, respectively. Obtain the detection loss function of the association detection model, perform supervised training on the association detection model based on the detection loss function and the interaction probability value, and iteratively update the model parameters of the association detection model until the association detection model converges, thus obtaining the trained association detection model.

2. The method according to claim 1, wherein the step of extracting the hand selection box of each training sample in the training sample set to obtain a sub-training sample corresponding to each training sample, inputting each sub-training sample into the gesture recognition model, and obtaining the interaction gesture probability value corresponding to the training sample based on the sub-training sample and the gesture label, includes: Extract the hand selection box corresponding to each training sample in the training sample set, and use each hand selection box as a sub-training sample corresponding to each training sample. The sub-training samples are input into the gesture recognition model. Based on the sub-training samples and the gesture labels corresponding to the sub-training samples, the probability value of the interactive gestures corresponding to each sub-training sample is obtained. The probability value of the interactive gestures is used to indicate the recognition result of the training sample as an interactive gesture.

3. The method according to claim 1, wherein obtaining the recognition loss function of the gesture recognition model, performing supervised training on the gesture recognition model based on the recognition loss function and the interaction gesture probability value, and iteratively updating the model parameters of the gesture recognition model until the gesture recognition model converges, thereby obtaining a trained gesture recognition model, comprises: Obtain the recognition loss function of the gesture recognition model, the first number of sub-training samples input to the gesture recognition model, and the first preset training number; When the number of the first samples is greater than or equal to the first preset training number, the gesture recognition model is trained under supervision based on the recognition loss function and the interaction gesture probability value, and the model parameters of the gesture recognition model are iteratively updated. When the gesture recognition model converges, the trained gesture recognition model is obtained.

4. The method according to claim 1, wherein inputting each training sample into an association detection model to obtain a feature map corresponding to each training sample, fusing the feature map with the face selection map and hand selection map corresponding to each training sample to obtain a fused feature map corresponding to each training sample, and obtaining the interaction probability value corresponding to the training sample based on the fused feature map, includes: Each training sample is input into the feature extraction module of the association detection model to obtain the feature map corresponding to the training sample; Obtain the face selection image and hand selection image corresponding to the training sample, and reduce the resolution of the face selection image and hand selection image to obtain the face blur image corresponding to the face selection image and the hand blur image corresponding to the hand selection image. The blurred face image, the blurred hand image, and the feature map are simultaneously input into the fusion module of the association detection model to obtain a fused feature map. The fused feature map is input into the prediction module of the association detection model to obtain the interaction probability value corresponding to the fused feature map.

5. The method according to claim 1, wherein obtaining the detection loss function of the association detection model, performing supervised training on the association detection model based on the detection loss function and the interaction probability value, and iteratively updating the model parameters of the association detection model until the association detection model converges, thereby obtaining a trained association detection model, comprises: Obtain the detection loss function of the association detection model, the second number of training samples input to the association detection model, and the second preset training number; When the number of the second samples is greater than or equal to the second preset training number, the association detection model is trained under supervision based on the detection loss function and the interaction probability value, and the model parameters of the association detection model are iteratively updated. When the association detection model converges, the trained association detection model is obtained.

6. The method according to claim 1, further comprising: The hand selection image used in the association detection model and the display content corresponding to the sub-training samples in the gesture recognition model are not limited to the same image.

7. An interaction detection method, the method comprising: Image data is collected, and the target face image and target hand image corresponding to the image data are obtained based on the pre-trained face detection model and hand detection model. The target human hand image is input into the trained gesture recognition model obtained by the association detection model training method as described in any one of claims 1 to 6, and the probability value of the interactive gesture corresponding to the target human hand image is output. Obtain a preset gesture probability value. When the interaction gesture probability value is greater than or equal to the preset gesture probability value, input the target face image and the target hand image into the trained association detection model obtained by the association detection model training method as described in any one of claims 1 to 6 to obtain the interaction probability value corresponding to the image data, and execute the interaction transaction based on the interaction probability value.

8. A training device for an association detection model, comprising: A sample acquisition unit is used to acquire a training sample set, the training sample set including at least one training sample, the training sample being image data that identifies face selection boxes, hand selection boxes and gesture labels; The gesture recognition training unit is used to extract the human hand selection box of each training sample in the training sample set, obtain the sub-training sample corresponding to each training sample, input each sub-training sample into the gesture recognition model, and obtain the interaction gesture probability value corresponding to the training sample based on the sub-training sample and the gesture label. The gesture recognition model completion unit is used to obtain the recognition loss function of the gesture recognition model, perform supervised training on the gesture recognition model based on the recognition loss function and the probability value of the interaction gesture, and iteratively update the model parameters of the gesture recognition model until the gesture recognition model converges, thereby obtaining the trained gesture recognition model. The association detection training unit is used to input each training sample into the association detection model, obtain the feature map corresponding to each training sample, fuse the feature map with the face selection map and hand selection map corresponding to each training sample to obtain the fused feature map corresponding to each training sample, and obtain the interaction probability value corresponding to the training sample based on the fused feature map. The interaction probability value is used to indicate the interaction intention prediction result corresponding to the training sample. The face selection map and hand selection map are image data of the face and hand selected in the training sample, respectively. The association detection model completion unit is used to obtain the detection loss function of the association detection model, perform supervised training on the association detection model based on the detection loss function and the interaction probability value, and iteratively update the model parameters of the association detection model until the association detection model converges, thus obtaining the trained association detection model.

9. An interactive detection device, comprising: The image acquisition unit is used to collect image data and obtain the target face image and target hand image corresponding to the image data based on the pre-trained face detection model and hand detection model. A gesture recognition unit is used to input the target human hand image into a trained gesture recognition model obtained by the association detection model training method as described in any one of claims 1 to 6, and output the probability value of the interactive gesture corresponding to the target human hand image. An association detection unit is used to obtain a preset gesture probability value. When the interaction gesture probability value is greater than or equal to the preset gesture probability value, the target face image and the target hand image are input into the association detection model that has been trained using the association detection model training method as described in any one of claims 1 to 6 to obtain the interaction probability value corresponding to the image data, and the interaction transaction is executed based on the interaction probability value.

10. A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the method as claimed in any one of claims 1 to 7.

11. An electronic device, comprising: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as claimed in any one of claims 1 to 7.

12. A computer program product having at least one instruction stored thereon, wherein the at least one instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images