Valet behavior detection method and device, equipment, storage medium and program product

Abstract

The invention provides a valet behavior detection method and device, equipment, a storage medium and a program product, and relates to the field of financial science and technology or other related fields. Comprising the following steps: acquiring to-be-detected video data; performing personnel identification on the to-be-detected video data to identify all to-be-detected personnel; performing classification detection processing on the to-be-detected personnel to determine internal personnel and non-internal personnel; performing first tracking detection on the insider to determine whether the insider has a valet intention; if it is determined that the insider has the valet intention, performing second tracking detection on the insider to determine whether the insider has a first valet behavior; and if it is determined that the internal personnel have the first valet behavior, first prompt information is sent to a first terminal device to prompt the detection personnel to have the valet behavior, and / or second prompt information is sent to a second terminal device to prompt the internal personnel to detect the valet behavior. According to the method provided by the invention, the accuracy of valet behavior detection is improved.

Description

Technical Field

[0001] The present application relates to financial technology or other related fields, and in particular to a method, device, equipment, storage medium and program product for detecting customer behavior. Background Art

[0002] Agent behavior is a type of agency behavior, which refers to the behavior of one person or entity performing specific operations on behalf of another person or entity. In the financial industry, agent behavior requires clear authorization to ensure that the agent can legally act on behalf of the client and protect the client's rights and interests. However, in actual operations, some agent behaviors are not authorized. These agent behaviors are illegal operations and have certain problems. Therefore, agent behaviors need to be detected and managed.

[0003] In the prior art, a neural network algorithm is used to identify people, obtain internal personnel and non-internal personnel, and target tracking is performed on the identified internal personnel and non-internal personnel; if the distance between the internal personnel and the non-internal personnel is determined to be within a preset distance range, and a mobile phone is identified in the rectangular frame of the internal personnel, it is determined that the valet behavior has occurred. However, the prior art only identifies the valet behavior operated by the mobile phone, and the application scenarios are limited. In addition, the algorithm accuracy is insufficient, the accuracy of the judgment standard is insufficient, and misjudgment is prone to occur.

[0004] Based on this, the existing methods for detecting valet behavior have the problem of low accuracy in detecting valet behavior. Summary of the invention

[0005] The present application provides a method, device, equipment, storage medium and program product for detecting valet behavior, which are used to solve the technical problem of low accuracy of valet behavior detection in the prior art.

[0006] In a first aspect, the present application provides a method for detecting valet behavior, comprising:

[0007] Obtain the video data to be detected;

[0008] Perform personnel recognition on the video data to be detected to identify all persons to be detected;

[0009] Classify the inspectors to identify internal and non-internal personnel;

[0010] Conduct a first tracking test on the insider to determine whether the insider has the intention of acting as a proxy;

[0011] If it is determined that the insider has the intention to act as a proxy, a second tracking test is performed on the insider to determine whether the insider has the first proxy behavior;

[0012] If it is determined that the internal person has performed the first valet behavior, a first prompt message is sent to the first terminal device to prompt the detection person that the valet behavior has occurred, and / or a second prompt message is sent to the second terminal device to prompt the internal person that the valet behavior has been detected.

[0013] In a second aspect, the present application provides a valet behavior detection device, comprising:

[0014] A first acquisition module, used to acquire video data to be detected;

[0015] The recognition module is used to perform personnel recognition on the video data to be detected, so as to identify all persons to be detected;

[0016] The classification module is used to classify the personnel to be tested and to determine whether they are internal personnel or non-internal personnel;

[0017] The first detection module is used to perform a first tracking detection on the internal personnel to determine whether the internal personnel has the intention of acting as a proxy for the customer;

[0018] The second detection module is used for performing a second tracking detection on the insider if it is determined that the insider has the intention of acting as a proxy, so as to determine whether the insider has the first proxy behavior;

[0019] The first processing module is used to send a first prompt message to the first terminal device to prompt the detection personnel that the valet behavior has occurred if it is determined that the internal personnel has performed the first valet behavior, and / or send a second prompt message to the second terminal device to prompt the internal personnel that the valet behavior has been detected.

[0020] In a third aspect, the present application provides a valet behavior detection device, including:

[0021] A processor, and a memory communicatively connected to the processor;

[0022] Memory stores computer-executable instructions;

[0023] The processor executes the computer-executable instructions stored in the memory to implement the first aspect or various possible implementations of the first aspect as described above.

[0024] In a fourth aspect, the present application proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above first aspect or various possible implementation methods of the first aspect.

[0025] In a fifth aspect, the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect or various possible implementations of the first aspect.

[0026] The present application provides a method, device, equipment, storage medium and program product for detecting valet behavior. By performing personnel identification and classification detection processing on the acquired video data to be detected, internal personnel and non-internal personnel are determined; a first tracking detection is performed on the internal personnel to determine whether the internal personnel has the intention of valet; if it is determined that the internal personnel has the intention of valet, a second tracking detection is performed on the internal personnel to determine whether the internal personnel has the behavior of valet. Compared with the prior art, which only determines the valet behavior by the distance between the internal personnel and the non-internal personnel being less than the preset distance threshold, and the mobile phone appears in the detection frame of the internal personnel, the present application performs more accurate valet behavior detection after determining that the internal personnel has the intention of valet through two tracking monitorings, reduces the possibility of misjudgment, and improves the accuracy of valet behavior detection. And after determining that the internal personnel has the behavior of valet, prompt information is sent to the first terminal device and / or the second terminal device respectively to realize the prompt of the detection personnel and / or the internal personnel; it is conducive to timely termination of valet behavior and improves the safety of operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0028] Figure 1 An architecture diagram of a valet behavior detection system provided for this application;

[0029] Figure 2 A schematic diagram of a method for detecting customer behavior provided in this application Figure 1 ;

[0030] Figure 3 A schematic diagram of a method for detecting customer behavior provided in this application Figure 2 ;

[0031] Figure 4 A schematic diagram of a method for detecting customer behavior provided in this application Figure 3 ;

[0032] Figure 5 A schematic diagram of a method for detecting customer behavior provided in this application Figure 4 ;

[0033] Figure 6 A schematic diagram of a method for detecting customer behavior provided in this application Figure 5 ;

[0034] Figure 7 A schematic diagram of a method for detecting customer behavior provided in this application Figure 6 ;

[0035] Figure 8A schematic diagram of a method for detecting customer behavior provided in this application Figure 7 ;

[0036] Fig. 9 A schematic diagram of the structure of a valet behavior detection device provided in this application;

[0037] Fig.10 A schematic diagram of the structure of a valet behavior detection device provided in this application.

[0038] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0039] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0040] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0041] In addition, this application involves conducting big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and using artificial intelligence technology to make automated decisions, and making technical solutions that have a significant impact on personal rights and interests based on the results of automated decisions. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decisions; if the user chooses to reject, the expert decision-making process will be entered.

[0042] It should be noted that the method, device, equipment, storage medium and program product for detecting customer behavior provided in this application can be used in the field of financial technology, and can also be used in any field other than financial technology. The application field of the method, device, equipment, storage medium and program product for detecting customer behavior in this application is not limited.

[0043] The specific application scenario of this application is that in bank branches or financial institutions, by detecting agency behavior, abnormal transaction activities can be identified, fraud can be prevented, and customer account security can be protected.

[0044] In the prior art, facial recognition is performed on the surveillance images captured by the camera to identify internal personnel and non-inside personnel; target tracking is performed on internal personnel and non-inside personnel respectively. If the distance between the internal personnel and the non-inside personnel is less than the preset distance, mobile phone detection is performed on the monitoring range of the internal personnel and the non-inside personnel. If it is determined that the position of the mobile phone coincides with the rectangular frame of the internal personnel's position, the overlapping time period is recorded, and the transaction data in the background is monitored. If sensitive transaction data appears in the background during the overlapping time period, it is determined to be an act of proxy service.

[0045] However, the usage scenarios of the existing technology have certain limitations. At present, there are various types of equipment in various bank branches or financial institutions. The detection of valet behavior is only carried out by detecting whether the internal personnel's standing rectangular frame and the mobile phone overlap, which cannot meet the detection needs in various actual application scenarios; and the valet behavior detection method in the existing technology is not accurate enough. When the internal personnel demonstrate the equipment, the demonstration behavior may be misjudged as valet behavior.

[0046] Based on this, in the prior art, there is a problem of low accuracy in detecting valet behavior.

[0047] The method for detecting customer service provided by the present application aims to solve the above technical problems of the prior art, including: determining internal personnel and non-internal personnel by performing personnel identification and classification detection processing on the acquired video data to be detected; after determining that the internal personnel have the intention to act as a customer service through tracking and monitoring, more accurate customer service behavior detection is performed to meet the needs of customer service behavior detection in various application scenarios, thereby reducing the misjudgment of customer service behavior and improving the accuracy of customer service behavior detection. After determining that the internal personnel have acted as a customer service, prompt information is sent to the first terminal device and / or the second terminal device respectively to prompt the detection personnel and / or the internal personnel; the customer service behavior is terminated in time to ensure the security of the customer account.

[0048] Figure 1 A diagram of the architecture of a valet behavior detection system provided for this application, such as Figure 1As shown, the valet behavior detection system 10 includes a target recognition and tracking unit 101, a valet intention recognition unit 102, and a valet operation detection unit 103; wherein, after acquiring the video data to be detected, the target recognition and tracking unit 101 performs personnel recognition and classification detection processing on the video data to be detected to determine the internal personnel and non-inside personnel, and tracks and detects the internal personnel and non-inside personnel; the valet intention recognition unit 102 performs a first tracking detection on the internal personnel to determine whether the internal personnel has the valet intention; if it is determined that the internal personnel has the valet intention, the valet operation detection unit 103 performs a second tracking detection on the internal personnel to determine whether the internal personnel has the first valet behavior; if it is determined that the internal personnel has the first valet behavior, the corresponding prompt information is sent to each terminal device to prompt the valet behavior detection result.

[0049] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0050] Figure 2 A schematic diagram of a method for detecting customer behavior provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0051] S201: Obtain video data to be detected.

[0052] In this embodiment, the video data to be detected is video data acquired by a camera.

[0053] S202: Perform personnel recognition on the video data to be detected to identify all persons to be detected.

[0054] In this embodiment, a face recognition algorithm is used to perform person recognition on the video data to be detected to obtain the person to be detected; wherein the face recognition algorithm includes DeepFace (deep face recognition algorithm), FaceNet (face recognition network), CosFace (cosine face recognition algorithm) or ArcFace (arc face recognition algorithm).

[0055] S203, classify and inspect the personnel to be inspected to determine whether they are internal personnel or non-internal personnel.

[0056] In this embodiment, the person to be detected is classified and detected by a classification algorithm; wherein the classification algorithm includes Xception (Extremely Deep Separable Convolutional Network) or ResNeXt (Extended Residual Network).

[0057] S204: Perform a first tracking test on the internal personnel to determine whether the internal personnel intends to act as a proxy for the customer.

[0058] In this embodiment, a first tracking detection is performed on the insider through a multi-target tracking algorithm, wherein the multi-target tracking algorithm includes DeepSort (deep sorting algorithm), JDE (joint detection and embedding algorithm) or FairMOT (fair multi-target tracking algorithm);

[0059] Through the trained intent recognition model, it is determined whether the internal personnel has the intention to act as a proxy for the customer; the trained intent recognition model is CNN for Sentence Classification (Convolutional Neural Networks for Sentence Classification, a convolutional neural network in natural language processing).

[0060] S205: If it is determined that the insider has the intention to act as a proxy, a second tracking test is performed on the insider to determine whether the insider has the first proxy behavior.

[0061] Alternatively, if Figure 3 As shown, a second tracking test is performed on the insider to determine whether the insider has the first surrogate behavior, including:

[0062] S2051. Conduct behavior tracking detection on the internal personnel according to the trained behavior detection model to determine whether the internal personnel have first-generation customer behavior, wherein the trained behavior detection model is trained by multiple video frame samples and behavior labels corresponding to each video frame sample.

[0063] Also optionally, the behavior detection model includes a state space model block and a transformer model block that use efficient visual representation learning based on a bidirectional state space model.

[0064] In this embodiment, the bidirectional state-space model is an extended state-space model that processes time series data;

[0065] Efficient visual representation learning based on bidirectional state space model is a visual representation learning model that combines the bidirectional state space model and the visual representation of position embedding technology;

[0066] The Transformer model block is a deep learning architecture based on a self-attention mechanism to capture global dependencies between elements in a sequence.

[0067] By combining the state space model block based on efficient visual representation learning of the bidirectional state space model with the converter model block, a behavior detection model is obtained, which improves the processing efficiency and detection accuracy of the behavior detection model for data images.

[0068] If Figure 4 As shown, according to the trained behavior detection model, the behavior tracking detection is performed on the insider to determine whether the insider has the first generation customer behavior, including:

[0069] Acquire a video frame sequence; wherein the video frame sequence is a set of continuous image frames of the video data to be detected;

[0070] Based on 3D convolution, the video frame sequence is projected into multiple non-overlapping spatiotemporal segments to obtain multiple spatial blocks corresponding to each video frame; each video frame represents a time, and each video frame is decomposed into multiple spatial blocks;

[0071] Performing bidirectional sequence block rearrangement on multiple spatial blocks corresponding to each video frame to obtain a first sequence and a second sequence; wherein the bidirectional sequence block rearrangement includes sequential rearrangement and reverse scanning, the first sequence is a sequential rearrangement sequence, and the second sequence is a reverse scanning sequence;

[0072] A state space model block for efficient visual representation learning based on a bidirectional state space model, respectively obtaining a context sequence of the first sequence and a context sequence of the second sequence;

[0073] Based on the self-attention mechanism in the transformer model block, the context sequence of the first sequence and the corresponding spatial block are captured, and the dependency relationship between the context sequence of the second sequence and the corresponding spatial block is obtained to obtain the optimized spatial block.

[0074] Based on the converter model block, three feature extractions are performed on the optimized space block to obtain the first feature vector, the second feature vector and the third feature vector in sequence;

[0075] The first eigenvector, the second eigenvector and the third eigenvector are fused to obtain a fused eigenvector;

[0076] Based on the fully connected linear layer, the fused feature vector is calculated and the probability of occurrence of the first generation of customer behavior is output;

[0077] If the probability of the first-generation customer behavior is greater than or equal to the preset threshold, it indicates that the insider has the first-generation customer behavior.

[0078] In this embodiment, the behavior detection model is trained to track the behavior of the internal personnel to determine whether the internal personnel have first-generation customer behavior, thereby improving the processing efficiency of the behavior detection model and the detection accuracy of the first-generation customer behavior.

[0079] After the behavior tracking detection of the insider is performed according to the trained behavior detection model to determine whether the insider has the first generation of customer behavior, it also includes:

[0080] S2052: If it is determined that the internal personnel has the first-generation customer behavior, the terminal device distance detection is performed on the internal personnel.

[0081] In this embodiment, for example, detecting the distance of the terminal device to the internal personnel includes determining whether the key points of the hand of the internal personnel and the terminal device overlap;

[0082] Furthermore, the detection of the distance between the terminal device and the internal personnel also needs to be combined with the human posture of the internal personnel and the identification of the terminal device.

[0083] S2053. Verify whether the internal personnel have first-generation customer behavior based on the detection result of the terminal device distance detection.

[0084] In this embodiment, if the distance between the key points of the insider's hand and the terminal device is less than the preset distance threshold, it is considered that the key points of the insider's hand and the terminal device coincide with each other, and the body posture of the insider and the recognition result of the terminal device are consistent with the state of the agent behavior, then it is confirmed that the insider has the first agent behavior.

[0085] In this embodiment, the terminal device distance detection is performed on the internal personnel to verify whether the internal personnel have the first-generation customer behavior, thereby achieving end-to-end processing, reducing the detection error of the first-generation customer behavior, and improving the detection accuracy of the first-generation customer behavior.

[0086] S206. If it is determined that the internal person has performed the first proxy behavior, a first prompt message is sent to the first terminal device to prompt the detection person that the proxy behavior has occurred, and / or a second prompt message is sent to the second terminal device to prompt the internal person that the proxy behavior has been detected.

[0087] In this embodiment, if it is determined that the internal personnel has the first proxy behavior, a first prompt message is sent to the first terminal device to prompt the detection personnel to have the proxy behavior, so that the detection personnel can promptly remind the internal personnel to stop the proxy behavior to ensure the security of the customer account;

[0088] A second prompt message is sent to the second terminal device to remind the internal personnel that the valet behavior is detected, so as to ensure that if a misjudgment occurs, the internal personnel can file a complaint in time.

[0089] The method for detecting customer service behavior provided in this embodiment determines internal personnel and non-internal personnel by performing personnel identification and classification detection processing on the acquired video data to be detected; after determining that the internal personnel has the intention of customer service through the first tracking detection, the second tracking detection is performed to determine whether the internal personnel has the first customer service behavior; through the two tracking detections, the misjudgment of customer service behavior is reduced and the accuracy of customer service behavior detection is improved. After determining that the internal personnel has the first customer service behavior, prompt information is sent to the first terminal device and / or the second terminal device respectively to realize prompting of the detection personnel and / or the internal personnel; the customer service behavior is terminated in time to ensure the security of the customer account.

[0090] Figure 5 Schematic diagram of a method for detecting valet behavior provided in this application Figure 4 ,exist Figure 2 Based on the embodiment shown, Figure 5 As shown, the classification detection process of the persons to be detected in the above step S203 to determine the internal personnel and the non-internal personnel includes:

[0091] S501. Perform a first classification process on the person to be detected according to the trained classification model, so as to determine whether the person to be detected is an employee based on the clothing information of the person to be detected, wherein the trained classification model is trained by multiple video frame samples and clothing labels corresponding to each video frame sample.

[0092] In this embodiment, for example, the clothing label corresponding to each video frame sample includes a label of 1 for the person to be detected wearing work clothes, and a label of 0 for the person to be detected wearing non-work clothes;

[0093] The detected persons with clothing labels of 1 are determined as employees, and the labels of internal personnel are set to 1; the detected persons with clothing labels of 0 are determined as non-internal personnel, and the labels of non-internal personnel are set to 0.

[0094] S502. Perform face detection on the employee according to the trained face detection model to determine the employee's name, wherein the trained face detection model is trained by using a plurality of video frame samples and a name label corresponding to each video frame sample.

[0095] In this embodiment, a face detection model is trained in a preset detection scenario using multiple video frame samples and name tags corresponding to each video frame sample to obtain a trained face detection model; wherein the name tag corresponding to each video frame sample includes: a face image of an internal person and a corresponding name tag, and / or a face image of a non-inside person and a corresponding "other" tag; the preset detection scenario includes preset different shooting angles, light conditions of different brightness, different types of persons to be detected, and a second training data set using different degrees of occlusion.

[0096] According to the trained face detection model, the employee's face detection is performed to obtain the employee's corresponding name label to determine the employee's name.

[0097] S503: Obtain work status information.

[0098] Obtain a preset employee work clock-in record data table for the day, wherein the preset employee work clock-in record data table for the day includes clock-in time, employee name, and employee face information;

[0099] Compare the employee name in the preset employee work clock-in record data table for the day with the name of the confirmed employee to obtain a comparison result;

[0100] If the comparison result is the employee name in the employee work clock-in record data table of the day, which is consistent with the confirmed employee's name, the employee's work status information is "Work Status";

[0101] If the comparison result is that the employee name in the employee work clock-in record data table for that day is inconsistent with the confirmed employee's name, the employee's work status information is "non-working status".

[0102] S504. Filter out internal personnel based on work status information and employee names.

[0103] In this embodiment, employees whose work status information is "working status" are regarded as internal personnel, and the employee's name is updated to the label of internal personnel; employees whose work status information is "non-working status" are regarded as non-internal personnel, and "other" is updated to the label of non-internal personnel.

[0104] The method for detecting valet behavior provided in the present embodiment performs a first classification process on the persons to be detected according to their clothing information, determines the employee information, confirms the employee's work status through a preset data table of employee work clock-in records for the day, obtains employees in a working state, and determines that the employees in a working state are internal personnel, thereby eliminating the interference of non-working employees in the detection area with the detection of valet behavior, thereby further improving the accuracy of the detection of internal personnel; and adding name tags to internal personnel facilitates timely recording of the information of personnel performing valet behavior.

[0105] Figure 6 Schematic diagram of a method for detecting valet behavior provided in this application Figure 5 ,exist Figure 2 Based on the embodiment shown, Figure 6 As shown, the first tracking detection of the internal personnel in the above step S204 to determine whether the internal personnel has the intention of acting as a proxy for the customer includes:

[0106] S601: Obtain a detection frame of an internal person.

[0107] In this embodiment, a detection frame of the internal personnel in each video frame in the video data to be detected is generated by a multi-target tracking algorithm.

[0108] S602: Obtain a detection frame of a non-internal person.

[0109] In this embodiment, a detection frame of non-inside personnel in each video frame in the video data to be detected is generated by a multi-target tracking algorithm.

[0110] S603: If there is a detection frame of a non-inside person whose distance from the detection frame of the inside person is less than a preset distance threshold, it is determined that the inside person has the intention to serve as a valet.

[0111] In this embodiment, the position coordinates and sizes of the detection frame of the insider and the detection frame of the non-insider are obtained, and based on the position coordinates and sizes of the detection frame of the insider and the detection frame of the non-insider, the center point coordinates of the detection frame of the insider and the detection frame of the non-insider are calculated respectively. The distance between the detection frame of the insider and the detection frame of the non-insider is obtained through the center point coordinates of the detection frame of the insider and the detection frame of the non-insider. If there is a detection frame of the non-insider whose distance to the detection frame of the insider is less than a preset distance threshold, it is determined that the insider has the intention of valeting.

[0112] The method for detecting valet behavior provided in this embodiment determines whether an insider has the intention to act as a valet through the distance between the detection frame of the insider and the detection frame of the non-insider, thereby improving the accuracy of detecting the valet intention, reducing the number of detections to determine the valet operation, and reducing the detection load of valet behavior, thereby improving the detection efficiency.

[0113] Figure 7 A schematic diagram of a method for detecting customer behavior provided in this application Figure 6 ,exist Figure 2 Based on the embodiment shown, Figure 7 As shown, the method also includes:

[0114] S701: Determine whether the scene where the video data to be detected is located is a sound-receivable scene.

[0115] In this embodiment, the sound-collectable scenes include a financial management office that can collect sound, or a self-service device that enables recording.

[0116] S702: If the scene where the video data to be detected is located is a sound-receiving scene, obtain corresponding audio data.

[0117] In this embodiment, if the scene where the video data to be detected is located is a sound-receiving scene, the corresponding audio data is obtained through a sound-receiving device in the sound-receiving scene.

[0118] S703: Determine whether the internal personnel have second-generation customer behavior based on the audio data.

[0119] Alternatively, if Figure 8 As shown, based on the audio data, determine whether the insider has second-generation customer behavior, including:

[0120] S7031. Convert the audio data into text to obtain text data.

[0121] In this embodiment, the audio data is converted into text by using an ANN (artificial neural network), HMM (hidden Markov model) or VQ (vector quantization technology) algorithm to obtain text data.

[0122] S7032. Perform key information identification processing on the text data to determine whether the text data includes key information of preset misleading content.

[0123] In this embodiment, key information recognition processing is performed on the text data based on the trained intent recognition model to determine whether the text data includes key information of preset inducement content, wherein the preset inducement content includes passwords, text messages, verification codes, and inducement signature content.

[0124] For example, the training process of the intent recognition model includes:

[0125] Perform text cleaning, word segmentation, and word stem extraction on text data to obtain preprocessed keywords;

[0126] The pre-processed keywords are converted into high-dimensional vectors through pre-trained word embedding models; the pre-trained word embedding models include Word2Vec (word vector model), FastText (fast text model) or Glove (global vector model);

[0127] Extract features from high-dimensional vectors and optimize them by stacking blocks (also known as multi-layer convolution and pooling layers) to obtain optimized features;

[0128] Based on the optimized features, text modeling is performed to obtain an intent recognition model;

[0129] The intent recognition model is trained by using an intent training data set to obtain a trained intent recognition model, wherein the intent training data set includes standard guidance demonstration content and preset inductive content.

[0130] S7033. If it is determined that the text data includes key information with preset misleading content, it is determined that the insider has engaged in second-generation customer behavior.

[0131] In this embodiment, for example, if it is determined that the key information including preset inducement content in the text data includes: when the insider operates the terminal device, the insider induces the customer to enter a password or a verification code, then it is determined that the insider has second-generation customer behavior.

[0132] In this embodiment, by converting audio data into text data, key information of preset inductive content is identified in the text data to determine whether the insider has second-generation customer behavior, thereby improving the accuracy of second-generation customer behavior detection.

[0133] Optionally, if it is determined that the text data includes key information of preset inductive content, then it is determined that the insider has a second-generation customer behavior, the method further includes:

[0134] S7034. If it is determined that the internal personnel has the first agent behavior and the internal personnel has the second agent behavior, an alarm is issued.

[0135] In this embodiment, if it is determined that the internal personnel has the first proxy behavior and the internal personnel has the second proxy behavior, an alarm is issued to promptly notify the internal personnel to terminate the proxy behavior to protect the security of the customer account.

[0136] S7035. If it is determined that the internal personnel does not have the first agent behavior, and the internal personnel has the second agent behavior, then the audio data is saved.

[0137] In this embodiment, if it is determined that the internal personnel does not have the first behavior of acting as a customer, and the internal personnel has the second behavior of acting as a customer, the audio data is saved for subsequent manual review to enhance the service awareness of the internal personnel.

[0138] Optionally, if it is determined that the insider has the first guest behavior, and the insider does not have the second guest behavior, an alarm is issued.

[0139] The method for detecting the behavior of a second customer proposed in this embodiment determines the scene where sound can be collected, obtains the corresponding audio data, and detects the second customer behavior based on the corresponding audio data, thereby reducing the possibility of misjudgment of the customer behavior and further improving the accuracy of detecting the customer behavior.

[0140] Optionally, the valet behavior detection method proposed in the present application, in the first exemplary embodiment, can be performed by only Figure 7 The method shown is implemented.

[0141] Alternatively, the valet behavior detection method proposed in the present application can be combined with Figure 2 and Figure 7 The method shown is implemented.

[0142] Fig. 9 A schematic diagram of the structure of a valet behavior detection device provided in this application, such as Fig. 9 As shown, the valet behavior detection device includes:

[0143] A first acquisition module 901 is used to acquire video data to be detected;

[0144] The identification module 902 is used to perform personnel identification on the video data to be detected, so as to identify all persons to be detected;

[0145] A classification module 903 is used for classifying and testing the personnel to be tested to determine whether they are internal personnel or non-internal personnel;

[0146] The first detection module 904 is used to perform a first tracking detection on the internal personnel to determine whether the internal personnel has the intention of acting as a proxy for the customer;

[0147] The second detection module 905 is used to perform a second tracking detection on the internal personnel if it is determined that the internal personnel has the intention of acting as a proxy, so as to determine whether the internal personnel has the first proxy behavior;

[0148] The first processing module 906 is used to send a first prompt message to the first terminal device to prompt the detection personnel that the valet behavior has occurred if it is determined that the internal personnel has performed the first valet behavior, and / or send a second prompt message to the second terminal device to prompt the internal personnel that the valet behavior has been detected.

[0149] Optionally, the classification module 903 may also be used for:

[0150] According to the trained classification model, a first classification process is performed on the person to be detected, so as to determine whether the person to be detected is an employee according to the clothing information of the person to be detected, wherein the trained classification model is obtained by training a plurality of video frame samples and clothing labels corresponding to each video frame sample;

[0151] Performing face detection processing on the employee according to the trained face detection model to determine the employee's name, wherein the trained face detection model is trained by using a plurality of video frame samples and a name label corresponding to each video frame sample;

[0152] Get work status information;

[0153] Filter out internal personnel based on work status information and employee names.

[0154] Optionally, the first detection module 904 may also be specifically used for:

[0155] Get the detection frame of the internal personnel;

[0156] Get the detection frame of non-internal personnel;

[0157] If there is a detection frame of a non-inside person whose distance from the detection frame of the inside person is less than a preset distance threshold, it is determined that the inside person has the intention of valeting.

[0158] Optionally, after performing behavior tracking detection on the internal personnel according to the trained behavior detection model to determine whether the internal personnel has the first customer behavior, the first detection module 904 may be specifically used to:

[0159] If it is determined that the internal personnel have the first-time customer behavior, the internal personnel's terminal device distance detection is performed;

[0160] Based on the detection results of the terminal device distance detection, verify whether the insider has the first-generation customer behavior.

[0161] Optionally, the second detection module 905 may also be specifically used for:

[0162] According to the trained behavior detection model, the behavior tracking detection is performed on the insider to determine whether the insider has the first-generation customer behavior, wherein the trained behavior detection model is trained by multiple video frame samples and the behavior label corresponding to each video frame sample.

[0163] Optionally, the behavior detection model includes a state space model block and a transformer model block using efficient visual representation learning based on a bidirectional state space model.

[0164] Optionally, the valet behavior detection device further includes a third detection module, and the third detection module is specifically used to:

[0165] Determine whether the scene where the video data to be detected is located is a scene where sound can be collected;

[0166] If the scene where the video data to be detected is located is a scene where sound can be collected, the corresponding audio data is obtained;

[0167] Based on the audio data, determine whether the insider has second-generation customer behavior.

[0168] Optionally, the third detection module may also be used for:

[0169] Performing text conversion processing on the audio data to obtain text data;

[0170] Performing key information identification processing on the text data to determine whether the text data includes key information of preset misleading content;

[0171] If it is determined that the text data includes key information with preset misleading content, it is determined that the insider has engaged in second-generation customer behavior.

[0172] Optionally, if it is determined that the text data includes key information of preset inductive content, then after it is determined that the insider has the second agent behavior, the first processing module 906 may be specifically used to:

[0173] If it is determined that the insider has the first-generation guest behavior, and the insider has the second-generation guest behavior, an alarm is issued;

[0174] If it is determined that the insider does not have the first agent behavior, and the insider has the second agent behavior, the audio data is saved.

[0175] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0176] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0177] Fig.10A schematic diagram of the structure of a valet behavior detection device provided in this application, such as Fig.10 As shown, the valet behavior detection device includes: at least one processor 1001 and a memory 1002. Optionally, the device also includes a communication component 1003. The processor 1001, the memory 1002 and the communication component 1003 are connected via a bus 1004.

[0178] In a specific implementation process, at least one processor 1001 executes the computer-executable instructions stored in the memory 1002, so that at least one processor 1001 executes the above method.

[0179] The specific implementation process of the processor 1001 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0180] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0181] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0182] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0183] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0184] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0185] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0186] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in a dedicated integrated circuit. Of course, the processor and the readable storage medium can also exist in a device as discrete components.

[0187] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0188] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0190] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes.

[0191] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0192] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for detecting valet behavior, characterized in that: include: Obtain the video data to be detected; Performing personnel recognition on the video data to be detected to identify all persons to be detected; Classify and inspect the persons to be inspected to determine internal personnel and non-internal personnel; Performing a first tracking detection on the insider to determine whether the insider has the intention of acting as a proxy for the customer; If it is determined that the insider has the intention to act as a customer, a second tracking test is performed on the insider to determine whether the insider has the first behavior of acting as a customer; If it is determined that the insider has performed the first valet behavior, a first prompt message is sent to the first terminal device to prompt the detection personnel that the valet behavior has occurred, and / or a second prompt message is sent to the second terminal device to prompt the insider that the valet behavior has been detected.

2. The method according to claim 1, characterized in that The classification detection process of the persons to be detected to determine internal personnel and non-internal personnel includes: According to the trained classification model, a first classification process is performed on the person to be detected, so as to determine whether the person to be detected is an employee according to the clothing information of the person to be detected, wherein the trained classification model is obtained by training a plurality of video frame samples and a clothing label corresponding to each of the video frame samples; Performing face detection processing on the employee according to a trained face detection model to determine the name of the employee, wherein the trained face detection model is trained by using a plurality of video frame samples and a name label corresponding to each of the video frame samples; Get work status information; Internal personnel are screened out according to the work status information and the names of the employees.

3. The method according to claim 1, characterized in that The performing a second tracking detection on the insider to determine whether the insider has the first customer behavior includes: According to the trained behavior detection model, the insider is subjected to behavior tracking detection to determine whether the insider has the first on-behalf-of behavior, wherein the trained behavior detection model is obtained by training with multiple video frame samples and behavior labels corresponding to each of the video frame samples.

4. The method according to claim 3, characterized in that The behavior detection model includes a state space model block and a transformer model block using efficient visual representation learning based on a bidirectional state space model.

5. The method according to claim 3, characterized in that: After performing behavior tracking detection on the insider according to the trained behavior detection model to determine whether the insider has the first customer behavior, the method further includes: If it is determined that the insider has the first proxy behavior, performing a terminal device distance detection on the insider; According to the detection result of the distance detection of the terminal device, it is verified whether the insider has the first agent behavior.

6. The method according to claim 1, characterized in that The first tracking detection of the internal personnel to determine whether the internal personnel has the intention of acting as a proxy for the customer includes: Obtaining a detection frame of the internal person; Obtaining a detection frame of the non-internal personnel; If there is a detection frame of a non-inside person whose distance from the detection frame of the inside person is less than a preset distance threshold, it is determined that the inside person has the intention of valeting.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: Determining whether the scene where the video data to be detected is located is a scene where sound can be collected; If the scene where the video data to be detected is located is a scene where sound can be collected, obtaining corresponding audio data; Based on the audio data, it is determined whether the insider has a second-generation guest behavior.

8. The method according to claim 7, characterized in that Determining, based on the audio data, whether the insider has a second-guest behavior includes: Converting the audio data into text to obtain text data; Performing key information recognition processing on the text data to determine whether the text data includes key information of preset misleading content; If it is determined that the text data includes key information of preset misleading content, it is determined that the insider has engaged in secondary customer behavior.

9. The method according to claim 8, characterized in that After determining that the text data includes key information of preset inductive content, and determining that the insider has second-hand customer behavior, the method further includes: If it is determined that the insider has the first guest behavior, and the insider has the second guest behavior, an alarm is issued; If it is determined that the insider does not have the first guest behavior, and the insider has the second guest behavior, the audio data is saved.

10. A device for detecting valet behavior, characterized in that: include: A first acquisition module, used to acquire video data to be detected; An identification module, used for performing personnel identification on the video data to be detected, so as to identify all persons to be detected; A classification module, used for classifying and detecting the persons to be detected to determine internal personnel and non-internal personnel; A first detection module is used to perform a first tracking detection on the internal personnel to determine whether the internal personnel has the intention of acting as a proxy for the customer; A second detection module is used for, if it is determined that the insider has the intention of acting as a customer, performing a second tracking detection on the insider to determine whether the insider has the first behavior of acting as a customer; The first processing module is used to send a first prompt message to the first terminal device to prompt the detection personnel that the valet behavior has occurred if it is determined that the insider has performed the first valet behavior, and / or send a second prompt message to the second terminal device to prompt the insider that the valet behavior has been detected.

11. A valet behavior detection device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images