Signature action authenticity detection method for dual-recording video quality inspection auditing

Abstract

The invention discloses a signature action authenticity detection method for dual-recording video quality inspection, and relates to the technical field of signature action detection, the method comprises the following steps: detecting pen holders and pen points in each frame of video stream image, counting the number of the pen holders and the number of effective pen points in the video stream, and recording the coordinate change of the effective pen points; judging whether an empty-hand simulation signature condition exists or not; judging whether there is no paper as a signature medium; judging whether the condition that the pen is held but the pen point is not moved exists; and if the judgment results do not exist, outputting a detection result that no false signature action exists in the video, and otherwise, outputting a detection result that the false signature action exists in the video. According to the method, intelligent judgment can be carried out on the condition of empty-hand simulation signature, the condition that no paper is used as a signature medium and the condition that a pen is held but a pen point is not moved in the double-recording video in the financial field, true and false signature behaviors are efficiently and accurately recognized, and the safety and reliability of financial activities are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of signature action detection, and in particular to a method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing. Background Art

[0002] "Dual-recording" refers to the process of recording the entire sales process through audio and video during the sale of financial products. This measure aims to ensure the transparency of the sales process and the integrity of evidence. As an important business operation norm, dual-recording can standardize sales behavior, protect customer rights and interests, and improve the transparency of the sales process. Through audio and video recording, it can ensure that the communication content between sales personnel and customers is completely recorded, avoiding misleading or fraudulent behavior during the sales process. During dual-recording, customers need to sign various documents to prove that their intention to handle business is genuine.

[0003] Both the real-time dual-recording video quality inspection system and the post-event dual-recording video quality inspection system need to verify the authenticity of the customer's signature action. In actual business, the following three situations of false signatures may be encountered: 1) Simulating signature with an empty hand: The participant has no pen in hand and only imitates the signature action through gestures.

[0004] 2) Without paper as the signature medium: The participant holds a pen and makes a signature action on the desktop or in the air, but there is no paper as the signature medium.

[0005] 3) Holding the pen but the tip not moving: The participant holds the pen, but the tip of the pen does not move on the paper and actually does not complete the signature.

[0006] In view of the risks that these false signature behaviors may bring, there is an urgent need for a method and device for detecting the authenticity of signature actions for dual-recording video quality inspection, which can efficiently and accurately identify true and false signature behaviors, thereby ensuring the safety and reliability of financial activities. Summary of the Invention

[0007] To solve the above technical problems, the present invention provides a method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing. The following technical solutions are adopted: A method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing includes the following steps: Step 1, obtaining a dual-recording video stream; Step 2, processing the video stream in step 1 frame by frame, detecting the pen-holding hand and the pen tip existing in each frame of the video stream image, counting the number of pen-holding hands and the number of valid pen tips in the video stream, and recording the coordinate changes of the valid pen tips; Step 3, judging whether there is a false signature action based on the processing result of step 2, including the following sub-steps: Step 31: Determine whether there is a situation of signing by simulating with an empty hand; Step 32: Determine whether there is a situation where there is no paper as the signing medium; Step 33: Determine whether there is a situation where the pen is held but the tip of the pen does not move; If the judgment results of Step 31, Step 32 and Step 33 are all negative, output the detection result that there is no fake signing action in the video. Otherwise, output the detection result that there is a fake signing action in the video.

[0008] The video source can be the following three situations: a real-time video stream obtained from a local camera; a real-time video stream obtained from a remote camera; a recorded dual-recording video file. It is necessary to ensure that the resolution and frame rate of the video are high enough to capture the subtle actions during the signing process.

[0009] By adopting the above technical solution, it is possible to make an intelligent judgment on the situations of signing by simulating with an empty hand, having no paper as the signing medium, and holding the pen but the tip of the pen not moving in the dual-recording video in the financial field, and efficiently and accurately identify true and false signing behaviors, improving the security and reliability of financial activities. This method can not only be applied to the dual-recording quality inspection in the financial field, but also be extended to other scenarios that require verifying the authenticity of signatures, such as contract signing, legal document confirmation, etc.

[0010] Optionally, Step 2 includes the following sub-steps: Step 21, detect the pen-holding hand in each frame image frame_i of the video; Step 22, detect the tip of the pen in frame_i; Step 23, determine whether the tip of the pen in frame_i is valid, that is, whether it is a valid tip of the pen; Loop through Step 21, Step 22 and Step 23 until all M frame images in the video stream are processed.

[0011] By adopting the above technical solution, detecting the pen-holding hand and the tip of the pen and judging the validity of the tip of the pen provide basic data for subsequent true and false signature detection.

[0012] Optionally, Step 21 includes the following sub-steps: Step 211, use an object detection algorithm to detect all hands in each frame image frame_i of the video; Suppose there are Ni hands in frame_i, mark the rectangular box of the jth hand as rect_hand_i_j, and mark the sub-image containing the jth hand as sub_image_hand_i_j, j = 1, 2, 3,..., Ni; Step 212, if Ni > 0, then divide the Ni sub-images sub_image_hand_i_j into two categories: the pen-holding hand and the non-pen-holding hand. Let the number of pen-holding hands be Ki. If Ki == 0, then increment the frame number i by 1 and go back to Step 211; If Ki == 1, then increment the pen-holding hand count count_hand_pen by 1, mark the sub-image containing the pen-holding hand as sub_image_hand_with_pen_i, and mark the corresponding rectangular box as rect_hand_with_pen_i.

[0013] The object detection algorithm in Step 211 can be SSD, yolov5, yolov8, etc.

[0014] By adopting the above technical solution, all hands in the video can be detected and the detected hands can be divided into two categories: the pen-holding hand and the non-pen-holding hand, providing data for subsequent judgment of whether there is a situation of signing in the air without a pen.

[0015] Optionally, in Step 22, use the object detection algorithm to detect the pen tip target in the sub-image sub_image_hand_with_pen_i containing the pen-holding hand, map the rectangular box coordinates of the detected pen tip target to frame_i, and represent the rectangular box coordinates of the pen tip target mapped to frame_i as rect_pen_i.

[0016] The object detection algorithm can be SSD, yolov5, yolov8, etc.

[0017] Considering that the pen tip is a very small target relative to frame_i, directly detecting the pen tip on the entire image frame frame_i may result in a low recall rate. Detect the pen tip in the sub-image sub_image_hand_with_pen_i containing the pen-holding hand to obtain a higher recall rate.

[0018] Optionally, if the pen tip target is not detected, then increment the frame number i by 1 and go back to Step 211. If the pen tip target is detected, use the center (x_i, y_i) of rect_pen_i as the coordinates of the pen tip point_pen_i.

[0019] Optionally, the criterion for judging whether the pen tip is valid is whether there is paper as the signing medium around the pen tip. Take the sub-image of W_i * W_i (W_i > 0) from frame_i with the pen tip coordinates point_pen_i as the center, and divide the sub-image into two categories: valid and invalid; If the sub-image is invalid, increment the frame number i by 1 and go back to Step 211; If the sub-image is valid, increment the valid pen tip count count_valid_pen by 1, mark the valid pen tip coordinates point_valid_pen_i as (x_i, y_i); increment the frame number i by 1, and go back to step 211.

[0020] By adopting the above technical solution, data is provided for subsequent judgment of the situation where the pen is held but the pen tip does not move.

[0021] Optionally, the specific method of step 31 is: if it is determined that the count of the pen-holding hand count_hand_pen is 0, it is determined that there is an empty-handed signature simulation in the video, it is determined that there is a fake signature action in the video, and the process terminates.

[0022] By adopting the above technical solution, it can be determined with high precision that there is an empty-handed signature simulation in the video.

[0023] Optionally, the specific method of step 32 is: If it is judged that , and it is judged that there is no paper as the signature medium, it is determined that there is a fake signature action in the video, and the process is aborted.

[0024] By adopting the above technical solution, it can be determined with high precision that there is no paper as the signature medium in the video.

[0025] Optionally, the specific method of step 33 is: connect the coordinates point_valid_pen_i of all valid pen tips in the video obtained in step 23 to form the movement trajectory of the valid pen tip, smooth the movement trajectory, and then calculate the statistic L of the trajectory. If it is judged that L is less than the set judgment threshold, it is judged that there is a situation where the pen is held but the pen tip does not move in the video, it is determined that there is a fake signature action in the video, and the process terminates.

[0026] By adopting the above technical solution, the coordinates point_valid_pen_i of all valid pen tips in the video are obtained in step 23, and these coordinate points form the movement trajectory of the valid pen tip. First, smooth the movement trajectory, and then calculate the statistic of the trajectory, such as the trajectory length L. If L is less than thr_valid_pen_length, it means that there is a situation where the pen is held but the pen tip does not move in the video, it is determined that there is a fake signature action in the video, and the process terminates.

[0027] A signature action authenticity detection device for dual-recording video quality inspection and auditing includes a memory and a processor. The memory stores a detection program designed by using the signature action authenticity detection method for dual-recording video quality inspection and auditing. The processor is communicatively connected to the memory, runs the detection program, and outputs the detection result.

[0028] Optionally, it further includes a display, which is communicatively connected to the processor, and the processor controls the display to show the detection result of the fake signature action.

[0029] In summary, the present invention includes at least the following beneficial technical effects: The present invention can provide a method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing, which can make intelligent judgments on situations of simulating signatures with empty hands, situations without paper as a signature medium, and situations where the pen tip does not move while holding a pen in dual-recording videos in the financial field, efficiently and accurately identify true and false signature behaviors, and improve the security and reliability of financial activities; this method can not only be applied to the dual-recording quality inspection in the financial field, but also be extended to any other scenarios that require verifying the authenticity of signatures, such as contract signing, legal document confirmation, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic flowchart of the method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing of the present invention; Figure 2a is a schematic diagram of a typical valid pen tip sample in a specific embodiment of the method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing of the present invention; Figure 2b is a schematic diagram of a typical invalid pen tip sample in a specific embodiment of the method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing of the present invention; Figure 2c is a schematic diagram of a typical invalid pen tip sample in a specific embodiment of the method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing of the present invention; Figure 2d is a schematic diagram of a typical sample of the hand without holding a pen in a specific embodiment of the method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing of the present invention; Figure 3a is a schematic diagram of outputting a true signature detection result in a specific embodiment of the method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing of the present invention; Figure 3b is a schematic diagram of outputting the hand detection result of the situation where there is no paper as a signature medium in a specific embodiment of the method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing of the present invention; Figure 3c is a schematic diagram of outputting the hand detection result of the situation where there is no paper as a signature medium in a specific embodiment of the method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing of the present invention; Figure 3d is a schematic diagram of outputting the hand detection result of the situation of simulating signatures with empty hands in a specific embodiment of the method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing of the present invention. Detailed Implementation Manner

[0031] The present invention will be further described in detail below with reference to the accompanying drawings.

[0032] An embodiment of the present invention discloses a method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing.

[0033] Refer to Figure 1 , a method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing, includes the following steps: Step 1, obtain a dual-recording video stream; Step 2, process the video stream in step 1 frame by frame, detect the pen-holding hand and the pen tip existing in each frame of the video stream image, count the number of pen-holding hands and the number of valid pen tips in the video stream, and record the coordinate changes of the valid pen tips; Step 3, based on the processing result of step 2, judge whether there is a false signature action, including the following sub-steps: Step 31: Judge whether there is a situation of simulating signature with an empty hand; Step 32: Judge whether there is a situation without paper as the signature medium; Step 33: Judge whether there is a situation of holding the pen but the pen tip not moving; If the judgment results of step 31, step 32, and step 33 are all non-existent, output the detection result that there is no false signature action in the video, otherwise output the detection result that there is a false signature action in the video.

[0034] The video source can be the following three situations: a real-time video stream obtained from a local camera; a real-time video stream obtained from a remote camera; a recorded dual-recording video file. It is necessary to ensure that the resolution and frame rate of the video are high enough to capture the subtle actions during the signature process.

[0035] It can make an intelligent judgment on the situations of simulating signature with an empty hand, without paper as the signature medium, and holding the pen but the pen tip not moving existing in the dual-recording video in the financial field, efficiently and accurately identify true and false signature behaviors, and improve the security and reliability of financial activities. This method can not only be applied to the dual-recording quality inspection in the financial field, but also be extended to other scenarios that require verifying the authenticity of signatures, such as contract signing, legal document confirmation, etc.

[0036] Step 2 includes the following sub-steps: Step 21, detect the pen-holding hand in each frame image frame_i of the video; Step 22, detect the pen tip in frame_i; Step 23, judge whether the pen tip in frame_i is valid; Loop through step 21, step 22, and step 23 until all M frame images in the video stream are processed.

[0037] Detect the pen - holding hand and the pen tip, and judge the validity of the pen tip to provide basic data for subsequent signature authenticity detection.

[0038] Step 21 includes the following sub - steps: Step 211, use the object detection algorithm to detect all hands in each frame image frame_i of the video; Suppose there are Ni hands in frame_i. Mark the rectangular box of the j - th hand as rect_hand_i_j, and mark the sub - image containing the j - th hand as sub_image_hand_i_j, where j = 1, 2, 3,..., Ni; Step 212, if Ni>0, then divide the Ni sub - images sub_image_hand_i_j into two categories: pen - holding hands and non - pen - holding hands. Suppose the number of pen - holding hands is Ki. If Ki == 0, then increment the frame number i by 1 and return to Step 211; If Ki == 1, then increment the pen - holding hand count count_hand_pen by 1, mark the sub - image containing the pen - holding hand as sub_image_hand_with_pen_i, and mark the corresponding rectangular box as rect_hand_with_pen_i.

[0039] The object detection algorithm in Step 211 can be SSD, yolov5, yolov8, etc.

[0040] By adopting the above technical solution, all hands in the video can be detected, and the detected hands can be divided into two categories: pen - holding hands and non - pen - holding hands, providing data for subsequent judgment of whether there is a situation of empty - hand simulated signature.

[0041] In Step 22, use the object detection algorithm to detect the pen tip target in the sub - image sub_image_hand_with_pen_i containing the pen - holding hand, map the rectangular box coordinates of the detected pen tip target to frame_i, and represent the rectangular box coordinates of the pen tip target mapped to frame_i as rect_pen_i.

[0042] The object detection algorithm can be SSD, yolov5, yolov8, etc.

[0043] Considering that the pen tip is a very small target relative to frame_i, directly detecting the pen tip on the entire image frame frame_i may result in a low recall rate. Detect the pen tip in the sub - image sub_image_hand_with_pen_i containing the pen - holding hand to obtain a higher recall rate. Optionally, if the pen tip target is not detected, increment the frame number i by 1 and return to step 211. If the pen tip target is detected, use the center (x_i, y_i) of rect_pen_i as the coordinate point_pen_i of the pen tip.

[0044] Provide data for subsequent judgment of the situation where the pen is held but the pen tip does not move.

[0045] In step 23, the criterion for judging whether the pen tip is valid is whether there is paper as the signature medium around the pen tip. Taking the pen tip coordinate point_pen_i as the center, crop a sub-image of W_i * W_i (W_i > 0) from frame_i, and classify the sub-image into two categories: valid and invalid. If the sub-image is invalid, increment the frame number i by 1 and return to step 211. If the sub-image is valid, increment the valid pen tip count count_valid_pen by 1, and mark the valid pen tip coordinate point_valid_pen_i as (x_i, y_i); increment the frame number i by 1 and return to step 211.

[0046] By adopting the above technical solution, data is provided for subsequent judgment of the situation where the pen is held but the pen tip does not move.

[0047] Optionally, the specific method of step 31 is: if it is determined that the number of the pen-holding hand count count_hand_pen is 0, it is determined that there is an empty-handed simulated signature in the video, and it is determined that there is a fake signature action in the video, and the process terminates.

[0048] By adopting the above technical solution, it is possible to accurately determine the situation of an empty-handed simulated signature in the video.

[0049] The specific method of step 32 is: If it is judged that , and it is judged that there is no paper as the signature medium, it is determined that there is a fake signature action in the video, and the process is aborted.

[0050] The specific method of step 33 is: Connect the coordinates point_valid_pen_i of all the valid pen tips in the video obtained in step 23 to form the movement trajectory of the valid pen tips, smooth the movement trajectory, and then calculate the statistic L of the trajectory. If it is judged that L is less than the set judgment threshold, it is judged that there is a situation where the pen is held but the pen tip does not move in the video, and it is determined that there is a fake signature action in the video, and the process terminates.

[0051] In step 23, the coordinates point_valid_pen_i of the effective pen tips for all frames in the video are obtained, and these coordinate points form the movement trajectory of the effective pen tips. First, smooth the movement trajectory, and then calculate the statistics of the trajectory, such as the trajectory length L. If L is less than thr_valid_pen_length, it indicates that the pen is held but the pen tip does not move in the video, and it is determined that there is a false signature action in the video, and the process terminates.

[0052] A signature action authenticity detection device for dual-recording video quality inspection and auditing, including a memory and a processor. The memory stores a detection program designed by the signature action authenticity detection method for dual-recording video quality inspection and auditing. The processor is communicatively connected to the memory, runs the detection program, and outputs the detection result.

[0053] It further includes a display, and the display is communicatively connected to the processor. The processor controls the display to show the detection result of the false signature action.

[0054] The following uses specific implementation cases to illustrate the implementation principle of the present invention: First, collect hand samples in various different situations for training the required object detection and classification algorithms. Typical samples are shown in Figure 2a - Figure 2d ; A method and device for detecting false signature actions for dual-recording quality inspection are as follows: Step 1: Obtain the dual-recording video stream.

[0055] The video source can be the following three situations: 1) a real-time video stream obtained from a local camera, 2) a real-time video stream obtained from a remote camera, 3) a recorded dual-recording video file. It is necessary to ensure that the resolution and frame rate of the video are high enough to capture the subtle actions during the signature process.

[0056] In a specific embodiment, false signature action detection is performed on the real-time video stream of the customer signature link of the real-time intelligent dual-recording system.

[0057] In another specific embodiment, for the counter service dual-recording system, when the entire video recording process ends, the system will automatically save the complete dual-recording video data and accurately record the exact start and end time points of each key step. Subsequently, based on these time markers, the part specifically containing the signature process is extracted from the overall dual-recording video file as a sub-video, and false signature action detection is performed on it.

[0058] Step 2: Process the video stream described in step 1 frame by frame, count the number of pen-holding hands and the number of effective pen tips in the video stream, and record the coordinate changes of the effective pen tips.

[0059] Suppose there are M frames of images in the video stream, and the i-th frame of the video stream is marked as frame_i. Let the number of pen-holding hands be count_hand_pen = 0, and the number of valid pen tips be count_valid_pen = 0.

[0060] Let i = 0.

[0061] Step 21: Detect the pen-holding hand in each frame of the video frame_i.

[0062] Step 211: Use an object detection algorithm to detect all hands in each frame of the video frame_i. It is not limited to which object detection algorithm is used, and it can be SSD, yolov5, yolov8, etc.

[0063] There may be Ni > 0 hands in frame_i. Mark the rectangular box of the j-th hand as rect_hand_i_j, and mark the sub-image containing the j-th hand as sub_image_hand_i_j, where j = 1, 2, 3,..., Ni; In one embodiment, first obtain hand samples in various different situations, and manually annotate the hand object detection boxes (to facilitate the subsequent detection of the pen tip in step 22 and ensure that the pen tip area is annotated within the hand object detection box), and then use them to train the yolov5 object detection model, which can detect hands in all possible poses including hand postures.

[0064] Step 212: If Ni > 0, then divide the Ni sub-images sub_image_hand_i_j into two categories: pen-holding hands and non-pen-holding hands. Let the number of pen-holding hands be Ki. Generally, only one person signs in the video, so Ki == 0 or 1.

[0065] Typical samples of pen-holding hands are as Figure 2a , Figure 2b , Figure 2c shown, and typical samples of non-pen-holding hands are as Figure 2d shown.

[0066] If Ki == 0, then i = i + 1, and go back to step 211.

[0067] If Ki == 1, then count_hand_pen = count_hand_pen + 1, mark the sub-image containing the pen-holding hand as sub_image_hand_with_pen_i, and the corresponding rectangular box as rect_hand_with_pen_i = (x_lt_i, y_rb_i, w_i, h_i).

[0068] Typical samples of the sub-images of pen-holding hands are as Figure 3a, Figure 3b , Figure 3c The sub-images within the red frames.

[0069] In a specific embodiment, the hand detection result (the sub-images within the red frames) is normalized into a 256*256 picture, and then a classifier is trained using the VGG network and the Mean Squared Loss.

[0070] Step 22: Detect the pen tip in frame_i.

[0071] Considering that the pen tip is a very small target relative to frame_i, directly detecting the pen tip on the entire image frame frame_i may result in a low recall rate. Therefore, the present invention detects the pen tip in the sub-image sub_image_hand_with_pen_i containing the pen-holding hand to obtain a higher recall rate.

[0072] In the sub-image sub_image_hand_with_pen_i containing the pen-holding hand, a target detection algorithm is used to detect the pen tip target, and the rectangular box coordinates of the detected pen tip target are mapped into frame_i. The rectangular box coordinates of the pen tip target mapped into frame_i are denoted as rect_pen_i. Similarly, regardless of the target detection algorithm adopted, it can be SSD, yolov5, yolov8, etc.

[0073] If the pen tip target is not detected, then i = i + 1, and return to step 211.

[0074] If the pen tip target is detected, the center (x_i, y_i) of rect_pen_i is used as the coordinate point_pen_i of the pen tip.

[0075] Step 23: Determine whether the pen tip in frame_i is valid. The criterion for whether the pen tip is valid is whether there is paper as the signature medium around the pen tip.

[0076] Typical valid pen tip samples are as Figure 2a shown, and typical invalid pen tip samples are as Figure 2b , Figure 2c shown.

[0077] Taking the pen tip coordinate point_pen_i as the center, a sub-image of W_i*W_i (W_i > 0) is cropped from frame_i, and the sub-image is divided into two categories: valid and invalid.

[0078] In one embodiment, W_i = int(max(w_i,h_i)*0.2+0.5), and the sub-image of W_i*W_i (W_i > 0) cropped from frame_i is normalized to a 128*128 picture, and then a classifier is trained using the VGG network and the Mean Squared Loss.

[0079] If the sub-image is invalid, then i = i + 1, and go back to step 211.

[0080] If the sub-image is valid, then count_valid_pen = count_valid_pen + 1, and set the valid pen tip coordinates point_valid_pen_i = (x_i, y_i) Loop through steps 21, 22, and 23 until all images in the video stream are processed.

[0081] Step 3: Determine whether there is a false signature action.

[0082] Step 31: Determine whether there is a situation of simulating signature with an empty hand.

[0083] If count_hand_pen < thr_hand_pen, where thr_hand_pen is the set judgment threshold, it means there is a situation of simulating signature with an empty hand in the video, and it is determined that there is a false signature action in the video. The process terminates.

[0084] In a specific embodiment, false signature action detection is performed on the real-time video stream of the customer signature link of the real-time intelligent dual recording system. The business rule is set that the time of the hand holding the pen lasts at least 5 seconds. At the same time, in order to improve the real-time performance of the detection result, only 2 frames of images are collected per second for false signature detection. At this time, thr_hand_pen = 5 * 2 = 10.

[0085] In another specific embodiment, for the self-service dual recording system, when the entire video recording process ends, the system will automatically save the complete dual recording video data and accurately record the exact start and end time points of each key step. Subsequently, based on these time marks, the part specifically containing the signature process is extracted from the overall recording as a sub-video, and false signature action detection is performed on it. Assuming the fps of the video data is 10 frames per second, the business rule is set that the time of the hand holding the pen lasts at least 4 seconds. At the same time, in order to improve the accuracy of the detection result, 5 frames of images are collected per second for false signature detection. At this time, thr_hand_pen = 4 * 5 = 20.

[0086] Step 32: Determine whether there is a situation where there is no paper as the signature medium.

[0087] If , it indicates that there is a situation in the video where there is no paper as the signature medium, and it is determined that there is a false signature action in the video. The process terminates.

[0088] In a specific embodiment, the camera captures the upper body of the signer, and the video resolution is only 640*480. Considering that the tip target will be relatively blurred in the sub-image sub_image_hand_with_pen_i containing the pen-holding hand, and the recall rate is relatively low. To improve the recall rate, thr_valid_pen = 0.6 is set.

[0089] In a specific embodiment, the camera only captures the range of the arm + paper, and the video resolution reaches 1080p. Considering that the tip target will be clearer in the sub-image sub_image_hand_with_pen_i containing the pen-holding hand, to ensure the accuracy of the false signature action and reduce false alarms, thr_valid_pen = 0.8 is set.

[0090] Step 33: Determine whether there is a situation where the pen is held but the tip does not move.

[0091] In step 23, the coordinates point_valid_pen_i of the valid pen tips in all frames of the video are obtained, and these coordinate points form the movement trajectory of the valid pen tips. First, smooth the movement trajectory, and then calculate the statistics of the trajectory, such as the trajectory length L. If L < thr_valid_pen_length, it indicates that there is a situation where the pen is held but the tip does not move in the video, and it is determined that there is a false signature action in the video.

[0092] In a specific embodiment, remove the frames in the video that do not have valid pen tips, and retain the frames containing valid pen tips. Represent the trajectories of all valid pen tips as: (x_1,y_1), (x_2,y_2),..., (x_K,y_K), where K = count_valid_pen.

[0093] L = 0; for i in 1,2,3,...,K - 1; j = i + 1; D = sqrt((x_i - x_j)*(x_i - x_j) + (y_i - y_j)*(y_i - y_j)); L = L + D; If the judgment results of step 31, step 32, and step 33 are all negative, it is determined that there is no false signature action in the video, and the signature action in the video is real.

[0094] The above are all preferred embodiments of the present invention, and the protection scope of the present invention is not limited thereby. Therefore, all equivalent changes made according to the structure, shape, and principle of the present invention shall be covered within the protection scope of the present invention.

Claims

1. A method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing, characterized in that, It includes the following steps: Step 1, obtain the dual-recorded video stream; Step 2, process the video stream frame by frame, detect the pen-holding hand and the pen tip existing in each frame of the video stream image, count the number of pen-holding hands and the number of valid pen tips in the video stream, and record the coordinate changes of the valid pen tips; Step 3, judge whether there is a false signature action based on the processing result of Step 2, including the following sub-steps: Step 31: Judge whether there is a situation of simulating signature with an empty hand; Step 32: Judge whether there is a situation without paper as the signature medium; Step 33: Judge whether there is a situation where the pen is held but the pen tip does not move; If the judgment results of Step 31, Step 32 and Step 33 are all non-existent, output the detection result that there is no false signature action in the video, otherwise output the detection result that there is a false signature action in the video.

2. The authenticity detection method for signature actions used in dual-recording video quality inspection and auditing according to claim 1, characterized in that: Step 2 includes the following sub-steps: Step 21, detect the pen-holding hand in each frame image frame_i of the video; Step 22, detect the pen tip in frame_i; Step 23, judge whether the pen tip in frame_i is valid; Loop Step 21, Step 22 and Step 23 until all M frame images in the video stream are processed.

3. The authenticity detection method of signature actions for dual-recording video quality inspection and auditing according to claim 2, wherein: Step 21 includes the following sub-steps: Step 211, use the object detection algorithm to detect all hands in each frame image frame_i of the video; Suppose there are Ni hands in frame_i, mark the rectangular box of the jth hand as rect_hand_i_j, and mark the sub-image containing the jth hand as sub_image_hand_i_j, j = 1, 2, 3,..., Ni; Step 212, if Ni > 0, divide the Ni sub-images sub_image_hand_i_j into two categories: pen-holding hands and non-pen-holding hands. Suppose the number of pen-holding hands is Ki. If Ki == 0, increment the frame number i by 1 and return to Step 211; If Ki == 1, increment the pen-holding hand count count_hand_pen by 1, mark the sub-image containing the pen-holding hand as sub_image_hand_with_pen_i, and mark the corresponding rectangular box as rect_hand_with_pen_i.

4. The authenticity detection method of signature actions for dual-recording video quality inspection and auditing according to claim 2, wherein: In Step 22, use the object detection algorithm to detect the pen tip target in the sub-image sub_image_hand_with_pen_i containing the pen-holding hand, map the rectangular box coordinates of the detected pen tip target to frame_i, and represent the rectangular box coordinates of the pen tip target mapped to frame_i as rect_pen_i.

5. The method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing according to claim 4, wherein: If the pen tip target is not detected, increment the frame number i by 1 and return to Step 211. If the pen tip target is detected, use the center (x_i, y_i) of rect_pen_i as the coordinate of the pen tip point_pen_i.

6. The method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing according to claim 2, wherein: The standard for judging whether the pen tip is valid in Step 23 is whether there is paper as the signature medium around the pen tip. Take the pen tip coordinate point_pen_i as the center, crop a sub-image of W_i * W_i (W_i > 0) from frame_i, and divide the sub-image into two categories: valid and invalid; If the sub-image is invalid, increment the frame number i by 1 and return to step 211; If the sub-image is valid, increment the valid pen tip count count_valid_pen by 1, and mark the valid pen tip coordinates point_valid_pen_i as (x_i, y_i); increment the frame number i by 1 and return to step 211.

7. The method for detecting the authenticity of signature actions for dual-recording video quality inspection and auditing according to claim 1, characterized in that: The specific method of step 31 is: if it is determined that the number of pen-holding hand counts is 0, it is determined that there is an empty-handed signature simulation in the video, and it is determined that there is a false signature action in the video, and the process terminates.

8. The authenticity detection method for signature actions used in dual-recording video quality inspection and auditing according to claim 1, characterized in that: The specific method of step 32 is: If it is determined that , in the case where there is no paper as the signature medium, it is determined that there is a fake signature action in the video, and the process is aborted.

9. The authenticity detection method for signature actions in the dual-recording video quality inspection and auditing according to claim 1, characterized in that: The specific method of step 33 is: connect the coordinates point_valid_pen_i of all the valid pen tips in the video obtained in step 23 to form the movement trajectory of the valid pen tip, smooth the movement trajectory, and then calculate the statistic L of the trajectory. If it is determined that L is less than the set judgment threshold, it is determined that there is a situation where the pen is held but the pen tip does not move in the video, and it is determined that there is a false signature action in the video, and the process terminates.

10. A signature action authenticity detection device for dual-recording video quality inspection and auditing, characterized in that: It includes a memory and a processor. The memory stores a detection program designed by the signature action authenticity detection method for dual-recording video quality inspection and auditing according to any one of claims 1-9. The processor is communicatively connected to the memory, runs the detection program, and outputs the detection result.

Images