Method and device for determining signer in video, computer equipment and storage medium
By obtaining the voice content of the video and extracting target frame clips, combining action classification and human body detection models, the problem of low efficiency and accuracy of signers in the financial field is solved, and more efficient and accurate signer recognition is achieved.
Patent Information
- Application Number
- CN202411011304.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-12-06
AI Technical Summary
In the existing technology, in the dual recording system in the financial field, the signer's determination efficiency and accuracy are low. This is mainly due to the jitter of the recording device and the face of the signature subject is not in the video, resulting in inaccurate face tracking results. Traditional methods require frame-by-frame detection. time consuming.
By obtaining the voice content of the video to be identified, the target video frame clip is extracted, the signed action clip is identified using the action classification model, and the signer is determined in combination with the human body detection model, reducing the amount of tasks, and improving the efficiency and accuracy of action recognition.
The efficiency and accuracy of the signer's determination are improved, combined with voice content and action recognition, the full processing of the video is reduced, and the accuracy of the signature action clip is improved.
Smart Images

Figure CN119091464A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of video data processing and financial technology, and in particular to a method, apparatus, computer equipment and storage medium for determining the signatory in a video. Background Technology
[0002] With the development of computer vision technology, dual recording (audio and video recording), face detection, and human motion recognition technologies are widely used in the financial sector. In financial institutions such as banks, securities firms, and insurance companies, dual recording systems are used to record the communication and transaction processes between customers and service personnel. This primarily involves documenting the customer's business process, especially the risk disclosure process, to standardize the sales behavior of financial institutions and provide evidence in case of future disputes. During business processing, customers need to sign contracts, terms, and other materials for confirmation. The dual recording stage can record the customer's signing process. In the subsequent quality inspection stage, staff need to review each dual recording video to confirm whether it is indeed the customer's signature, which is inefficient. For the signing action, the traditional method is to detect objects such as hands and pens in each frame of the video and make logical judgments. Furthermore, if it is necessary to determine who is signing, face tracking is required from the start of the signing action until the end, and the signer is identified through the tracking results. However, during the signing process, due to the need to record the signing action, the recording equipment will experience significant shaking, and the face of the signatory is not included in the recording, resulting in inaccurate tracking results. Moreover, the frame-by-frame detection and tracking method is more time-consuming, leading to low efficiency and accuracy in identifying the signatory. Therefore, improving the efficiency and accuracy of signatory identification has become an urgent problem to be solved. Summary of the Invention
[0003] This application provides a method, apparatus, computer equipment, and storage medium for determining the signatory in a video, so as to improve the efficiency and accuracy of signatory determination.
[0004] Firstly, this application provides a method for identifying a signatory in a video, the method comprising:
[0005] Acquire the video to be identified, and perform speech recognition on the video to be identified to obtain the speech content of the video to be identified;
[0006] Based on the voice content, a preset number of target video frame segments are extracted from the video to be identified;
[0007] Based on a preset action classification model, action recognition is performed on each of the target video frame segments to determine the signing action segment.
[0008] Based on a preset human detection model, human detection is performed on the signing action segment to determine the signer.
[0009] Secondly, this application also provides a device for identifying a signatory in a video, the device comprising:
[0010] The speech content acquisition module is used to acquire the video to be recognized and perform speech recognition on the video to be recognized to obtain the speech content of the video to be recognized.
[0011] The target segment extraction module is used to extract a preset number of target video frame segments from the video to be identified based on the speech content.
[0012] The action segment determination module is used to perform action recognition on each of the target video frame segments based on a preset action classification model, and determine the signature action segment.
[0013] The signatory determination module is used to perform human detection on the signing action segment based on a preset human detection model to determine the signatory.
[0014] Thirdly, this application also provides a computer device, the computer device including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the method for determining the signatory in the video as described above.
[0015] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the method for determining the signatory in the video as described above.
[0016] This application discloses a method, apparatus, computer device, and storage medium for determining a signatory in a video. The method involves acquiring a video to be identified, performing speech recognition on the video to obtain its speech content, extracting a predetermined number of target video frame segments based on the speech content, performing action recognition on each target video frame segment based on a predetermined action classification model to determine a signing action segment, and performing human detection on the signing action segment based on a predetermined human detection model to determine the signatory. This application can determine target video frame segments in the video to be identified based on its speech content, then perform action recognition on the target video frame segments to determine the signing action segment, and finally perform human detection on the signing action segment. This eliminates the need for action recognition on the entire video to be identified and human detection on all target video frame segments, reducing the workload and improving action recognition efficiency, thereby improving the efficiency of signatory determination. Furthermore, by combining speech content and action recognition to determine the signing action segment, the accuracy of the signing action segment determination is improved, thus improving the accuracy of signatory determination. Attached Figure Description
[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0018] Figure 1 This is a schematic flowchart of a first embodiment of a method for determining a signatory in a video provided by an embodiment of this application;
[0019] Figure 2 This is a schematic flowchart of a second embodiment of a method for determining a signatory in a video provided by the embodiments of this application;
[0020] Figure 3 A schematic block diagram of a device for determining a signatory in a video, provided for embodiments of this application;
[0021] Figure 4 A schematic block diagram of the structure of a computer device provided for an embodiment of this application. Detailed Implementation
[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0023] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.
[0024] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0025] It should also be understood that the term "and / or" as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0026] This application provides a method, apparatus, computer device, and storage medium for determining a signatory in a video. The method for determining a signatory in a video can be applied to a server. Based on the audio content of the video to be identified, target video frame segments are determined within the video. Then, action recognition is performed on the target video frame segments to determine the signing action segments. Finally, human detection is performed on the signing action segments. This eliminates the need for action recognition on the entire video to be identified and for human detection on all target video frame segments, reducing the workload and improving action recognition efficiency, thereby improving the efficiency of signatory determination. Furthermore, combining audio content and action recognition to determine the signing action segments improves the accuracy of the signing action segments, thus improving the accuracy of signatory determination. The server can be a standalone server or a server cluster.
[0027] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0028] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a method for determining a signatory in a video, provided by an embodiment of this application. This method can be applied to a server, whereby it determines target video frame segments within the video to be identified based on the audio content. Then, it performs action recognition on the target video frame segments to determine the signing action segments, and finally performs human detection on the signing action segments. This method eliminates the need for action recognition on the entire video to be identified and for human detection on all target video frame segments, reducing the workload and improving action recognition efficiency, thereby improving the efficiency of signatory determination. Furthermore, by combining audio content and action recognition to determine the signing action segments, the accuracy of the signing action segments is improved, thus enhancing the accuracy of signatory determination.
[0029] like Figure 1 As shown, the method for determining the signatory in the video specifically includes steps S101 to S104.
[0030] S101. Obtain the video to be identified, and perform speech recognition on the video to be identified to obtain the speech content of the video to be identified;
[0031] Furthermore, the step of acquiring the video to be identified includes: detecting the number of people in the recorded video based on the human body detection model; and when the number of people is greater than or equal to a preset threshold, using the recorded video as the video to be identified.
[0032] In one embodiment, a camera device is used to dual-record all or part of the business processing process. At the start of dual-recording, a human detection model (e.g., MoveNet (real-time pose detection model), YOLOv5 (object detection model), etc.) is used to detect people appearing in the video. When the number of people is greater than or equal to a preset number threshold, the currently recorded video is considered to meet the requirements and is identified as the video to be identified. The number threshold can be freely set by the user according to the actual situation.
[0033] In one embodiment, dual recording refers to both audio and video recording. Dual recording allows for the acquisition of audio and video information during business processing.
[0034] In one embodiment, the video to be identified includes both video and audio information. Speech recognition technology is used to identify the audio information in the video to obtain the audio content. Specifically, speech recognition can be performed using an ASR (Automatic Speech Recognition) model.
[0035] S102. Based on the voice content, extract a preset number of target video frame segments from the video to be identified;
[0036] Further, the step of extracting a preset number of target video frame segments from the video to be identified based on the voice content includes: determining the start frame and end frame of the signing process in the video to be identified based on the voice content; and extracting a preset number of target video frame segments from a target video frame interval based on a preset video frame extraction rule, wherein the target video frame interval is the video frame interval between the start frame and the end frame.
[0037] In one embodiment, the voice content includes dialogue information between the customer and the sales personnel. Analyzing this dialogue information can determine whether the signing process has begun and ended. Specifically, the voice content can be converted to text and analyzed to identify keywords or phrases related to the signing process, such as "start signing" or "signed." Then, the timestamps of the voice content and video frames are aligned. The video frame corresponding to the time when the keyword indicating the start of the signing process appears is determined as the start frame, and the video frame corresponding to the time when the keyword indicating the end of the signing process appears is determined as the end frame.
[0038] In one embodiment, the target video frame interval is defined as the video frame interval between the start frame and the end frame. Video frames within this interval are extracted uniformly, with T frames extracted from every t frames. seq Frame segments are used as target video frame segments, and a total of T is extracted. seq *n frames, T seqThe number of video frames for a single inference by the action classification model is typically 16 or 8, and n is a preset number.
[0039] S103. Based on a preset action classification model, perform action recognition on each of the target video frame segments to determine the signature action segment.
[0040] In one embodiment, the action classification model can identify and classify the actions present in the target video segment, determine whether there is a signing action in the target video segment, and identify it as a signing action segment if a signing action exists.
[0041] In one embodiment, the action classification model can be Inception 3D (dilated 3D convolutional neural network model), SlowFast Networks (multi-scale spatiotemporal model), Two-Stream Networks model, etc., or other models specified by the user that can recognize signature actions.
[0042] In a specific embodiment, the output result of the action classification model after performing action recognition on each target video frame segment is R. n ={x|x=0or1}, where 0 indicates that there is no signing action segment and 1 indicates that there is a signing action frequency band.
[0043] Furthermore, before determining the signature action segment by performing action recognition on each of the target video frame segments based on the preset action classification model, the method further includes: acquiring at least one model training video, wherein the model training video includes signature segments and non-signature segments; classifying the model training video by action based on the preset pre-trained model to obtain action classification results; obtaining the classification accuracy of the pre-trained model based on the action classification results and the signature and non-signature segments in the model training video; and determining the pre-trained model as the action classification model when the classification accuracy is greater than or equal to a preset threshold.
[0044] In one embodiment, at least one model training video containing signed and unsigned segments is obtained.
[0045] In one embodiment, the model is pre-trained using the input values of the training video to obtain the action classification results output by the model. The action classification results include whether the training video contains a signing action and the corresponding signature segment.
[0046] In one embodiment, the action classification results are compared with signature and non-signature segments in the model training video to determine the accuracy of the classification. Specifically, the signature and non-signature segments identified by the pre-trained model are compared with the actual signature and non-signature segments in the model training video. If the signature and non-signature segments identified by the pre-trained model match the actual signature and non-signature segments in the model training video, the pre-trained model has correctly classified the data; otherwise, the classification is incorrect. After the pre-trained model has classified all the model training videos, the classification accuracy of the current pre-trained model is calculated based on the number of correctly classified and incorrectly classified data.
[0047] In one embodiment, if the classification accuracy is greater than or equal to a preset accuracy threshold, then the pre-trained model is determined to be an action classification model; if the classification accuracy is less than the preset accuracy threshold, the parameters of the pre-trained model are adjusted, and the model training videos are reclassified until the classification accuracy is equal to or greater than the preset accuracy threshold.
[0048] S104. Based on a preset human detection model, perform human detection on the signing action segment to determine the signer.
[0049] In one embodiment, a human detection model detects the human body region within the signing action segment, identifies the largest human body region within that region, and inputs this largest region into a ReID (Re-Identification) model to extract features. Simultaneously, the human body region detected by the human detection model during dual-recording is acquired and input into the ReID model for feature extraction. Finally, the distance between features is calculated, and a preset distance threshold is used to determine the signer from the individuals detected by the human detection model during dual-recording. This distance threshold can be set by the user according to specific circumstances.
[0050] Furthermore, after performing human detection on the signing action segment based on a preset human detection model to determine the signer, the method further includes: acquiring the signer's facial features; comparing the facial features with a preset facial database to obtain the signer's identity information, so as to determine whether the signer is the target person based on the identity information.
[0051] In one embodiment, facial features can be obtained by extracting features from the facial region of the person signing the document using a face detection model (e.g., MTCNN (Multi-task Cascaded Convolutional Networks), SCRFD (Single-stage Convolutional Regression Face Detector)). Facial features can include geometric features, texture features, and depth features.
[0052] In one embodiment, the face database can be a face database of staff members. In this case, when the facial features of the signer match a face in the face database, it indicates that the signer is a staff member; when the facial features of the signer do not match a face in the face database, it indicates that the signer is a customer.
[0053] In another embodiment, the face database can also be a customer face database. In this case, when the signatory's facial features match a face in the database, it indicates that the signatory is a customer; when the signatory's facial features do not match a face in the database, it indicates that the signatory is a staff member. The customer face database can be collected when a customer conducts business.
[0054] In one embodiment, the target person is the individual who needs to sign during the business transaction. For example, in selling insurance, a policy agreement needs to be signed with the customer, and the customer's signature on the agreement makes the customer the target person. In other business processes requiring authorization from a salesperson's signature, the salesperson is the target person.
[0055] In one embodiment, comparing the signatory's facial features against a face database can be done by comparing the distance between feature vectors.
[0056] In a specific embodiment, when facial features of staff or customers are collected, the facial features are encoded to obtain feature vectors, which are then stored to obtain a facial database. The distance between the feature vector corresponding to the signature's facial features and each feature vector in the facial database is calculated, and the identity information of the person corresponding to the feature with the largest distance exceeding a preset threshold is taken as the signature's identity information.
[0057] The above embodiments provide a method, apparatus, computer device, and storage medium for determining a signatory in a video. The method involves acquiring a video to be identified, performing speech recognition on the video to obtain its speech content, extracting a preset number of target video frame segments based on the speech content, performing action recognition on each target video frame segment based on a preset action classification model to determine a signing action segment, and performing human detection on the signing action segment based on a preset human detection model to determine the signatory. This application can determine target video frame segments in the video to be identified based on its speech content, then perform action recognition on the target video frame segments to determine the signing action segment, and finally perform human detection on the signing action segment. This eliminates the need for action recognition on the entire video to be identified and human detection on all target video frame segments, reducing the workload and improving action recognition efficiency, thereby improving the efficiency of signatory determination. Furthermore, combining speech content and action recognition to determine the signing action segment improves the accuracy of the signing action segment determination, thus improving the accuracy of signatory determination.
[0058] Please see Figure 2 , Figure 2 This is a schematic flowchart illustrating a method for determining a signatory in a video, provided by an embodiment of this application. This method can be applied to a server, whereby it calculates the distance between the features of the largest human body region in the intermediate frame image of a signing action segment and the features of the human body region image in the video to be identified, and determines the signatory based on the distance. This avoids missed detections during the tracking process and eliminates the need for continuous human body detection, thus improving the efficiency and accuracy of signatory determination.
[0059] like Figure 2 As shown, the method for determining the signatory in the video specifically includes steps S201 to S204.
[0060] S201. Based on the human body detection model, perform human body detection on the intermediate frame image of the signing action segment to determine the human body area of each person in the intermediate frame image.
[0061] S202. Take the area where the human body with the largest area is located as the target area, and extract features from the target area to obtain the first feature;
[0062] S203. Obtain the person region image of each person in the video to be identified, and extract features from each person region image to obtain the features corresponding to each person in the video to be identified, and generate a second feature set.
[0063] S204. Based on the first feature and the second feature set, determine the signatory.
[0064] In one embodiment, the intermediate frame image of the signing action segment is extracted, and a human detection model is used to perform human detection on the intermediate frame image to obtain the human body region of the person present in the intermediate frame image.
[0065] In a specific embodiment, the output result of the action classification model after performing action recognition on each target video frame segment is R. n ={x|x=0 or 1}, where 0 represents the absence of a signing action segment, and 1 represents the presence of a signing action segment. In the segment where x is 1, the intermediate frame T is extracted. mid (i.e., the intermediate frame image) contains the signing action and the signer. Human detection is used to detect people in this intermediate frame image, obtaining the region {p} where the human body is located. t1 ,p t2}, take the maximum p in the region max =max(p t1 ,p t2 Further judgment will be made based on the location of the human body.
[0066] In one embodiment, the determined human body p max The region in question is cropped and input into the ReID (Re-Identification) model to extract p. max The feature, as the first feature f max .
[0067] In one embodiment, the human detection model obtains the people detected in the video during dual recording. If two people are detected, they are labeled as {p1, p2}.
[0068] In one embodiment, the regions where p1 and p2 are located are input into the ReID model to extract features f1 and f2, which together form a second feature set.
[0069] In one embodiment, f is calculated max The distances d1 and d2 between f1 and f2. Take the minimum distance d. min =min(d1,d2), if d min <d thres If d1, d2, then argmin(d1, d2) is considered the signatory. min ≥d thres If the extracted features are insufficient to determine who signed, manual verification is required. Where d thres The preset distance threshold can be set by the user according to the actual situation.
[0070] Further, determining the signatory based on the first feature and the second feature set includes: calculating the distance between the first feature and each feature in the second feature set; determining the minimum distance and the feature corresponding to the minimum distance based on the distance between the first feature and each feature in the second feature set, and taking the feature corresponding to the minimum distance as the target feature; when the minimum distance is less than a preset distance threshold, determining the person corresponding to the target feature as the signatory.
[0071] In one embodiment, the first feature f is calculated. max The distance to each feature in the second feature set. For example, assuming the second feature set includes features f1 and f2, then the distance to the first feature f is calculated. max The distance d1 to feature f1 and the first feature f max The distance d2 to feature f2.
[0072] In one embodiment, the minimum distance, d1, is determined from the distances d2 and d1. min =min(d1,d2). This will set d... min Distance threshold d thres When comparing, when d min <d thres When, then d is considered min The corresponding feature is the target feature, and the person corresponding to the target feature is the signatory.
[0073] For example, suppose d min =d1,d1 <d thres If feature f1 is the target feature, then the person p1 corresponding to feature f1 is the signatory.
[0074] In the above embodiments, the features of the largest human body region in the intermediate frame image of the signing action segment are used to calculate the distance with the features of the human body region image in the video to be identified, and the signer is determined based on the distance. Using the ReID features of the initial video frame and the keyframe of the signing action in the video to be identified, the signer is located, avoiding missed detections during the tracking process and eliminating the need for continuous human body detection, thus improving the efficiency and accuracy of signer identification.
[0075] Please see Figure 3 , Figure 3 This embodiment of the present application provides a schematic block diagram of a device for determining a signatory in a video. This device is used to execute the aforementioned method for determining a signatory in a video. The device for determining a signatory in a video can be configured on a server.
[0076] like Figure 3 As shown, the device 300 for determining the signatory in the video includes:
[0077] The speech content acquisition module 301 is used to acquire the video to be recognized and perform speech recognition on the video to be recognized to obtain the speech content of the video to be recognized.
[0078] The target segment extraction module 302 is used to extract a preset number of target video frame segments from the video to be identified based on the speech content.
[0079] The action segment determination module 303 is used to perform action recognition on each of the target video frame segments based on a preset action classification model, and determine the signature action segment.
[0080] The signatory determination module 304 is used to perform human detection on the signing action segment based on a preset human detection model to determine the signatory.
[0081] Furthermore, the target fragment extraction module 302 includes:
[0082] The signature stage start determination unit is used to determine the start frame and end frame of the signature stage in the video to be identified based on the voice content.
[0083] A video frame segment extraction unit is used to extract a preset number of target video frame segments from a target video frame interval based on a preset video frame extraction rule, wherein the target video frame interval is the video frame interval between the start frame and the end frame.
[0084] Furthermore, the signatory determination module 304 includes:
[0085] The human body region determination unit is used to perform human body detection on the intermediate frame image of the signing action segment based on the human body detection model, and determine the human body region of each person in the intermediate frame image.
[0086] The first feature extraction unit is used to take the largest human body region as the target region and extract features from the target region to obtain the first feature.
[0087] The feature set generation unit is used to acquire the person region image of each person in the video to be identified, and to extract features from each person region image to obtain the features corresponding to each person in the video to be identified, and generate a second feature set.
[0088] The signatory determination unit is used to determine the signatory based on the first feature and the second feature set.
[0089] Furthermore, the signatory determination unit includes:
[0090] The distance calculation subunit is used to calculate the distance between the first feature and each feature in the second feature set, respectively.
[0091] The minimum distance determination subunit is used to determine the minimum distance and the feature corresponding to the minimum distance based on the distance between the first feature and each feature in the second feature set, and to take the feature corresponding to the minimum distance as the target feature;
[0092] The signatory determination subunit is used to identify the person corresponding to the target feature as the signatory when the minimum distance is less than a preset distance threshold.
[0093] Furthermore, the device 300 for determining the signatory in the video also includes a model training module, wherein the model training model includes:
[0094] A training video acquisition unit is used to acquire at least one model training video, wherein the model training video includes signed segments and unsigned segments;
[0095] The classification result acquisition unit is used to perform action classification on the training video of the model based on a preset pre-trained model, and obtain action classification results;
[0096] An accuracy acquisition unit is used to obtain the classification accuracy of the pre-trained model based on the action classification results and the signed and unsigned segments in the model training video.
[0097] The classification model determination unit is used to determine the pre-trained model as the action classification model when the classification accuracy is greater than or equal to a preset threshold.
[0098] Furthermore, the voice content acquisition module 301 includes:
[0099] The human body detection unit is used to detect the number of people in the recorded video based on the human body detection model.
[0100] The video to be identified unit is used to identify the recorded video as the video to be identified when the number of people is greater than or equal to a preset threshold number of people.
[0101] Furthermore, the device 300 for determining the signatory in the video also includes an identity verification module, which includes:
[0102] A facial feature acquisition unit is used to acquire the facial features of the signatory;
[0103] The identity information acquisition unit is used to compare the facial features with a preset facial database to obtain the identity information of the signatory, so as to determine whether the signatory is the target person based on the identity information.
[0104] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0105] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 4 It runs on the computer device shown.
[0106] Please see Figure 4 , Figure 4 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.
[0107] See Figure 4 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.
[0108] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause a processor to perform any method for determining the signatory in a video.
[0109] The processor provides computing and control capabilities, supporting the operation of the entire computer device.
[0110] Internal memory provides an environment for the execution of computer programs on non-volatile storage media, which, when executed by a processor, enable the processor to perform any method for determining the signatory in a video.
[0111] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0112] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.
[0113] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:
[0114] Acquire the video to be identified, and perform speech recognition on the video to be identified to obtain the speech content of the video to be identified;
[0115] Based on the voice content, a preset number of target video frame segments are extracted from the video to be identified;
[0116] Based on a preset action classification model, action recognition is performed on each of the target video frame segments to determine the signing action segment.
[0117] Based on a preset human detection model, human detection is performed on the signing action segment to determine the signer.
[0118] In one embodiment, when the processor extracts a preset number of target video frame segments from the video to be recognized based on the speech content, it is configured to:
[0119] Based on the audio content, determine the start and end frames of the signing process in the video to be identified;
[0120] Based on preset video frame extraction rules, a preset number of target video frame segments are extracted from the target video frame interval, wherein the target video frame interval is the video frame interval between the start frame and the end frame.
[0121] In one embodiment, when the processor performs human detection on the signing action segment based on a preset human detection model to determine the signer, it is used to:
[0122] Based on the human detection model, human detection is performed on the intermediate frame images of the signing action segment to determine the human body regions of each person in the intermediate frame images.
[0123] The region containing the largest human body is taken as the target region, and features are extracted from the target region to obtain the first feature;
[0124] The system acquires the person region images of each person in the video to be identified, and extracts features from each person region image to obtain the features corresponding to each person in the video to be identified, thereby generating a second feature set.
[0125] The signatory is determined based on the first feature and the second feature set.
[0126] In one embodiment, when the processor determines the signatory based on the first feature and the second feature set, it is configured to:
[0127] Calculate the distance between the first feature and each feature in the second feature set;
[0128] Based on the distance between the first feature and each feature in the second feature set, the minimum distance and the feature corresponding to the minimum distance are determined, and the feature corresponding to the minimum distance is taken as the target feature;
[0129] When the minimum distance is less than a preset distance threshold, the person corresponding to the target feature is identified as the signatory.
[0130] In one embodiment, before implementing action recognition for each target video frame segment based on a preset action classification model to determine the signature action segment, the processor is further configured to implement:
[0131] Obtain at least one model training video, wherein the model training video includes signed and unsigned segments;
[0132] Based on a pre-trained model, action classification is performed on the training video of the model to obtain action classification results;
[0133] Based on the action classification results and the signed and unsigned segments in the model training video, the classification accuracy of the pre-trained model is obtained;
[0134] When the classification accuracy is greater than or equal to a preset threshold, the pre-trained model is determined as the action classification model.
[0135] In one embodiment, when acquiring the video to be identified, the processor is configured to:
[0136] Based on the human detection model, the number of people in the recorded video is detected;
[0137] When the number of people is greater than or equal to a preset threshold, the recorded video is used as the video to be identified.
[0138] In one embodiment, after the processor performs human detection on the signing action segment based on a preset human detection model to determine the signer, it is further configured to:
[0139] Obtain the facial features of the signatory;
[0140] The facial features are compared with a preset facial database to obtain the identity information of the signatory, and based on the identity information, it is determined whether the signatory is the target person.
[0141] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the methods for determining the signatory in a video provided in the embodiments of this application.
[0142] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.
[0143] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A method for determining the signer in a video. It is characterized in that include: Acquire a video to be identified, and perform voice recognition on the video to be identified to obtain the voice content of the video to be identified; Based on the speech content, extracting a preset number of target video frame segments from the video to be recognized; Based on a preset action classification model, performing action recognition on each of the target video frame segments to determine a signing action segment; Based on a preset human body detection model, human body detection is performed on the signing action clip to determine the signatory.
2. The method for determining the signer in a video according to claim 1, It is characterized in that The step of extracting a preset number of target video frame segments from the video to be recognized based on the voice content includes: Based on the voice content, determining the starting frame and the ending frame of the signature link in the video to be identified; Based on a preset video frame extraction rule, a preset number of the target video frame segments are extracted from a target video frame interval, wherein the target video frame interval is a video frame interval between the start frame and the end frame.
3. The method for determining the signer in a video according to claim 1, It is characterized in that The step of performing human body detection on the signing action segment based on a preset human body detection model to determine the signatory includes: Based on the human body detection model, human body detection is performed on the intermediate frame image of the signing action clip to determine the human body area of each person in the intermediate frame image; The area where the human body with the largest area is located is taken as the target area, and features are extracted from the target area to obtain a first feature; Acquire a character region image of each character in the video to be identified, and perform feature extraction on each character region image to obtain features corresponding to each character in the video to be identified, and generate a second feature set; The signatory is determined based on the first feature and the second feature set.
4. The method for determining the signer in a video according to claim 3, It is characterized in that The determining the signatory based on the first feature and the second feature set includes: Calculating the distance between the first feature and each feature in the second feature set respectively; Based on the distance between the first feature and each feature in the second feature set, determine a minimum distance and a feature corresponding to the minimum distance, and use the feature corresponding to the minimum distance as a target feature; When the minimum distance is less than a preset distance threshold, the person corresponding to the target feature is determined as the signatory.
5. The method for determining the signer in a video according to claim 1, It is characterized in that Before performing action recognition on each of the target video frame segments based on a preset action classification model and determining the signing action segment, the method further includes: Acquire at least one model training video, wherein the model training video includes a signed segment and a non-signed segment; Based on a preset pre-trained model, performing action classification on the model training video to obtain an action classification result; Obtaining a classification accuracy rate of the pre-trained model based on the action classification result and the signature segment and the non-signature segment in the model training video; When the classification accuracy is greater than or equal to a preset threshold, the pre-trained model is determined as the action classification model.
6. The method for determining the signer in a video according to claim 1, It is characterized in that The step of obtaining the video to be identified includes: Based on the human body detection model, detecting the number of people in the recorded video; When the number of persons is greater than or equal to a preset threshold number of persons, the recorded video is used as the video to be identified.
7. The method for determining the signer in a video according to any one of claims 1 to 6, It is characterized in that After performing human body detection on the signing action clip based on a preset human body detection model and determining the signatory, the method further includes: Obtaining facial features of the signatory; The facial features are compared with a preset facial database to obtain the identity information of the signatory, so as to determine whether the signatory is the target person based on the identity information.
8. A device for determining a signatory in a video. It is characterized in that include: A voice content acquisition module is used to acquire a video to be recognized, and perform voice recognition on the video to be recognized to obtain the voice content of the video to be recognized; A target segment extraction module, used to extract a preset number of target video frame segments from the video to be identified based on the voice content; An action segment determination module, used to perform action recognition on each of the target video frame segments based on a preset action classification model to determine a signature action segment; The signatory determination module is used to perform human body detection on the signing action segment based on a preset human body detection model to determine the signatory.
9. A computer device, It is characterized in that The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the method for determining the signatory in the video according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method for determining a signatory in a video according to any one of claims 1 to 7.
