Video shooting scene recognition method, device, equipment and medium
By constructing the expected rotating optical flow field and calculating the residual optical flow field, the coupling interference problem between the device's rotational motion and the actual motion of the scene was solved, enabling accurate scene recognition and improving the conversion rate of financial lending business.
Patent Information
- Application Number
- CN202611216954.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-11
- Publication Date
- 2026-11-03
AI Technical Summary
Existing technologies struggle to effectively distinguish between the device's own rotational motion during video recording and the actual motion of the scene. This can easily lead to misinterpreting a user's handheld device's panoramic rotation in a static environment as a vehicle driving scene, resulting in a low conversion rate for financial lending businesses.
By acquiring the original observed optical flow field and three-dimensional spatial attitude data of the video frame sequence, the expected rotating optical flow field is constructed, and the residual optical flow field is calculated to remove the coupling interference between the device's own rotational motion and the real motion of the scene, thereby achieving accurate scene recognition.
This effectively avoids misinterpreting equipment rotation in a stationary environment as a vehicle driving scenario, thereby improving the business conversion rate of financial lending.
Smart Images
Figure CN123035372A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, device, and medium for recognizing video shooting scenes. Background Technology
[0002] With the deepening of digital transformation in the financial sector, online video interviews have become the mainstream service for banks and consumer finance companies to conduct financial lending business. Users can conveniently complete identity verification, risk control inquiries, and other interview processes through the client, effectively improving the efficiency of financial business processing. In practical applications, there are instances where some users initiate and participate in online interviews while driving. This behavior not only poses traffic safety hazards but also leads to a decline in the quality of interview verification due to user distraction, which in turn causes deviations in business risk assessment. This is a prominent problem that urgently needs to be solved in the field of remote interview risk control, and it also places higher demands on video shooting scene recognition technology so that the interview process can be stopped in time when the shooting scene is identified as a vehicle driving scene, especially a driving scene.
[0003] When performing video shooting scene recognition, the relevant technologies have difficulty effectively distinguishing between the rotational motion of the device shooting the video and the actual motion state of the scene. They are prone to misjudging the user's panoramic rotation behavior in a static environment as a vehicle driving scene, thus incorrectly stopping the face-to-face interview process and resulting in a low business conversion rate for financial lending business. Summary of the Invention
[0004] This invention provides a method, apparatus, computer equipment, and medium for video shooting scene recognition, which can avoid misjudging the rotating view of a device shooting video in a static environment as a vehicle driving scene and thus incorrectly stopping the normal face-to-face interview process, thereby effectively improving the business conversion rate of financial credit business.
[0005] Firstly, a method for recognizing video shooting scenes is provided, including: Acquire a video, the video comprising a sequence of video frames; Determine the original observed optical flow field of adjacent video frames in the video frame sequence to obtain the original observed optical flow field corresponding to each video frame; Obtain the 3D spatial pose data corresponding to each video frame in the video frame sequence; Based on the three-dimensional spatial attitude data and camera intrinsic parameters corresponding to adjacent video frames, the expected rotational optical flow field corresponding to each video frame is determined. The difference between the original observed optical flow field and the expected rotated optical flow field corresponding to each video frame is calculated to obtain the residual optical flow field corresponding to each video frame. Based on the residual optical flow field corresponding to each video frame, the shooting scene of the video is identified, and the identification result of the shooting scene of the video is obtained.
[0006] Secondly, a video shooting scene recognition device is provided, comprising: The first acquisition module is used to acquire video, the video including a video frame sequence; The first determining module is used to determine the original observed optical flow field of adjacent video frames in the video frame sequence, and obtain the original observed optical flow field corresponding to each video frame; The second acquisition module is used to acquire the three-dimensional spatial pose data corresponding to each video frame in the video frame sequence. The second determining module is used to determine the expected rotational optical flow field corresponding to each video frame based on the three-dimensional spatial attitude data and camera intrinsic parameters corresponding to adjacent video frames. The calculation module is used to calculate the difference between the original observed optical flow field and the expected rotated optical flow field corresponding to each video frame, so as to obtain the residual optical flow field corresponding to each video frame. The recognition module is used to identify the shooting scene of the video based on the residual optical flow field corresponding to each video frame, and obtain the recognition result of the shooting scene of the video.
[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described video shooting scene recognition method.
[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described video shooting scene recognition method.
[0009] The above-mentioned video shooting scene recognition method, device, computer equipment, and storage medium can be implemented by acquiring video, which includes a video frame sequence; determining the original observed optical flow field from the previous video frame to the next video frame for adjacent video frames in the video frame sequence, thus obtaining the original observed optical flow field corresponding to each video frame; acquiring the three-dimensional spatial attitude data corresponding to each video frame in the video frame sequence; determining the expected rotational optical flow field corresponding to each video frame based on the three-dimensional spatial attitude data and camera intrinsic parameters; calculating the difference between the original observed optical flow field and the expected rotational optical flow field corresponding to each video frame, thus obtaining the residual optical flow field corresponding to each video frame; and recognizing the shooting scene of the video based on the residual optical flow field corresponding to each video frame, thereby obtaining the recognition result of the shooting scene of the video. This invention introduces three-dimensional spatial attitude data to construct the expected rotational optical flow field, thereby determining the residual optical flow field to remove the coupling interference between the device's own rotational motion and the actual motion of the scene when shooting the video. This can avoid misjudging the device's rotation in a static environment as a vehicle driving scene and incorrectly stopping the normal face-to-face review process, thereby effectively improving the business conversion rate of financial credit business. Attached Figure Description
[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 This is a schematic diagram of an application environment for a video shooting scene recognition method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a video shooting scene recognition method according to an embodiment of the present invention; Figure 3 yes Figure 1 A schematic diagram of a specific implementation of step S40; Figure 4 yes Figure 1 A schematic diagram of a specific implementation method for step S60; Figure 5 This is a schematic diagram of a video shooting scene recognition device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0013] The video shooting scene recognition method provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The client can capture videos and upload them to the server. The videos include a sequence of video frames. The server determines the original observed optical flow field from one video frame to the next for adjacent video frames in the video frame sequence, obtaining the original observed optical flow field corresponding to each video frame. It also acquires the three-dimensional spatial attitude data corresponding to each video frame in the video frame sequence. Based on the three-dimensional spatial attitude data and camera intrinsic parameters, it determines the expected rotational optical flow field corresponding to each video frame. It calculates the difference between the original observed optical flow field and the expected rotational optical flow field corresponding to each video frame, obtaining the residual optical flow field corresponding to each video frame. Based on the residual optical flow field corresponding to each video frame, it identifies the shooting scene of the video, obtaining the identification result of the shooting scene. This invention introduces three-dimensional spatial attitude data to construct the expected rotational optical flow field, thereby determining the residual optical flow field to isolate the coupling interference between the device's own rotational motion and the actual motion of the scene. This avoids misjudging the device's rotation in a static environment as a vehicle driving scene and incorrectly stopping the normal face-to-face review process, thus effectively improving the business conversion rate of financial credit business.
[0014] Those skilled in the art should understand that the above-mentioned video shooting scene recognition method can also be executed entirely locally on the client side, without uploading the video to the server, and the video shooting scene recognition can be completed independently on the client side.
[0015] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a video shooting scene recognition method provided in an embodiment of the present invention includes the following steps: S10: Acquire video, which includes a sequence of video frames.
[0016] The video can be a face-to-face video recording during the financial lending process, a continuous digital video stream captured in real time by the front or rear camera of a computer device such as a smartphone or tablet. It consists of a sequence of video frames composed of multiple consecutive images strictly ordered by timestamps. In the face-to-face audit process for financial lending, the video can be proactively captured by the user's mobile terminal camera through the client-side interface of qualified financial service institutions such as banks and consumer finance companies.
[0017] The video encoding format, resolution, and frame rate can be set according to actual conditions, and no specific restrictions are imposed here. For example, the video encoding format can be set to H.264 / H.265, the resolution to 1280×720, and the frame rate to 25fps to balance recognition accuracy, the processing performance of the device capturing the video, and network transmission bandwidth. After video recording begins, a unique system timestamp can be assigned to each video frame, with timestamp accuracy down to the microsecond level, for subsequent precise synchronization with sensor data.
[0018] Financial lending business refers to a type of financial service business in which institutions such as banks and consumer finance companies with financial service qualifications provide individuals or entities with identity verification, qualification review, credit assessment and online business processing. This includes business types that require compliance verification through online video interviews, such as online loan applications, credit approval, credit card activation and remote qualification verification.
[0019] These types of services generally use remote video interviews to verify identity authenticity, conduct compliance checks on the processing environment, and control business risks. Therefore, it is necessary to identify the shooting scene of the video taken by the client to avoid the safety risks and business risk control deviations caused by users participating in interviews while driving.
[0020] The face-to-face verification process refers to a remote, visual verification process conducted by qualified financial service institutions such as banks and consumer finance companies using real-time video footage captured by their client applications. This process mainly includes video recording, identity verification, environmental monitoring, and risk control inquiries. It is used to determine the authenticity of the user's identity, the compliance of their actions, and the security of the environment in which they conduct business, and is an important remote risk control tool in financial lending.
[0021] S20: Determine the original observed optical flow field of adjacent video frames in the video frame sequence to obtain the original observed optical flow field corresponding to each video frame.
[0022] The original observed optical flow field is a two-dimensional vector field that reflects the apparent motion of all pixels in a video frame between two adjacent video frames. Each pixel corresponds to a two-dimensional optical flow vector (u, v), where u represents the horizontal displacement of the pixel, v represents the vertical displacement of the pixel, the magnitude of the vector represents the magnitude of the pixel's motion, and the direction of the vector represents the direction of the pixel's motion.
[0023] The original observed optical flow field is a hybrid motion field, which contains two completely different types of motion information: one is the global motion generated by the rotation, shaking, and translation of the shooting device itself, which is unrelated to the shooting scene itself; the other is the relative motion generated by the actual movement of objects in the scene, which is the effective information for us to judge the state of the scene.
[0024] In this step, the i-th and (i+1)-th frames in the video frame sequence are taken sequentially as adjacent video frame pairs. The original observed optical flow field from the i-th frame to the (i+1)-th frame is calculated and stored along with the timestamp of the (i+1)-th frame. This process is repeated until the original observed optical flow fields for all video frames in the video frame sequence, excluding the first frame, are obtained. For the first frame in the video frame sequence, since there is no preceding frame as a reference frame for optical flow calculation, the corresponding original observed optical flow field cannot be directly calculated. Therefore, the original observed optical flow field for the first frame can be initialized as a zero vector field. i is a positive integer greater than 0.
[0025] In some embodiments, the original observed optical flow field can be calculated using a preset visual algorithm. Specifically, step S20, which involves determining the original observed optical flow field of adjacent video frames in the video frame sequence, includes the following step S21: S21: Using a preset visual algorithm, calculate the original observed optical flow field of adjacent video frames in the video frame sequence.
[0026] The preset visual algorithm can be a mature optical flow calculation algorithm in the field of computer vision.
[0027] In some embodiments, a suitable algorithm can be automatically selected based on the RAM and processor performance of the device capturing the video. For example, for mid-to-low-end smartphones with less than 4GB of RAM, the Lucas-Kanade sparse optical flow algorithm can be selected. This algorithm calculates the optical flow of only feature points in the image, has a fast calculation speed, low resource consumption, and can guarantee a high real-time processing speed. For high-end smartphones with 4GB or more of RAM, the Farneback dense optical flow algorithm can be selected. This algorithm calculates the optical flow of all pixels, has higher accuracy, and can provide richer motion information.
[0028] For example, in the client application of a certain consumer finance company, the Lucas-Kanade sparse optical flow algorithm is automatically enabled when the user uses a mid-to-low-end mobile phone, and the Farneback dense optical flow algorithm is automatically enabled when the user uses a high-end mobile phone. Users can also manually switch the optical flow algorithm in the client settings according to their own needs.
[0029] In other embodiments, the Lucas-Kanade sparse optical flow algorithm can also be used to calculate the original observed optical flow field of adjacent video frames.
[0030] Step S21 calculates the original observed optical flow field of adjacent video frames using a preset visual algorithm, which can accurately extract the pixel motion features of the video image. It has high computational efficiency and strong anti-interference ability, providing accurate and reliable motion basis data for subsequent motion compensation and shooting scene recognition.
[0031] S30: Obtain the three-dimensional spatial pose data corresponding to each video frame in the video frame sequence.
[0032] The three-dimensional spatial attitude data describes the rotational state of the device capturing the video in three-dimensional space, and is acquired in real time by its built-in inertial measurement unit (IMU). The IMU typically integrates a three-axis gyroscope and a three-axis accelerometer; some high-end devices also include a three-axis magnetometer. The three-axis gyroscope measures the instantaneous angular velocity of the device around three orthogonal axes, with a sampling frequency usually much higher than the video capture frame rate. The three-axis accelerometer measures the linear acceleration and gravitational acceleration of the device, helping to correct for gyroscope integral drift. Specifically, the three-dimensional spatial attitude data includes the device's pitch angle (rotation around the x-axis), roll angle (rotation around the y-axis), and yaw angle (rotation around the z-axis), as well as the corresponding angular velocity and acceleration values. To ensure that the attitude data is perfectly aligned with the video frames in time, a timestamp interpolation synchronization method can be used: for each video frame, linear interpolation is performed on the continuous attitude data sequence acquired by the IMU based on its timestamp to obtain the precise three-dimensional spatial attitude data corresponding to the capture time of that video frame. This synchronization method can control the time synchronization error within a small range, fully meeting the accuracy requirements of optical flow calculation.
[0033] S40: Based on the three-dimensional spatial attitude data and camera intrinsic parameters corresponding to adjacent video frames, determine the expected rotating optical flow field corresponding to each video frame.
[0034] Among them, the expected rotating optical flow field is the core intermediate quantity proposed in this invention. It specifically refers to the optical flow field that should theoretically be observed, which is generated solely by the rotational motion of the device itself. It does not contain any information about the actual motion of the scene.
[0035] Camera intrinsic parameters are the inherent physical parameters of a camera, determined by its optical characteristics. These include focal length (fx, fy), principal point coordinates (cx, cy), radial distortion coefficients (k1, k2, k3), and tangential distortion coefficients (p1, p2). These parameters are typically calibrated by the manufacturer using specialized calibration equipment before the device leaves the factory and stored in the device's read-only memory. Clients can directly read these parameters through the system API.
[0036] The core logic of this step is that the pure rotational motion of the device does not change the depth information of objects in the scene. Therefore, the theoretical displacement of each pixel due to the device's rotation can be accurately calculated through pure geometric transformation. Specifically, firstly, based on the 3D spatial pose data of two adjacent frames, the change in the device's rotation angle between these two frames is calculated. Then, the rotation angle is converted into a 3×3 3D rotation matrix. Finally, combined with the camera intrinsic parameter matrix, the theoretical motion vector of each pixel in the image due to the device's rotation is calculated through projection geometric transformation. The theoretical motion vectors of all pixels together form the expected rotating optical flow field.
[0037] For the first frame in a video frame sequence, there is no preceding reference frame to calculate the relative attitude change. Therefore, the first frame is used as the initial reference frame, with its own relative rotation assumed to be zero. The expected rotational optical flow field under the reference state is generated as the expected rotational optical flow field corresponding to the first video frame. For each frame other than the first frame, the previous frame is used as a reference, and the attitude rotation change between adjacent frames is calculated frame by frame, generating the expected rotational optical flow field corresponding to each frame in sequence. By distinguishing between the reference setting of the first frame and the relative solution of subsequent frames, the expected rotational optical flow field matching its own attitude change in each frame of the video frame sequence can be obtained completely, ensuring the continuity and consistency of the optical flow modeling of the entire video.
[0038] In some embodiments, such as Figure 3 As shown, in step S40, that is, based on the three-dimensional spatial pose data and camera intrinsic parameters corresponding to adjacent video frames, the expected rotational optical flow field of adjacent video frames is determined, and the expected rotational optical flow field corresponding to each video frame is obtained. Specifically, it includes the following steps S41-S42: S41: Solve by integrating the three-dimensional spatial pose data corresponding to adjacent video frames to obtain the three-dimensional rotation matrix corresponding to each video frame.
[0039] For the video frame sequence, the first frame is used as the initial reference frame. Since there is no preceding reference frame, the relative rotation of the first frame is assumed to be zero, and an initial reference 3D rotation matrix is generated as the 3D rotation matrix corresponding to the first video frame. For any two adjacent frames other than the first frame, all gyroscope angular velocity data within the timestamp interval of the two frames are extracted. The trapezoidal numerical integration method is used to solve for the deflection angle of the device around the three axes within this time interval. Then, using the Rodriguez rotation formula, the angle change is converted into a 3×3 inter-frame 3D rotation matrix, obtaining the 3D rotation matrix corresponding to the video frames other than the first frame. To suppress the attitude deviation caused by gyroscope integration drift, this embodiment uses a complementary filtering algorithm to fuse the sensor data from the gyroscope and accelerometer, optimize the attitude calculation accuracy, and further improve the reliability of the rotation matrix.
[0040] S42: Based on the three-dimensional rotation matrix and camera intrinsic parameters corresponding to each video frame, determine the expected rotating optical flow field corresponding to each video frame.
[0041] For the first video frame, the expected rotating optical flow field corresponding to the first video frame is generated based on its corresponding 3D rotation matrix and camera intrinsic parameters. For the remaining video frames, the inter-frame 3D rotation matrix and camera intrinsic parameters are subjected to projective geometric transformation. The pixel offset position caused by the device's own rotation is calculated for any pixel in the image, thereby obtaining the theoretical optical flow vector of a single pixel. The offset calculation is performed on all pixels in the image, and the expected rotating optical flow field corresponding to the remaining video frames is finally generated.
[0042] The above steps rely on multi-sensor data fusion to reduce gyroscope integral drift interference, ensuring that the expected rotational optical flow field corresponding to each video frame can accurately represent the device's own rotational motion, eliminating irrelevant information about the motion of the real scene, and providing a solid data foundation for the reliable calculation of the subsequent residual optical flow field.
[0043] S50: Calculate the difference between the original observed optical flow field and the expected rotating optical flow field corresponding to each video frame to obtain the residual optical flow field corresponding to each video frame.
[0044] The residual optical flow field is obtained by subtracting the original observed optical flow field from the expected rotating optical flow field pixel by pixel. Since the expected rotating optical flow field has accurately quantified all the motion generated by the device's own rotation, the global motion components generated by the device's rotation are completely canceled out after the two are subtracted. The residual optical flow field only retains the relative motion information generated by the actual motion of objects in the scene.
[0045] This step is crucial in solving the misjudgment problem of traditional optical flow methods. For example, when a user rapidly rotates their phone to survey their surroundings in a stationary room, the original observed optical flow field will show the entire scene as being in violent motion. However, the IMU accurately detects the phone's rotation, and the calculated expected rotational optical flow field is almost identical to the original observed optical flow field. The average motion amplitude of the residual optical flow field obtained by subtracting the two is close to 0, thus correctly identifying that the scene is actually stationary. When calculating the difference, a pixel-by-pixel vector subtraction operation can be performed on the optical flow vector, that is, for each pixel, the residual optical flow vector = original optical flow vector - expected rotational optical flow vector.
[0046] S60: Based on the residual optical flow field corresponding to each video frame, the shooting scene of the video is identified to obtain the identification result of the shooting scene of the video.
[0047] This step analyzes the motion features of the residual optical flow field of each video frame, and relies on the real motion information of the scene retained only in the residual optical flow field to identify the shooting scene of the video and output the corresponding scene recognition result.
[0048] In some embodiments, in step S60, such as Figure 4 As shown, based on the residual optical flow field corresponding to each video frame, the shooting scene of the video is identified to obtain the identification result of the shooting scene of the video, specifically including the following steps S61-S62: S61: Divide each video frame into multiple categories of regions to be identified, and extract the regional residual optical flow field corresponding to each category of regions to be identified from the residual optical flow field corresponding to each video frame.
[0049] Among them, the multiple regions to be identified are segmented into semantic regions of video frames based on the semantic features of the face-to-face interview scenario and the needs of risk control business, forming several target regions with independent semantic attributes. Different semantic regions have different motion characteristics and risk control value. Motion feature analysis is carried out independently by region, which can shield irrelevant background motion interference and improve the accuracy of shooting scene recognition.
[0050] The regional residual optical flow field is a local optical flow component in the residual optical flow field that corresponds one-to-one with the spatial location of a single type of region to be identified. It only contains the residual optical flow vector corresponding to all pixels within that region. The specific extraction method is as follows: Based on the region division results, a binary mask image of each type of region to be identified is generated. Pixels in the mask image corresponding to a region are marked as 1, and pixels in non-corresponding regions are marked as 0. The binary mask image and the residual optical flow field are multiplied pixel by pixel to separate the regional residual optical flow fields corresponding to each type of region to be identified.
[0051] S62: Based on the regional residual optical flow field corresponding to each type of region to be identified in each video frame, the shooting scene of the video is identified to obtain the identification result of the shooting scene of the video.
[0052] This step combines the regional residual optical flow field motion features of multiple regions to be identified, matches the semantic attributes and motion correlations of each region, comprehensively determines the shooting scene of the video, and outputs the final recognition result.
[0053] The above steps S61-S62, by dividing the video frames into multi-category semantic regions and extracting the independent residual optical flow fields of each region for analysis, can reduce the probability of misjudgment caused by irrelevant motion while ensuring the effectiveness of risk control detection and reduce the situation where the compliance review process is interrupted for no reason.
[0054] In some embodiments, the multiple types of regions to be identified include the vehicle exterior region, the vehicle interior region, and the vehicle steering wheel region.
[0055] The video frame includes three areas: the vehicle interior area, which is the view of the vehicle's interior space, including components such as seats, dashboard, door trim, and roof; the vehicle exterior area, which is the view of the external environment outside the windows, including road, vegetation, buildings, traffic signs, and other scene elements; and the steering wheel area, which is the view of the steering wheel and its surrounding components. The motion characteristics of these three areas directly reflect the vehicle's driving status and the user's position inside the vehicle, serving as the core criterion for distinguishing between driving and passenger scenarios.
[0056] Step S62, which is to identify the shooting scene of the video based on the regional residual optical flow field corresponding to each type of region to be identified in each video frame, and obtain the identification result of the shooting scene of the video, specifically includes the following steps S621-S622: S621: If at least one video frame contains a vehicle steering wheel area, the average motion amplitude of the residual optical flow field corresponding to the vehicle interior area of at least one video frame is less than a first preset threshold, and the average motion amplitude of the residual optical flow field corresponding to the vehicle exterior area of at least one video frame is greater than a second preset threshold, then the video shooting scene is determined to be a driving scene.
[0057] S622: If there is no vehicle steering wheel area in all video frames, the average motion amplitude of the regional residual optical flow field corresponding to the vehicle interior area in at least one video frame is less than the first preset threshold, and the average motion amplitude of the regional residual optical flow field corresponding to the vehicle exterior area in at least one video frame is greater than the second preset threshold, then the shooting scene of the video is determined to be a vehicle riding scene.
[0058] The first preset threshold measures whether the device capturing the video and the interior space of the vehicle remain relatively stationary; the second preset threshold determines whether the vehicle as a whole is in motion. Both the first and second preset thresholds can be empirical thresholds verified through extensive sample testing in financial interview scenarios, or they can be adaptively fine-tuned through sample statistics based on business risk control strategies. Financial institutions can adapt them as needed: when risk control strategies become stricter, the thresholds can be appropriately lowered to prioritize the prevention of high-risk behaviors; when user experience is emphasized, the thresholds can be appropriately raised to reduce false interceptions of compliant business.
[0059] For example, in a financial interview scenario, the first preset threshold can be set to 3 pixels and the second preset threshold can be set to 20 pixels. The specific values can be calibrated and optimized based on sample data of different vehicle models and different driving speeds.
[0060] The logic for distinguishing between driving and passenger scenarios is as follows: Driving scene detection logic: If at least one frame in the video contains the steering wheel area, it indicates that the user is in the vicinity of the driver's seat; if the average motion amplitude of the vehicle's interior area is less than a first preset threshold, it indicates that the shooting device and the vehicle are synchronously stationary; if the average motion amplitude of the vehicle's exterior area is greater than a second preset threshold, it indicates that the vehicle is in motion. Setting the "at least one frame" criterion avoids missed detections due to the steering wheel briefly moving out of the frame caused by the user's slight adjustments to the device's posture, thus improving the sensitivity of high-risk scene detection.
[0061] The logic for determining the passenger scenario is as follows: If there is no steering wheel area in any video frame, it means that the user is in the front passenger seat or the rear non-driving seat; the motion amplitude characteristics of the vehicle interior and exterior areas are consistent with the driving scenario, indicating that the vehicle is in a normal driving state, and it can be determined as a compliant passenger scenario.
[0062] The above steps S621-S622, through the combination constraint logic of motion features of three types of semantic regions, achieve accurate differentiation between driving scenarios and passenger scenarios. This not only strictly controls the illegal behavior of participating in face-to-face interviews during the driving process, but also avoids the erroneous interception of normal passenger face-to-face interview business, reduces unnecessary business loss, and improves the conversion rate of financial credit business and user processing experience.
[0063] In some embodiments, the video is a video frame captured during the face-to-face interview process, and the method further includes: If the filming scene is a driving scene, the current interview process will be terminated.
[0064] Face-to-face verification is a crucial risk control step for financial institutions in preventing credit fraud and verifying qualifications. It applies to various remote video verification services, including personal consumer loans, business loans, and online credit card applications. During the face-to-face verification process, users use the camera of a recording device to capture real-time footage of themselves and their surroundings to verify the compliance of the scene.
[0065] When a video is detected to be taken while driving, a standardized risk control procedure is executed: immediately stop the current real-time video interaction and disconnect the communication link with the client; display a compliance prompt to the user's client, informing the user that they are prohibited from participating in the face-to-face interview while driving and need to stop the vehicle and re-submit the application; at the same time, upload the user's identifier, the time of occurrence, the scene identification result, and key video clips of this face-to-face interview event to the financial risk control backend for archiving, in order to meet the requirements of business risk control and regulatory retention.
[0066] For example, the bank's remote face-to-face interview server can automatically hang up the video call within 100 milliseconds after detecting a driving scenario, mark the behavior as a high-risk operation, and restrict the user from repeatedly initiating face-to-face interview applications within a short period of time.
[0067] This step deeply integrates scene recognition technology with financial credit risk control processes, enabling automatic identification and real-time closed-loop handling of high-risk driving behaviors. This not only avoids traffic safety hazards but also prevents issues such as inaccurate face-to-face verification and biased credit risk assessment caused by user distraction, thereby improving the overall level of intelligent risk control in financial credit business.
[0068] In some embodiments, step S61, which involves dividing each video frame into multiple categories of regions to be identified, specifically includes the following step S611: S611: Use a preset semantic segmentation model to perform semantic segmentation processing on each video frame, dividing each video frame into the vehicle interior area, the vehicle exterior area, and the vehicle steering wheel area.
[0069] The preset semantic segmentation model is a lightweight deep learning model specifically trained for in-vehicle inspection scenarios. It takes a single frame of RGB video image as input and outputs a semantic segmentation feature map with the same resolution as the input image. Each pixel in the segmentation feature map corresponds to a preset category label, with 0, 1, 2, and 3 representing the background area, the vehicle interior area, the vehicle exterior area, and the vehicle steering wheel area, respectively.
[0070] To adapt to the computational constraints of the devices used for video recording, the model can adopt the lightweight YOLO-Seg network architecture. The training dataset covers mainstream vehicle types such as sedans, SUVs, and MPVs, and includes various lighting environments and shooting angles, including daytime, nighttime, and rainy conditions. During the training phase, a transfer learning strategy is adopted, first completing pre-training on a general dataset, and then using labeled samples of in-vehicle scenes for fine-tuning and optimization to improve the model's generalization ability and segmentation accuracy.
[0071] In practical applications, when a user turns on the camera of the device used to record video to complete the environmental surround view shooting as required by the face-to-face interview, the client inputs the single-frame image captured in real time into the preset semantic segmentation model. After the model outputs the pixel-level semantic segmentation results, it generates binary mask images corresponding to three types of regions: vehicle interior, vehicle exterior, and vehicle steering wheel, respectively, which provides support for the accurate extraction of the residual optical flow field in the subsequent regions.
[0072] It should be noted that the semantic segmentation model outputs a multi-class labeled feature map with values of 0, 1, 2, and 3, used for pixel-level region classification. Subsequently, based on this multi-class labeled feature map, a separate binary mask is generated for each category of region. In this binary mask, only a pixel value of 1 represents the current target region, and a pixel value of 0 represents a non-target region. Using independent binary masks and residual optical flow fields for pixel-by-pixel calculations preserves only the true optical flow amplitude of the target region, avoiding the distortion caused by direct multi-class labeled values in the calculations, thus ensuring the accuracy of subsequent motion amplitude statistics and scene recognition.
[0073] Step S611: Through lightweight deep learning semantic segmentation technology, automatic high-precision segmentation of multiple semantic regions in video frames is achieved without manual labeling or intervention. The video can be processed in real time and has good robustness to interference such as changes in light, partial occlusion, and shooting angle shift, laying a reliable foundation for subsequent regional optical flow feature analysis and scene recognition.
[0074] In this embodiment of the invention, a video is acquired, comprising a sequence of video frames. For adjacent video frames in the video frame sequence, the original observed optical flow field from the previous video frame to the next video frame is determined, resulting in the original observed optical flow field corresponding to each video frame. Three-dimensional spatial attitude data corresponding to each video frame in the video frame sequence is acquired. Based on the three-dimensional spatial attitude data and camera intrinsic parameters corresponding to each video frame, the expected rotational optical flow field corresponding to each video frame is determined. The difference between the original observed optical flow field and the expected rotational optical flow field corresponding to each video frame is calculated, resulting in the residual optical flow field corresponding to each video frame. Based on the residual optical flow field corresponding to each video frame, the shooting scene of the video is identified, resulting in the identification result of the shooting scene. This invention introduces three-dimensional spatial attitude data to construct the expected rotational optical flow field, thereby determining the residual optical flow field to isolate the coupling interference between the device's own rotational motion and the actual motion of the scene. This avoids misjudging the device's rotation in a static environment as a vehicle driving scene, thus incorrectly stopping the normal face-to-face interview process, and effectively improving the business conversion rate of financial lending business.
[0075] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0076] In one embodiment, a video shooting scene recognition device is provided, which corresponds one-to-one with the video shooting scene recognition method in the above embodiments. For example... Figure 5 As shown, the video shooting scene recognition device includes a first acquisition module 101, a first determination module 102, a second acquisition module 103, a second determination module 104, a calculation module 105, and a recognition module 106. Detailed descriptions of each functional module are as follows: The first acquisition module 101 is used to acquire a video, the video including a video frame sequence; The first determining module 102 is used to determine the original observed optical flow field of adjacent video frames in the video frame sequence, and obtain the original observed optical flow field corresponding to each video frame. The second acquisition module 103 is used to acquire the three-dimensional spatial pose data corresponding to each video frame in the video frame sequence. The second determining module 104 is used to determine the expected rotational optical flow field corresponding to each video frame based on the three-dimensional spatial attitude data and camera intrinsic parameters corresponding to adjacent video frames. The calculation module 105 is used to calculate the difference between the original observed optical flow field and the expected rotating optical flow field corresponding to each video frame, so as to obtain the residual optical flow field corresponding to each video frame. The recognition module 106 is used to recognize the shooting scene of the video based on the residual optical flow field corresponding to each video frame, and obtain the recognition result of the shooting scene of the video.
[0077] In one embodiment, the identification module 106 is specifically used for: Each video frame is divided into multiple types of regions to be identified, and the region residual optical flow field corresponding to each type of region to be identified in each video frame is extracted from the residual optical flow field corresponding to each video frame. Based on the regional residual optical flow field corresponding to each type of region to be identified in each video frame, the shooting scene of the video is identified, and the identification result of the shooting scene of the video is obtained.
[0078] In one embodiment, the multiple types of regions to be identified include the vehicle exterior region, the vehicle interior region, and the vehicle steering wheel region; the identification module 106 is specifically used for: If at least one video frame has a non-empty vehicle steering wheel area, the average motion amplitude of the residual optical flow field corresponding to the vehicle interior area of at least one video frame is less than a first preset threshold, and the average motion amplitude of the residual optical flow field corresponding to the vehicle exterior area of at least one video frame is greater than a second preset threshold, then the recognition result is determined to be that the shooting scene of the video is a driving scene. If the vehicle steering wheel area in each video frame is empty, the average motion amplitude of the residual optical flow field corresponding to the vehicle interior area in at least one video frame is less than a first preset threshold, and the average motion amplitude of the residual optical flow field corresponding to the vehicle exterior area in at least one video frame is greater than a second preset threshold, then the recognition result is determined to be that the shooting scene of the video is a riding scene.
[0079] In one embodiment, the video is a video taken during the face-to-face interview process, and the video shooting scene recognition device further includes a stop module, which is specifically used for: If the shooting scene is a driving scene, then the face-to-face interview process will be stopped.
[0080] In one embodiment, the identification module 106 is specifically used for: A pre-defined semantic segmentation model is used to perform semantic segmentation on each video frame, dividing each video frame into multiple regions to be identified.
[0081] In one embodiment, the first determining module 102 is specifically used for: Using a pre-defined visual algorithm, the original observed optical flow field of adjacent video frames in a video frame sequence is calculated.
[0082] In one embodiment, the second determining module 104 is specifically used for: The three-dimensional rotation matrix corresponding to each video frame is obtained by integral calculation based on the three-dimensional spatial pose data corresponding to adjacent video frames. Based on the three-dimensional rotation matrix and camera intrinsic parameters corresponding to each video frame, the expected rotating optical flow field corresponding to each video frame is determined.
[0083] This invention provides a video shooting scene recognition device. It introduces three-dimensional spatial posture data to construct a expected rotating optical flow field, thereby determining the residual optical flow field to isolate the coupling interference between the device's own rotational motion and the actual motion of the scene. This can avoid misjudging the device's rotation in a static environment as a vehicle driving scene and incorrectly stopping the normal face-to-face review process, thereby effectively improving the business conversion rate of financial credit business.
[0084] Specific limitations regarding the video shooting scene recognition device can be found in the limitations of the intelligent question-answering method described above, and will not be repeated here. Each module in the aforementioned video shooting scene recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0085] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a video shooting scene recognition method on the server side.
[0086] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the client-side functions or steps of a video shooting scene recognition method. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire a video, the video comprising a sequence of video frames; Determine the original observed optical flow field of adjacent video frames in the video frame sequence to obtain the original observed optical flow field corresponding to each video frame; Obtain the 3D spatial pose data corresponding to each video frame in the video frame sequence; Based on the three-dimensional spatial attitude data and camera intrinsic parameters corresponding to adjacent video frames, the expected rotational optical flow field corresponding to each video frame is determined. The difference between the original observed optical flow field and the expected rotated optical flow field corresponding to each video frame is calculated to obtain the residual optical flow field corresponding to each video frame. Based on the residual optical flow field corresponding to each video frame, the shooting scene of the video is identified, and the identification result of the shooting scene of the video is obtained.
[0087] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire a video, the video comprising a sequence of video frames; Determine the original observed optical flow field of adjacent video frames in the video frame sequence to obtain the original observed optical flow field corresponding to each video frame; Obtain the 3D spatial pose data corresponding to each video frame in the video frame sequence; Based on the three-dimensional spatial attitude data and camera intrinsic parameters corresponding to adjacent video frames, the expected rotational optical flow field corresponding to each video frame is determined. The difference between the original observed optical flow field and the expected rotated optical flow field corresponding to each video frame is calculated to obtain the residual optical flow field corresponding to each video frame. Based on the residual optical flow field corresponding to each video frame, the shooting scene of the video is identified, and the identification result of the shooting scene of the video is obtained.
[0088] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0091] It should be noted that if any AI models, software tools, or components not belonging to this company appear in the embodiments of this invention, they are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this invention is authorized (with the knowledge and consent) by the relevant parties or fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.
[0092] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A method for recognizing video shooting scenes, characterized in that, include: Acquire a video, the video comprising a sequence of video frames; Determine the original observed optical flow field of adjacent video frames in the video frame sequence to obtain the original observed optical flow field corresponding to each video frame; Obtain the 3D spatial pose data corresponding to each video frame in the video frame sequence; Based on the three-dimensional spatial attitude data and camera intrinsic parameters corresponding to adjacent video frames, the expected rotating optical flow field corresponding to each video frame is determined. The difference between the original observed optical flow field and the expected rotated optical flow field corresponding to each video frame is calculated to obtain the residual optical flow field corresponding to each video frame. Based on the residual optical flow field corresponding to each video frame, the shooting scene of the video is identified, and the identification result of the shooting scene of the video is obtained.
2. The video shooting scene recognition method according to claim 1, characterized in that, The process of identifying the shooting scene of the video based on the residual optical flow field corresponding to each video frame, and obtaining the identification result of the shooting scene of the video, includes: Each video frame is divided into multiple types of regions to be identified, and the region residual optical flow field corresponding to each type of region to be identified in each video frame is extracted from the residual optical flow field corresponding to each video frame. Based on the regional residual optical flow field corresponding to each type of region to be identified in each video frame, the shooting scene of the video is identified, and the identification result of the shooting scene of the video is obtained.
3. The video shooting scene recognition method according to claim 2, characterized in that, The multiple types of regions to be identified include the vehicle exterior region, the vehicle interior region, and the vehicle steering wheel region; the video shooting scene is identified based on the regional residual optical flow field corresponding to each type of region to be identified in each video frame, resulting in the identification result of the video shooting scene, including: If at least one video frame has a non-empty vehicle steering wheel area, the average motion amplitude of the residual optical flow field corresponding to the vehicle interior area of at least one video frame is less than a first preset threshold, and the average motion amplitude of the residual optical flow field corresponding to the vehicle exterior area of at least one video frame is greater than a second preset threshold, then the recognition result is determined to be that the shooting scene of the video is a driving scene. If the vehicle steering wheel area in each video frame is empty, the average motion amplitude of the residual optical flow field corresponding to the vehicle interior area in at least one video frame is less than a first preset threshold, and the average motion amplitude of the residual optical flow field corresponding to the vehicle exterior area in at least one video frame is greater than a second preset threshold, then the recognition result is determined to be that the shooting scene of the video is a riding scene.
4. The video shooting scene recognition method according to claim 3, characterized in that, The video is a video taken during the face-to-face interview process, and the method further includes: If the shooting scene is a driving scene, then the face-to-face interview process will be stopped.
5. The video shooting scene recognition method according to claim 2, characterized in that, The process of dividing each video frame into multiple categories of regions to be identified includes: A pre-defined semantic segmentation model is used to perform semantic segmentation on each video frame, dividing each video frame into multiple regions to be identified.
6. The video shooting scene recognition method according to claim 1, characterized in that, Determining the original observed optical flow field of adjacent video frames in a video frame sequence includes: Using a pre-defined visual algorithm, the original observed optical flow field of adjacent video frames in a video frame sequence is calculated.
7. The video shooting scene recognition method according to claim 1, characterized in that, The determination of the expected rotational optical flow field for each video frame based on the 3D spatial pose data and camera intrinsic parameters corresponding to adjacent video frames includes: The three-dimensional rotation matrix corresponding to each video frame is obtained by integral calculation based on the three-dimensional spatial pose data corresponding to adjacent video frames. Based on the three-dimensional rotation matrix and camera intrinsic parameters corresponding to each video frame, the expected rotating optical flow field corresponding to each video frame is determined.
8. A video shooting scene recognition device, characterized in that, include: The first acquisition module is used to acquire video, the video including a video frame sequence; The first determining module is used to determine the original observed optical flow field of adjacent video frames in the video frame sequence, and obtain the original observed optical flow field corresponding to each video frame; The second acquisition module is used to acquire the three-dimensional spatial pose data corresponding to each video frame in the video frame sequence. The second determining module is used to determine the expected rotational optical flow field corresponding to each video frame based on the three-dimensional spatial attitude data and camera intrinsic parameters corresponding to adjacent video frames. The calculation module is used to calculate the difference between the original observed optical flow field and the expected rotated optical flow field corresponding to each video frame, so as to obtain the residual optical flow field corresponding to each video frame. The recognition module is used to identify the shooting scene of the video based on the residual optical flow field corresponding to each video frame, and obtain the recognition result of the shooting scene of the video.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the video shooting scene recognition method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the video shooting scene recognition method as described in any one of claims 1 to 7.