User safety detection method, apparatus, device, and medium

Abstract

The application relates to the technical field of image processing, and discloses a user safety detection method and device, equipment and a medium, which comprises the following steps: acquiring a face interview video stream of a user in real time; extracting first video frames from the face interview video stream according to a preset time interval, and detecting the video concentration state of the user in each second video frame obtained from the face interview video stream; when a preset safety detection condition is met, inputting the extracted first video frames and a preset prompt word into a trained scene analysis model to obtain a scene analysis result; acquiring the video concentration state in each second video frame detected in a historical period; and determining the safety state of the user based on the scene analysis result and each video concentration state. The application can be applied to the user safety detection scene of financial technology, and improves the recognition accuracy of user safety detection.

Description

Technical Field

[0001] This application relates to the field of image processing technology and is applied to financial technology scenarios, particularly to a user security detection method, device, equipment, and medium. Background Technology

[0002] In remote video interviews conducted by financial institutions, users typically interact with customer service via their mobile phone's front-facing camera. However, some users, in an effort to save time, conduct these interviews while driving. This practice greatly increases the risk of traffic accidents. Therefore, accurately and in real-time identifying whether a user is in a dangerous situation while driving during the interview is a critical pain point that urgently needs to be addressed in financial risk control systems.

[0003] In related technologies, it is only possible to identify whether a user is in a dangerous situation by recognizing the user's facial orientation or gaze deviation. However, this recognition method may result in misjudgments, such as a user looking down at documents and being mistakenly identified as being in a dangerous situation, resulting in low recognition accuracy. Summary of the Invention

[0004] This application provides an artificial intelligence-based user security detection method, apparatus, computer equipment, and medium, which aims to avoid misjudgment problems during user security detection, improve the accuracy of security detection during face-to-face interviews, and especially protect the security of users during face-to-face interviews in the financial sector.

[0005] Firstly, a user security detection method is provided, including: Real-time acquisition of user interview video streams; The first video frame is extracted from the face-to-face video stream at a preset time interval, and the user's video focus state in each second video frame obtained from the face-to-face video stream is detected. When the preset security detection conditions are met, the extracted first video frames and preset prompt words are input into the trained scene analysis model to obtain the scene analysis results. Obtain the video focus state in each second video frame detected within a historical time period; Based on the scene analysis results and the focus state of each video, the user's safety status is determined.

[0006] Secondly, a user security detection device is provided, comprising: The first acquisition module is used to acquire the user's face-to-face video stream in real time; The processing module is used to extract a first video frame from the face-to-face video stream at a preset time interval, and to detect the user's video focus state in each second video frame obtained from the face-to-face video stream. The analysis module is used to input multiple extracted first video frames and preset prompt words into the trained scene analysis model when the preset security detection conditions are met, and obtain the scene analysis results. The second acquisition module is used to acquire the video focus state in each second video frame detected within the historical time period; The determination module is used to determine the user's safety status based on the scene analysis results and the focus state of each video.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the aforementioned user security detection method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the aforementioned user security detection method.

[0009] The aforementioned user security detection method, apparatus, computer equipment, and storage medium can acquire the user's face-to-face video stream in real time; extract first video frames from the face-to-face video stream at preset time intervals, and detect the user's video focus state in each second video frame acquired from the face-to-face video stream; when preset security detection conditions are met, input the extracted first video frames and preset prompts into a trained scene analysis model to obtain scene analysis results; acquire the video focus state in each second video frame detected within a historical time period; and determine the user's security status based on the scene analysis results and each video focus state. In this application, the user scenario is analyzed by combining first video frames extracted at different time periods, thereby obtaining scene analysis results used as contextual information for subsequent security status detection. The user's security status is then comprehensively determined by combining the scene analysis results and the video focus state of each second video frame. This avoids misjudgment during user security detection, improves the accuracy of security detection during face-to-face interviews, and particularly ensures the security of users during face-to-face interviews in the financial sector. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a user security detection method in one embodiment of this application; Figure 2 This is a flowchart illustrating a user security detection method in one embodiment of this application; Figure 3 yes Figure 1 A schematic diagram of a specific implementation method for step S20; Figure 4 yes Figure 1 A flowchart illustrating another specific implementation of step S22; Figure 5 yes Figure 1 A schematic diagram of a specific implementation method for step S30; Figure 6 yes Figure 1 A schematic diagram of a specific implementation method for step S50; Figure 7 yes Figure 6 A flowchart illustrating a specific implementation of step S53; Figure 8 This is a schematic diagram of the user security detection device in one embodiment of this application; Figure 9 This is a schematic diagram of the structure of a computer device according to one embodiment of this application; Figure 10 This is another structural schematic diagram of a computer device in one embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The user security detection method provided in this application embodiment can be applied to, for example, Figure 1In this application environment, the client communicates with the server via a network. The server can obtain the real-time video stream of the user's face-to-face interview published by the client; extract the first video frame from the face-to-face interview video stream at preset time intervals, and detect the user's video focus state in each second video frame obtained from the face-to-face interview video stream; when preset security detection conditions are met, input the extracted first video frames and preset prompt words into the trained scene analysis model to obtain scene analysis results; obtain the video focus state in each second video frame detected within a historical time period; and determine the user's security status based on the scene analysis results and each video focus state. In this application, the user scene is analyzed by combining the first video frames extracted at different time periods, thereby obtaining the scene analysis results used as context information for subsequent security status detection. The user's security status is then comprehensively determined by combining the scene analysis results and the video focus state of each second video frame. This avoids misjudgment during user security detection, improves the accuracy of security detection during face-to-face interviews, and especially ensures the security of users during face-to-face interviews in the financial sector.

[0014] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the user security detection method provided in this application embodiment includes the following steps: S10: Real-time acquisition of the user's face-to-face video stream.

[0015] In this context, the face-to-face video stream refers to a continuous sequence of video data captured by the camera of a user's terminal device (such as a mobile phone or tablet) and transmitted in real time to the server during remote video face-to-face interviews (such as loan interviews, insurance claims interviews, and identity verification). The video stream includes the user's facial features, behavioral actions, and information about the surrounding environment.

[0016] S20: Extract the first video frame from the face-to-face video stream according to a preset time interval, and detect the user's video focus state in each second video frame obtained from the face-to-face video stream.

[0017] The preset time interval refers to a pre-defined time interval parameter used to extract video frames from the face-to-face video stream for scene analysis. This parameter can be flexibly configured according to business needs, computing resources, and detection accuracy requirements to control the frequency of scene analysis and balance detection real-time performance with computing costs. The first video frame refers to the key frame extracted from the face-to-face video stream according to the preset time interval for subsequent global scene analysis. The extraction frequency of the first video frame is significantly lower than the native frame rate of the video stream, mainly used to provide temporal contextual information for the scene analysis model. The second video frame refers to the video frame obtained from the face-to-face video stream for real-time detection of the user's video focus state. The processing frequency of the second video frame is consistent with the native frame rate of the video stream to ensure continuous tracking of changes in the user's focus state. Video focus state refers to the degree of user attention on the video interaction interface during video interaction. It is a core indicator reflecting whether the user is focused on the current video business. Video focus state is usually divided into two categories: focused state (user's attention is focused on the video interaction interface) and unfocused state (user's attention is deviated from the video interaction interface).

[0018] Specifically, this step employs a dual-stream parallel processing architecture, where two computational streams run independently and do not block each other: First video frame extraction stream: A separate background thread is started to extract the first video frame from the face-to-face video stream at preset time intervals, and the extracted frame is temporarily stored in a dedicated frame buffer. When the number of first video frames stored in the buffer reaches a preset value, the earliest stored frame is automatically overwritten, so that the buffer always contains the latest consecutive multi-frame data.

[0019] Second video frame focus state detection stream: The main thread obtains each second video frame at the native frame rate of the face-to-face video stream, performs video focus state detection on each frame, obtains the video focus state result corresponding to the frame, and stores the result in the historical state cache after associating it with the timestamp.

[0020] In some embodiments, such as Figure 3 As shown, step S20 involves detecting the user's video focus state in each second video frame obtained from the interview video stream, including the following steps S21 to S23: S21: Determine the user's head yaw angle in each second video frame obtained from the face-to-face video stream.

[0021] The user's head yaw angle refers to the angle at which the user's head rotates left and right around the vertical axis of the body (vertical direction), and is a core parameter describing the head's horizontal posture. With the user looking straight ahead (towards the phone screen) as the 0° reference, leftward rotation is a negative angle, and rightward rotation is a positive angle. The larger the absolute value of the angle, the greater the deviation of the head from the direction of direct gaze.

[0022] Specifically, for each second video frame of the input, a lightweight pre-trained model (such as MediaPipeFace Mesh or lightweight CNN) is invoked on the device to quickly detect and extract key feature points of the user's face (such as the corners of the eyes, the tip of the nose, the corners of the mouth, and the contours of the cheeks). Based on the extracted 2D facial key points, combined with a general 3D face model, the 3D pose parameters of the user's head are solved using the PnP (Perspective-n-Point) algorithm, from which the specific value of the head yaw angle is extracted.

[0023] S22: When the head yaw angle is greater than or equal to the preset yaw angle, the video focus state is determined to be non-focus state.

[0024] The preset yaw angle refers to a pre-set threshold parameter used to determine whether the user's head has deviated excessively from the screen orientation. This parameter can be flexibly configured according to business needs and detection sensitivity requirements. When the absolute value of the user's head yaw angle reaches or exceeds this threshold, it is determined that the user's attention has deviated from the video interaction interface.

[0025] Specifically, if the absolute value of the head yaw angle is greater than or equal to the preset yaw angle, the video focus state of the current frame is determined to be non-focused.

[0026] S23: When the head yaw angle is less than the preset yaw angle, the video focus state is determined to be a focus state.

[0027] If the absolute value of the head yaw angle is less than the preset yaw angle, then the current frame's video focus state is determined to be a focus state.

[0028] In this way, a lightweight pre-trained model on the device side is used to detect facial key points, and the PnP algorithm is used to quantify and calculate the user's head yaw angle. Then, based on a preset configurable angle threshold, the focus state of a single frame of video is quickly determined. The entire process is executed entirely on the device side, with extremely short single-frame inference time. It can run in real time at the native frame rate of the video stream without blocking video rendering and streaming. This ensures the real-time performance of high-frequency local behavior tracking and avoids the privacy leakage risk caused by uploading user facial images to the cloud. At the same time, the use of quantified angle indicators for determination results in an objective and stable manner. This provides reliable single-frame basic data for the subsequent statistics of the proportion of non-focused states in historical periods and the final determination of the safety state. Moreover, it has low requirements for terminal device performance and has wide adaptability.

[0029] In some embodiments, such as Figure 4 As shown, step S20 involves detecting the user's video focus state in each second video frame obtained from the interview video stream, including the following steps S24 to S26: S24: Determine the user's head pitch angle in each second video frame obtained from the face-to-face video stream.

[0030] The user's head pitch angle refers to the angle at which the user's head rotates up and down around the horizontal axis of the human body (left and right direction), and is a core parameter describing the vertical posture of the head. With the user looking straight ahead (towards the phone screen) as the 0° reference, looking down is a negative angle, and looking up is a positive angle. The larger the absolute value of the angle, the greater the deviation of the head from the direction of looking straight ahead.

[0031] Specifically, for each second video frame of the input, a lightweight pre-trained model (such as MediaPipeFace Mesh or lightweight CNN) is invoked on the device to quickly detect and extract key feature points of the user's face (such as the corners of the eyes, the tip of the nose, the corners of the mouth, and the contours of the cheeks). Based on the extracted 2D facial key points, combined with a general 3D face model, the 3D pose parameters of the user's head are solved using the PnP (Perspective-n-Point) algorithm, from which the specific values ​​of the head pitch angle are extracted.

[0032] S25: When the head pitch angle is greater than or equal to the preset pitch angle, the video focus state is determined to be a non-focus state.

[0033] The preset pitch angle refers to a pre-set threshold parameter used to determine whether the user's head has deviated excessively from the screen orientation. This parameter can be flexibly configured according to business needs and detection sensitivity requirements. When the absolute value of the user's head pitch angle reaches or exceeds this threshold, it is determined that the user's attention has deviated from the video interaction interface.

[0034] Specifically, if the absolute value of the head pitch angle is greater than or equal to the preset pitch angle, the current frame's video focus state is determined to be non-focused.

[0035] S26: When the head yaw angle is less than the preset pitch angle, the video focus state is determined to be a focus state.

[0036] If the absolute value of the head pitch angle is less than the preset pitch angle, the current frame's video focus state is determined to be a focus state.

[0037] Therefore, by combining a lightweight pre-trained model on the device with the PnP algorithm, the pitch angle of the user's head in the vertical direction is quantitatively calculated, and the single-frame focus state is determined based on a configurable threshold. This not only supplements the head rotation distraction scenarios not covered by horizontal yaw angle detection, further improving the comprehensiveness and accuracy of focus state detection, but also continues the advantages of device-side computing. The single-frame inference time is extremely short, supporting real-time operation at the native frame rate of the video stream without affecting video rendering and streaming. Moreover, it does not require uploading user facial data to the cloud, effectively protecting user privacy. Its quantitative and objective judgment results and flexible threshold configuration capabilities also provide reliable vertical dimension behavioral data support for subsequent statistics on the proportion of non-focused states in historical periods and the final fusion judgment of security states, and have broad terminal adaptability.

[0038] S30: When the preset security detection conditions are met, the extracted first video frames and preset prompt words are input into the trained scene analysis model to obtain the scene analysis results.

[0039] Among them, the preset security detection conditions refer to the pre-set trigger conditions used to initiate the scene analysis model's inference. These conditions control the timing of the scene analysis model's invocation, avoiding invalid or frequent model calls, thereby reducing computational power consumption and API costs, while ensuring timely detection. Pre-set prompts refer to pre-designed text instructions used to guide the scene analysis model to perform inference according to specified logic and format. The prompts clearly define the content the model needs to analyze, the inference steps, and the output format, ensuring that the scene analysis results output by the model meet the business requirements of this method. The trained scene analysis model refers to an artificial intelligence model that has undergone large-scale multimodal data pre-training and business scenario fine-tuning, possessing image semantic understanding and contextual reasoning capabilities. This model can receive multiple frames of images and text prompts as input.

[0040] Specifically, the system monitors in real time whether preset security detection conditions are met. When the conditions are met, it retrieves the latest multiple consecutive first video frames from the first video frame buffer, packages them with preset prompts, and submits them asynchronously to the trained scene analysis model for inference. After the model completes inference, it returns structured scene analysis results. During the model inference process, the main thread continues to perform focus state detection on the second video frame without interrupting the normal detection process.

[0041] In some embodiments, such as Figure 5 As shown, in step S30, the extracted first video frames and preset prompts are input into the trained scene analysis model to obtain the scene analysis results, including the following steps S31 to S32: S31: Input the extracted first video frames and preset prompts into the trained scene analysis model, and analyze the first video frames using the preset prompts to determine the user's camera behavior state, scene, specific location within the scene, and movement state of the scene.

[0042] The process involves retrieving multiple first video frames from the cache and arranging them in chronological order. The images are then uniformly resized, compressed, and converted to conform to the input requirements of the scene analysis model. The preprocessed image data is then packaged with preset prompts into a request format that the model can accept.

[0043] The trained scene analysis model first extracts visual features independently for each frame of the image to obtain the visual semantic representation of each frame. Then, guided by preset prompts, it performs temporal comparison and contextual reasoning on the features of multiple frames of the image, and sequentially analyzes and outputs results in four dimensions: camera behavior state, scene, specific location within the scene, and movement state.

[0044] The original inference results and their confidence scores of the four dimensions output by the model are temporarily stored for subsequent standardization processing.

[0045] Camera behavior status refers to the current purpose of use and type of content captured by the user's terminal camera, reflecting whether the camera is being used for remote video interviews. This status distinguishes between normal interview scenarios and scenarios where the user temporarily switches the camera to capture other content (such as documents or the surrounding environment). The current scenario refers to the macroscopic type of the user's current physical environment, a semantic classification of the overall environment around the user. This status is used to determine whether the user is in a specific environment where there may be security risks. The specific location within the scenario refers to the user's specific spatial location within the physical scenario, used to further refine the user's role and potential behavior within the scenario. This status can distinguish the risk level differences between users in different locations within the same macroscopic scenario. The movement status refers to whether the entire physical scenario in which the user is located is in a state of overall displacement, reflecting the dynamic attributes of the scenario. This status is inferred by comparing the positional changes of background elements in the first frame of multiple video frames, and is a key dynamic indicator for determining whether high-risk behavior exists.

[0046] S32: The scene analysis results are determined by the camera's behavior state, the scene it is in, its specific location within the scene, and its movement state.

[0047] The process involves standardizing the format and values ​​of the original inference results from the four dimensions of the model output, converting them into standard data types (such as Boolean values ​​and enumeration values) that the system can directly use; checking whether the values ​​of each dimension are within the preset legal range, and verifying whether the confidence level of each dimension's results reaches the preset threshold; and combining the standardized and legally validated state information of the four dimensions into a complete scene analysis result object, ready to be updated to the system's global state.

[0048] Taking the scenario of detecting whether a user is driving during a face-to-face interview as an example, the preset prompts contain clear logical reasoning requirements, such as: "Analyze this continuous front-facing camera footage: 1. Is the footage from a face-to-face interview? 2. Is the person in the car? 3. Is the person in the driver's seat or a passenger? 4. Determine whether the vehicle is in motion by observing changes in the background outside the car window? Please output JSON."

[0049] In this way, by combining the first video frames arranged in chronological order with structured step-by-step prompts into the multimodal scene analysis model, it can accurately infer the movement state of the scene. At the same time, by guiding the model to output results according to specified dimensions through a clear logical reasoning chain, the uncertainty of the model's free output is effectively reduced, and the accuracy of semantic recognition in the four dimensions of camera behavior, scene, specific location, and movement state is improved. After standardized format conversion and confidence verification, a unified structured scene analysis result is generated, which not only provides a comprehensive and reliable global context for subsequent security status fusion and determination, but also significantly improves the system's processing efficiency and the interpretability of the results.

[0050] S40: Obtain the video focus state in each second video frame detected within the historical time period.

[0051] Specifically, the video focus state detection results for all second video frames prior to the current time point and within a preset historical time period are extracted from the historical state cache. The historical state cache uses a sliding window mechanism to automatically discard expired data that exceeds the historical time period, always retaining the latest focus state records.

[0052] Specifically, the length of the historical period can be configured according to business needs to balance the sensitivity and anti-interference capability of detection; the cache area adopts a thread-safe data structure to ensure data consistency during multi-threaded parallel read and write.

[0053] S50: Based on the scene analysis results and the focus state of each video, determine the user's safety status.

[0054] The safety status refers to the risk level of a user's current behavior, determined by a combination of scenario analysis results and video attention data from historical time periods. Safety status is typically categorized into two types: safe (no high-risk behavior) and unsafe (high-risk behavior exists), used to guide subsequent business processing (such as normal access, warnings, or service interruption).

[0055] Specifically, the obtained global scene analysis results are fused with the video focus status data of all second video frames within the historical time period. Through a preset fusion decision logic, the user's current behavioral risk level is comprehensively determined, and the user's safety status is finally obtained.

[0056] In some embodiments, such as Figure 6 As shown, step S50, based on the scene analysis results and the focus state of each video, determines the user's safety status, including the following steps S51 to S53: S51: When the camera behavior state is an interview behavior, and the scene is a preset scene and the specific location within the scene is a preset location, count the number of target states where the video's focus state is a non-focus state.

[0057] The process involves checking whether the three core dimensions in the scene analysis results simultaneously meet the preset conditions: the camera behavior state is a face-to-face interview behavior, the scene is a preset scene, and the specific location within the scene is a preset location. The precondition verification uses an AND logic, and subsequent calculations are only triggered when all three conditions are met simultaneously. This can quickly filter out the vast majority of risk-free scenes and significantly reduce unnecessary computation.

[0058] Specifically, if all three conditions are met, the video focus state records of all second video frames in the historical time period are traversed, and it is determined one by one whether it is a non-focused state. The total number of frames in the non-focused state is accumulated to obtain the number of target states.

[0059] S52: Calculate the ratio of the number of target states to the total number of focused states in each video to obtain the proportion of non-focused states.

[0060] Specifically, the total number of all second video frames contained within the current sliding time window is determined, i.e., the total number of states; the number of target states obtained in step S51 is divided by the total number of states to obtain the proportion of non-focused states.

[0061] S53: When the proportion of non-focused state is greater than or equal to the preset proportion of non-focused state, determine the user's safety status based on the movement status.

[0062] If the proportion of non-focused states is greater than or equal to a preset threshold, the final safety status determination logic is triggered: based on the movement status in the scenario analysis results, the user's final safety status is determined through preset decision rules.

[0063] In this way, the vast majority of risk-free scenarios are quickly filtered out through AND logic verification, significantly reducing the amount of unnecessary computation in the system. At the same time, based on the sliding time window, the proportion of non-focused states is statistically analyzed, and the discrete single-frame detection results are transformed into a quantitative indicator of the degree of continuous distraction, effectively avoiding misjudgments caused by accidental factors in a single frame. Finally, the security judgment is made by combining the scene movement state, realizing the deep integration of global scene semantic information and local user behavior features. While significantly improving the accuracy of high-risk behavior detection, it reduces the interference with normal business processes, balancing system operating efficiency and risk control accuracy.

[0064] In some embodiments, such as Figure 7 As shown, step S53 determines the user's security status based on the movement status, including the following steps S531-S532: S531: When the mobile status is mobile, determine that the user's security status is not secure.

[0065] The movement state refers to the overall displacement state of the user's physical scene, as output by the scene analysis model, and is divided into two categories: "moving" and "non-moving". "Moving" indicates that the scene is undergoing continuous spatial displacement, while "non-moving" indicates that the scene is stationary.

[0066] Specifically, the system reads the movement status field in the current global status and confirms that its value is "moving". Combined with preconditions (interview behavior + preset scenario + preset location + non-focused state percentage meets the standard), the system comprehensively determines that the user's current security status is unsafe. This triggers the highest-level preset risk intervention measures, including but not limited to: a strong full-screen pop-up reminder on the client, a voice warning, automatically suspending the interview process, saving the current violation video clip as evidence, and reporting the violation record to the backend risk control system.

[0067] S532: When the mobile status is non-mobile, determine that the user's security status is secure and provide a prompt message to indicate that face-to-face verification is prohibited.

[0068] Specifically, the system reads the movement status field from the current global system status and confirms its value is "non-moving." Combined with pre-defined conditions, the system comprehensively determines the user's current safety status as safe. A pre-set risk warning message is sent to the user's terminal, reminding them of distracted behavior and suggesting they focus on the interview process and avoid continuing the interview while the vehicle is in motion. The system continues to perform high-frequency focus state detection and low-frequency scenario analysis at the original frequency, monitoring changes in user behavior and scenario status in real time.

[0069] Therefore, based on the confirmed high-risk scenario and persistent distracting behavior of users, the system performs refined risk level classification based on the mobile state of the scenario. Fatal high-risk behaviors in mobile scenarios are judged as unsafe and trigger strong intervention measures such as full-screen pop-ups, voice warnings, and business suspension. This promptly stops dangerous behaviors and preserves evidence of violations, effectively avoiding risks to user personal safety and reputation and compliance issues for financial institutions. General distracting behaviors in stationary scenarios are judged as safe and only mild prompts are sent without interrupting normal business processes. At the same time, continuous monitoring is maintained to respond to changes in the state. This system strictly adheres to the bottom line of security risk control while avoiding user experience degradation and business loss caused by excessive intervention, achieving a triple balance between strict risk control, smooth business operations, and user experience.

[0070] In some embodiments, such as Figure 8 As shown, determining the user's safety status based on the scene analysis results and the focus state of each video also includes the following steps: When the percentage of non-focused states is less than the preset percentage of non-focused states, the user's safe state is determined to be safe. When the camera behavior is not a face-to-face interview, or the scene is not a preset scene, or the specific location within the scene is not a preset location, the user's security status is determined to be safe.

[0071] If the percentage of non-focused states is less than the preset percentage of non-focused states, the user's current security status is directly determined to be unsafe. Then, the system checks whether any of the following conditions are met in sequence: the camera behavior is not a face-to-face interview; the scene is not a preset scene; or the specific location within the scene is not a preset location. If any of these conditions are met, the user's current security status is directly determined to be safe.

[0072] In this way, by using global scenario information for rapid pre-filtering, most risk-free scenarios can be directly excluded, significantly reducing unnecessary subsequent calculations and avoiding unnecessary interference with normal business processes, thereby improving user experience and business conversion rates.

[0073] In summary, this application acquires the user's face-to-face video stream in real time; extracts first video frames from the face-to-face video stream at preset time intervals, and detects the user's video focus state in each second video frame acquired from the face-to-face video stream; when preset security detection conditions are met, the extracted first video frames and preset prompts are input into a trained scene analysis model to obtain scene analysis results; the video focus state in each second video frame detected within a historical time period is obtained; based on the scene analysis results and each video focus state, the user's security status is determined. In this application, the user scenario is analyzed by combining first video frames extracted at different time periods, thereby obtaining scene analysis results used as contextual information for subsequent security status detection. Therefore, by combining the scene analysis results and the video focus state of each second video frame, the user's security status is comprehensively determined. This avoids misjudgment during user security detection, improving the accuracy of security detection during face-to-face interviews, especially ensuring the security of users during face-to-face interviews in the financial sector.

[0074] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0075] In one embodiment, a user security detection device is provided, which corresponds one-to-one with the user security detection methods described in the above embodiments. For example... Figure 8 As shown, the user security detection device includes a first acquisition module 101, a processing module 102, an analysis module 103, a second acquisition module 104, and a determination module 105. Detailed descriptions of each functional module are as follows: The first acquisition module 101 is used to acquire the user's face-to-face video stream in real time; Processing module 102 is used to extract a first video frame from the face-to-face video stream at a preset time interval, and to detect the user's video focus state in each second video frame obtained from the face-to-face video stream. The analysis module 103 is used to input multiple extracted first video frames and preset prompt words into the trained scene analysis model when the preset security detection conditions are met, so as to obtain the scene analysis results. The second acquisition module 104 is used to acquire the video focus state in each second video frame detected within the historical time period; The determination module 105 is used to determine the user's safety status based on the scene analysis results and the focus state of each video.

[0076] In one embodiment, the analysis module 103 includes: The analysis submodule is used to input multiple extracted first video frames and preset prompt words into the trained scene analysis model, and analyze the multiple first video frames through the preset prompt words to determine the user's camera behavior state, scene, specific location within the scene, and movement state of the scene. The first determining submodule is used to determine the camera behavior state, the scene it is in, its specific location within the scene, and its movement state as the scene analysis result.

[0077] In one embodiment, the determining module 105 includes: The statistics submodule is used to count the number of target states where the video's focus state is a non-focus state when the camera behavior state is a face-to-face interview behavior, the scene is a preset scene, and the specific location within the scene is a preset location. The calculation submodule is used to calculate the ratio of the number of target states to the total number of states for each video's focused state, and to obtain the proportion of non-focused states; The second determining submodule is used to determine the user's safety status based on the movement status when the proportion of the non-focused state is greater than or equal to the preset proportion of the non-focused state.

[0078] In one embodiment, the second determining submodule is configured to: When the movement status is "moving", the user's security status is determined to be "unsecured". When the mobile status is non-mobile, the user's security status is determined to be secure, and a prompt message is sent to indicate that face-to-face verification is prohibited.

[0079] In one embodiment, the determining module 105 further includes: The third determining submodule is used to determine the user's safety status as safe when the proportion of the non-focused state is less than the preset proportion of the non-focused state. The fourth determination submodule is used to determine the user's security status as safe when the camera behavior state is non-face-to-face review behavior, or the scene is a non-preset scene and the specific location within the scene is a non-preset location.

[0080] In one embodiment, the processing module 102 includes: The fifth determination submodule is used to determine the user's head yaw angle in each second video frame obtained from the face-to-face video stream; The sixth determining submodule is used to determine the video focus state as a non-focus state when the head yaw angle is greater than or equal to a preset yaw angle; The seventh determination submodule is used to determine the video focus state as a focus state when the head yaw angle is less than a preset yaw angle.

[0081] In one embodiment, the processing module 102 includes: The eighth determination submodule is used to determine the user's head pitch angle in each second video frame obtained in the face-to-face video stream; The ninth determining submodule is used to determine the video focus state as a non-focus state when the head pitch angle is greater than or equal to a preset pitch angle; The tenth determination submodule is used to determine the video focus state as a focus state when the head yaw angle is less than the preset pitch angle.

[0082] This application provides a user security detection device that analyzes user scenarios by combining first video frames extracted from different time periods. This yields scenario analysis results used as contextual information for subsequent security status detection. By combining the scenario analysis results with the focus status of each second video frame, the user's security status is comprehensively determined. This avoids misjudgments during user security detection, improving the accuracy of security detection during face-to-face interviews, and particularly ensuring user security during face-to-face interviews in the financial sector.

[0083] Specific limitations regarding the user security detection device can be found in the limitations of the user security detection method described above, and will not be repeated here. Each module in the aforementioned user security detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0084] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a user security detection method on the server side.

[0085] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a user security detection method on the client side. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Real-time acquisition of user interview video streams; The first video frame is extracted from the face-to-face video stream at a preset time interval, and the user's video focus state in each second video frame obtained from the face-to-face video stream is detected. When the preset security detection conditions are met, the extracted first video frames and preset prompt words are input into the trained scene analysis model to obtain the scene analysis results. Obtain the video focus state in each second video frame detected within a historical time period; Based on the scene analysis results and the focus state of each video, the user's safety status is determined.

[0086] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Real-time acquisition of user interview video streams; The first video frame is extracted from the face-to-face video stream at a preset time interval, and the user's video focus state in each second video frame obtained from the face-to-face video stream is detected. When the preset security detection conditions are met, the extracted first video frames and preset prompt words are input into the trained scene analysis model to obtain the scene analysis results. Obtain the video focus state in each second video frame detected within a historical time period; Based on the scene analysis results and the focus state of each video, the user's safety status is determined.

[0087] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0088] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0089] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0090] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0091] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A user security detection method, characterized in that, include: Real-time acquisition of user interview video streams; The first video frame is extracted from the face-to-face video stream at a preset time interval, and the user's video focus state in each second video frame obtained from the face-to-face video stream is detected. When the preset security detection conditions are met, the extracted first video frames and preset prompt words are input into the trained scene analysis model to obtain the scene analysis results. Obtain the video focus state in each second video frame detected within a historical time period; Based on the scene analysis results and the focus state of each video, the user's safety status is determined.

2. The user security detection method according to claim 1, characterized in that, The process involves inputting multiple extracted first video frames and preset prompts into the trained scene analysis model to obtain scene analysis results, including: Multiple extracted first video frames and preset prompts are input into the trained scene analysis model. The multiple first video frames are analyzed using the preset prompts to determine the user's camera behavior state, scene, specific location within the scene, and movement state within the scene. The camera behavior state, the scene it is in, its specific location within the scene, and its movement state are determined as the scene analysis results.

3. The user security detection method according to claim 2, characterized in that, The process of determining the user's safety status based on the scene analysis results and the focus state of each video includes: When the camera behavior state is a face-to-face interview, and the scene is a preset scene and the specific location within the scene is a preset location, count the number of target states where the video focus state is a non-focus state. Calculate the ratio of the number of target states to the total number of focused states for each video to obtain the proportion of non-focused states; When the proportion of the non-focused state is greater than or equal to the preset proportion of the non-focused state, the user's safety status is determined based on the movement status.

4. The user security detection method according to claim 3, characterized in that, Determining the user's security status based on the movement status includes: When the movement status is "moving", the user's security status is determined to be "unsecured". When the mobile status is non-mobile, the user's security status is determined to be secure, and a prompt message is sent to indicate that face-to-face verification is prohibited.

5. The user security detection method according to claim 3, characterized in that, The method further includes: When the proportion of the non-focused state is less than the preset proportion of the non-focused state, the user's safety status is determined to be safe; When the camera behavior is a non-face-to-face interview behavior, or the scene is a non-preset scene and the specific location within the scene is a non-preset location, the user's security status is determined to be safe.

6. The user security detection method according to claim 1, characterized in that, The detection of the user's video focus state in each second video frame obtained from the interview video stream includes: Determine the user's head yaw angle in each second video frame obtained from the face-to-face interview video stream; When the head yaw angle is greater than or equal to the preset yaw angle, the video focus state is determined to be a non-focus state. When the head yaw angle is less than the preset yaw angle, the video focus state is determined to be a focus state.

7. The user security detection method according to claim 1, characterized in that, The detection of the user's video focus state in each second video frame obtained from the interview video stream includes: Determine the user's head pitch angle in each second video frame obtained from the face-to-face interview video stream; When the head pitch angle is greater than or equal to the preset pitch angle, the video focus state is determined to be a non-focus state; When the head yaw angle is less than the preset pitch angle, the video focus state is determined to be a focus state.

8. A user safety detection device, characterized in that, include: The first acquisition module is used to acquire the user's face-to-face video stream in real time; The processing module is used to extract a first video frame from the face-to-face video stream at a preset time interval, and to detect the user's video focus state in each second video frame obtained from the face-to-face video stream. The analysis module is used to input multiple extracted first video frames and preset prompt words into the trained scene analysis model when the preset security detection conditions are met, and obtain the scene analysis results. The second acquisition module is used to acquire the video focus state in each second video frame detected within the historical time period; The determination module is used to determine the user's safety status based on the scene analysis results and the focus state of each video.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the user security detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the user security detection method as described in any one of claims 1 to 7.

Images