A video detection method and apparatus

Abstract

The embodiment of the invention discloses a video detection method and device. According to the method, firstly, a to-be-detected video is obtained, an optical flow field between adjacent frames in the video is calculated, then motion components (such as vertical or horizontal displacement) related to a preset risk category are extracted, an initial risk detection result of each frame is generated, then statistics is carried out on all the initial detection results to form structured risk statistical information, and the risk statistical information is obtained. And filling the information into a prompt template corresponding to a preset risk category, automatically generating a risk detection prompt statement, finally inputting the statement into a visual large model, and outputting a final risk detection result. The method does not need to depend on complex modules such as target detection, key dynamic features can be captured and initial detection can be completed only through optical flow, then risk detection is achieved through combination of the prompt template and the visual large model, the calculation overhead can be reduced, the risk identification efficiency and semantic accuracy can be improved, meanwhile, the good generalization ability is achieved, and the method is suitable for popularization and application. The method can be adapted to various risk scenes and has expansibility.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511923683.2 (22) Application Date 2025.12.18 (71) Applicant Alibaba (China) Co., Ltd. Address 310052, Room 508, 5th Floor, Building 4, No. 699, Wangshang Road, Changhe Street, Binjiang District, Hangzhou, Zhejiang Province (72) Inventors Zhang Minghao and Chen Dewang (74) Patent Agency Beijing Ruipai Intellectual Property Agency Co., Ltd. 11597 Patent Attorney Liu Feng (51) Int.Cl. G06V 20 / 40 (2022.01) G06V 10 / 25 (2022.01) G06V 20 / 54 (2022.01) G06V 20 / 59 (2022.01) G06V 10 / 62 (2022.01) (54) Invention Title: A Video Detection Method and Apparatus (57) Abstract: This invention discloses a video detection method and apparatus. The method of this invention first acquires the video to be detected, calculates the optical flow field between adjacent frames in the video, then extracts motion components (such as vertical or horizontal displacement) related to a preset risk category, generates the initial risk detection result for each frame, then statistically analyzes all the initial detection results to form structured risk statistics, fills the information into the prompt template corresponding to the preset risk category, automatically generates a risk inspection prompt statement, and finally inputs the statement into a large visual model to output the final risk detection result. The above method does not rely on complex modules such as target detection, and can capture key dynamic features and complete the initial detection by optical flow alone. Then, it combines the prompt template with the large visual model to realize risk inspection. It can improve the efficiency and semantic accuracy of risk identification while reducing computational overhead. It also has good generalization ability, can be adapted to various risk scenarios, and has scalability. Claims 2 pages, Description 15 pages, Drawings 7 pages, CN 121725397 A 2026.03.24 CN 1 21 72 53 97 A 1. A video detection method, characterized in that the method includes: acquiring a video to be detected; determining the optical flow field corresponding to each adjacent video frame in the video to be detected; determining the initial risk detection result corresponding to each video frame based on the motion components associated with the risk category to be detected in each optical flow field; determining the risk statistics information corresponding to the video to be detected based on the initial risk detection results; filling the risk statistics information into the prompt template corresponding to the risk category to be detected to generate a corresponding risk inspection prompt statement; inputting the risk inspection prompt statement into a large visual model to determine the corresponding risk detection result. 2. The method according to claim 1, characterized in that the step of determining the initial risk detection result based on the motion components associated with the risk category to be detected in each optical flow field...The method according to claim 2, wherein determining the target region from each optical flow field comprises: determining the corresponding detection direction based on the risk category to be detected; extracting the motion component in the detection direction from the target region; and determining the initial risk detection result of the corresponding video frame based on the motion component. 3. The method according to claim 2, wherein determining the target region from each optical flow field comprises: performing image recognition on the video frame to determine the corresponding detection target; and determining the target region based on the detection target. 4. The method according to claim 2, wherein determining the target region from each optical flow field comprises: determining the corresponding preset position based on the risk category to be detected; and determining the region in the video frame located at the preset position as the target region. 5. The method according to claim 2, wherein determining the initial risk detection result of the corresponding video frame based on the motion component comprises: dividing the target region into multiple sub-regions in the detection direction; determining the corresponding motion speed based on the motion component in each sub-region; and determining the initial risk detection result of the corresponding video frame based on each motion speed. 6. The method according to claim 5, wherein the risk category to be detected is vehicle collision risk, and the detection direction is vertical; the step of determining the initial risk detection result of the corresponding video frame based on each of the motion speeds includes: determining the corresponding time to collision based on the motion speed corresponding to the sub-region and the relative position of the sub-region within the target area; determining the risk level of the corresponding video frame based on the motion speed and the time to collision; and adding the video frame to the risk video frame sub-sequence corresponding to the risk level. 7. The method according to claim 5, wherein the risk category to be detected is alcohol consumption risk, and the detection direction is vertical; the step of determining the initial risk detection result of the corresponding video frame based on each of the motion speeds includes: determining the risk level of the corresponding video frame based on the motion speed corresponding to the sub-region; and adding the video frame to the risk video frame sub-sequence corresponding to the risk level. 8. The method according to claim 6 or 7, wherein adding the video frame to the risk video frame subsequence corresponding to the risk level comprises: in response to the existence of a risk level higher than a preset risk level for at least a predetermined number of consecutive data frames, adding each video frame to the risk video frame subsequence corresponding to the risk level. 9. The method according to claim 1, wherein the risk statistics information comprises at least one risk video frame subsequence corresponding to a risk level, and / or, the time point corresponding to each video frame in the risk video frame subsequence.10. The method according to claim 1, wherein after acquiring the video to be detected, the method further comprises: determining the type of risk to be detected corresponding to the video to be detected. 11. A video detection device, comprising: an acquisition module for acquiring the video to be detected; a first determination module for determining the optical flow field corresponding to each adjacent video frame in the video to be detected; a second determination module for determining the initial risk detection result corresponding to each video frame based on the motion component in the optical flow field associated with the risk category to be detected; a third determination module for determining the risk statistics information corresponding to the video to be detected based on the initial risk detection results; a generation module for filling the risk statistics information into a prompt template corresponding to the risk category to be detected, generating a corresponding risk inspection prompt statement; and a detection module for inputting the risk inspection prompt statement into a large visual model, determining the corresponding risk detection result. 12. An electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1-10. 13. A computer-readable storage medium, characterized in that a computer program is stored within the computer-readable storage medium, and the computer program, when executed by a processor, implements the method as described in any one of claims 1-10. 14. A computer program product comprising a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implements the method as described in any one of claims 1-10. Claims 2 / 2 Page 3 CN 121725397 A Video Detection Method and Apparatus Technical Field

[0001] This invention relates to the field of video recognition technology, specifically to a video detection method and apparatus. Background Art

[0002] With the development of artificial intelligence and computer vision technology, automatic risk identification and behavior analysis based on video have been widely applied in fields such as traffic monitoring, public safety, and content moderation. However, existing video detection methods still face many challenges in practical applications. Traditional methods typically rely on deep learning models for object detection, keypoint estimation, or behavior recognition to assess risk by identifying specific objects (such as wine bottles, mobile phones, and vehicles) or human poses. However, these methods have significant limitations: on the one hand, object detection is prone to failure in low-light, occluded, or small-object scenarios, and its generalization ability to unseen object categories is weak; on the other hand, end-to-end behavior recognition models often lack interpretability, and in complex scenarios (such as changes in lighting or occlusion), there are deviations in judging the start and end times of events, resulting in poor detection accuracy.

[0003] In view of this, embodiments of the present invention provide a video detection method and apparatus to improve the efficiency and effectiveness of risk identification.Semantic accuracy.

[0004] In a first aspect, a video detection method is provided, the method comprising: acquiring a video to be detected; determining the optical flow field corresponding to each adjacent video frame in the video to be detected; determining the initial risk detection result corresponding to each video frame based on the motion components associated with the risk category to be detected in each optical flow field; determining the risk statistics information corresponding to the video to be detected based on the initial risk detection results; filling the risk statistics information into the prompt template corresponding to the risk category to be detected to generate a corresponding risk inspection prompt statement; inputting the risk inspection prompt statement into a large visual model to determine the corresponding risk detection result.

[0005] Further, determining the initial risk detection result corresponding to each video frame based on the motion components associated with the risk category to be detected in each optical flow field comprises: determining a target region from each optical flow field; determining the corresponding detection direction based on the risk category to be detected; extracting the motion components in the detection direction from the target region; and determining the initial risk detection result of the corresponding video frame based on the motion components.

[0006] Further, determining the target region from each of the optical flow fields includes: performing image recognition on the video frame to determine the corresponding detection target; and determining the target region based on the detection target.

[0007] Further, determining the target region from each of the optical flow fields includes: determining the corresponding preset position based on the risk category to be detected; and determining the region in the video frame located at the preset position as the target region. Specification 1 / 15 page 4 CN 121725397 A

[0008] Further, determining the initial risk detection result of the corresponding video frame based on the motion components includes: dividing the target region into multiple sub-regions in the detection direction; determining the corresponding motion speed based on the motion components in each sub-region; and determining the initial risk detection result of the corresponding video frame based on each motion speed.

[0009] Further, the risk category to be detected is vehicle collision risk, and the detection direction is vertical; determining the initial risk detection result of the corresponding video frame based on each of the motion speeds includes: determining the corresponding time to collision based on the motion speed corresponding to the sub-region and the relative position of the sub-region within the target area; determining the risk level of the corresponding video frame based on the motion speed and the time to collision; adding the video frame to the risk video frame sub-sequence corresponding to the risk level.

[0010] Further, the risk category to be detected is drinking behavior risk, and the detection direction is vertical; determining the initial risk detection result of the corresponding video frame based on each of the motion speeds includes: determining the risk level of the corresponding video frame based on the motion speed corresponding to the sub-region; adding the video frame to the risk video frame sub-sequence corresponding to the risk level.

[0011] Further, adding the video frames to the risk video frame subsequence corresponding to the risk level includes: in response to the existence of at least a predetermined number of consecutive data frames with risk levels higher than a preset risk level, adding each video frame to the risk video frame subsequence corresponding to the risk level.

[0012] Further, the risk statistics information includes at least one risk video frame subsequence corresponding to a risk level, and / or, the time point corresponding to each video frame in the risk video frame subsequence.

[0013] Further, after acquiring the video to be detected, the method further includes: determining the risk type to be detected corresponding to the video to be detected.

[0014] In a second aspect, a video detection apparatus is provided, the apparatus comprising: an acquisition module for acquiring a video to be detected; a first determination module for determining the optical flow field corresponding to each adjacent video frame in the video to be detected; a second determination module for determining the initial risk detection result corresponding to each video frame based on the motion component in the optical flow field associated with the risk category to be detected; a third determination module for determining the risk statistics information corresponding to the video to be detected based on the initial risk detection results; a generation module for filling the risk statistics information into a prompt template corresponding to the risk category to be detected, and generating a corresponding risk inspection prompt statement; and a detection module for inputting the risk inspection prompt statement into a large visual model, and determining the corresponding risk detection result.

[0015] In a third aspect, an electronic device is provided, comprising a memory and a processor, the memory for storing one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect above.

[0016] In a fourth aspect, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the method described in the first aspect above.

[0017] In a fifth aspect, a computer program product is provided, including a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the method described in the first aspect above.

[0018] The technical solution of the present invention first acquires a video to be detected, calculates the optical flow field between adjacent frames in the video to be detected, then extracts motion components (such as vertical or horizontal displacement) related to a preset risk category, generates a preliminary risk detection result for each frame, then statistically analyzes all preliminary detection results to form structured risk statistics information, fills the risk statistics information into a prompt template corresponding to a preset risk category, automatically generates a risk inspection prompt statement, and finally inputs the statement into a large visual model to output the final risk detection result. The above technical solution does not rely on complex modules such as target detection, but only uses...Optical flow can capture key dynamic features and complete the initial risk detection. Then, by combining the prompt template with the visual large model, the risk verification is realized, achieving high-precision semantic understanding. This can improve the efficiency and semantic accuracy of risk identification while reducing computational overhead. It also has good generalization ability and can be adapted to various risk scenarios, combining practicality and scalability.

[0019] The above and other objects, features and advantages of the present invention will become clearer from the following description of the embodiments of the present invention with reference to the accompanying drawings, in which: Figure 1 is a flowchart of the video detection method of the present invention; Figure 2 is a flowchart of the method for determining the initial risk detection result of the present invention; Figure 3 is a flowchart of the method for determining the initial risk detection result corresponding to vehicle collision risk of the present invention; Figure 4 is a schematic diagram of a video frame of the present invention; Figure 5 is a flowchart of the video detection method corresponding to vehicle collision risk; Figure 6 is a flowchart of the data processing method of the visual large model of the present invention; Figure 7 is a schematic diagram of a video detection device of the present invention; Figure 8 is a schematic diagram of an electronic device of the present invention.

[0020] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the following detailed description of the present application, some specific details are described in detail. This application can be fully understood by those skilled in the art without the description of these details. In order to avoid obscuring the substance of this application, well-known methods, processes, flows, components and circuits are not described in detail.

[0021] In addition, those skilled in the art should understand that the drawings provided herein are for illustrative purposes and are not necessarily drawn to scale.

[0022] Unless the context explicitly requires it, the words "including," "comprising," and similar terms throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to."

[0023] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In addition, in the description of this application, unless otherwise stated, "multiple" means two or more.

[0024] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed under the premise of having a legal basis (e.g., obtaining the consent of the personal information subject, or being necessary for the performance of a contract, etc.), and will only be processed within the prescribed or agreed scope. The user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0025] With the explosive growth of video content on Internet platforms, the demand for video-based intelligent analysis is increasing. (Specification 3 / 15 pages 6 CN 121725397 A)The scope is broad, covering multiple fields such as advertising review, content security, and behavior recognition. Traditional video detection methods mostly adopt a fixed frame rate sampling strategy, that is, extracting video frames at constant time intervals for subsequent analysis. However, this type of time-series data processing method based on fixed frame rate has gradually revealed significant limitations when dealing with dynamic events.

[0026] First, the problem of missing information is prominent. Since fixed frame extraction cannot guarantee that the time of occurrence of key actions falls exactly at the sampling point, short-term dynamic behaviors (such as dangerous driving actions and sudden abnormal behaviors) are easily missed. At the same time, long-term continuous events may be incorrectly segmented into multiple isolated segments due to excessively large inter-frame intervals, resulting in contextual breaks and affecting overall understanding.

[0027] Second, the boundaries of dynamic events are blurred. Current mainstream Vision-Language Models (VLMs) usually rely on single-frame or short-segment inputs when processing video sequences, and there are deviations in the judgment of the start and end times of events. Especially in complex scenarios such as changes in lighting, occlusion, and motion blur, the model has difficulty accurately identifying the true duration of the event, which easily leads to misjudgment or missed detection.

[0028] Furthermore, the scene adaptability is insufficient, and the accuracy and efficiency requirements for dynamic information extraction vary significantly across different business scenarios. For example, advertising review requires high-precision capture of instantaneous violations, while content summarization focuses more on long-term trends. Traditional rule-based or fixed-window processing methods lack flexibility and are difficult to adapt to diverse task requirements, limiting the system's versatility and scalability.

[0029] Therefore, there is an urgent need for a video analysis method that can break through the fixed frame rate limitation, accurately capture the entire process of dynamic events, and has good scene generalization ability, so as to achieve efficient and reliable identification and understanding of complex dynamic behaviors. In this regard, this application provides a video detection method and apparatus to achieve the aforementioned functions.

[0030] Figure 1 is a flowchart of the video detection method of this embodiment of the invention. As shown in Figure 1, the video detection method includes the following steps: Step S100, acquiring the video to be detected.

[0031] Wherein, the video to be detected refers to digital video data containing dynamic visual content and requiring determination of whether there are specific risky behaviors or events in it, the video being composed of a continuous sequence of image frames. Specific risks refer to various pre-defined risk categories, including but not limited to collision risks when vehicles are traveling straight on the road, collision risks when vehicles change lanes too late, and risks related to drinking and other risky behaviors in short video review. Based on different risk categories, the video content and source of the video to be detected also differ. For example, if the risk category is collision risk when vehicles are traveling straight, then the video source is an in-vehicle camera, and the video content is road traffic conditions. The video to be detected can be obtained through real-time acquisition, current storage and retrieval, network transmission and reception, or access from a third-party system.

[0032] In one possible implementation, after obtaining the video to be detected, the corresponding target video can also be determined.Detect risk type.

[0033] Optionally, the video to be detected carries a corresponding risk category label, and the corresponding risk category to be detected can be determined directly based on the label. The risk category label can be obtained based on human interaction.

[0034] Optionally, the corresponding risk category to be detected is determined based on the video source or metadata of the video to be detected. For example, the default risk category to be detected for videos from vehicle-mounted forward-looking cameras is vehicle collision risk, and the default risk category to be detected for videos from driver monitoring system (DMS) cameras is abnormal behavior detection of occupants in the vehicle.

[0035] Optionally, the risk category to be detected can include multiple risk categories, and multiple risk category motion component extraction rules and prompt templates are loaded simultaneously. The corresponding initial risk detection results are generated for each risk category, and finally, the visual big model comprehensively judges the most likely risk type. Specification 4 / 15 pages 7 CN 121725397 A

[0036] Step S200, determine the optical flow field corresponding to each adjacent video frame in the video to be detected.

[0037] The optical flow field refers to the vector representation of the motion trend of each pixel between two adjacent video frames. It describes the instantaneous motion direction and velocity of each visible point in the image in the time dimension, and is usually presented in the form of a two-dimensional vector.

[0038] Specifically, for the t-th frame and the (t+1)-th frame in the video, the optical flow field is a two-dimensional array (or vector field) with the same spatial size as the video frame, where each position (x, y) corresponds to a vector: where represents the displacement component of the pixel in the horizontal direction (i.e., the x-axis), and represents the displacement component of the pixel in the vertical direction (i.e., the y-axis). Both are in pixels / frame. >0 indicates movement to the right, =0 indicates no movement in the horizontal direction, <0 indicates movement to the left, >0 indicates movement downward, =0 indicates no movement in the vertical direction, and <0 indicates movement upward.

[0039] The optical flow field, as a low-level dynamic feature carrier, is used to extract motion cues related to the risk category to be detected, such as the vertical negative velocity representing the approach of an object. The optical flow field has pixel-level accuracy, with each pixel having a corresponding motion vector (dense optical flow), which can capture subtle dynamics. Motion is estimated based solely on brightness / texture changes, without relying on object detection or semantic segmentation. It represents the relative motion between the camera and the scene (such as vehicle movement, pedestrian crossing, hand movements, etc.). It can still be calculated under low illumination or no color information (grayscale image) conditions. Through statistical analysis of specific components in the optical flow field, unsupervised or weakly supervised initial screening of risky behaviors can be achieved, providing reliable input for subsequent semantic reasoning.

[0040] In one possible implementation, the determination of the optical flow field includes methods based on traditional computer vision algorithms and...Regarding deep learning methods: Traditional methods such as Farnebäck, TV-L1, and DIS (Dense Inverse Search) directly solve dense motion vectors from pixel intensity changes in adjacent frames by modeling the assumption of constant image brightness and spatial smoothness constraints; deep learning methods such as FlowNet, PWC-Net, and RAFT utilize convolutional neural networks to learn optical flow mapping relationships from a large amount of data, which usually has higher accuracy but higher computational cost.

[0041] Optionally, strategies such as downsampling, region cropping, or hardware acceleration can be combined to improve computational efficiency while ensuring motion information in key areas, which is suitable for scenarios requiring high-efficiency detection, such as vehicle-mounted or real-time video analysis.

[0042] Step S300: Determine the initial risk detection result corresponding to each video frame based on the motion components in each optical flow field associated with the risk category to be detected.

[0043] Wherein, the motion component refers to the projection value of the optical flow vector in a specific direction, usually u (horizontal) or v (vertical), and can also be extended to derivative features such as amplitude ( ) or direction angle (arctan(v / u)). The motion component associated with the risk category to be detected refers to the motion data in the direction that has the most discriminative power in judging the risk category to be detected. Specifically, the motion component associated with the risk category to be detected refers to the optical flow direction or combination features that have a significant indicative role in risk judgment, which are predetermined according to the type of risk to be detected. For example, when the risk category to be detected is the risk of a straight-through collision, the associated motion component is the vertical component in the optical flow field. If the risk detected is a forward collision, the focus is on its negative value region; if the risk detected is a rear-end collision, the focus is on its positive value region. By extracting and analyzing this specific motion component, the initial risk detection result corresponding to each video frame can be generated.

[0044] For example, the motion component includes the motion component in the vertical direction (hereinafter referred to as the vertical motion component) and the motion component in the horizontal direction (hereinafter referred to as the horizontal motion component). The vertical motion component is as described above, and the horizontal motion component is as described above.

[0045] The association between the risk category to be detected and the motion component is a mapping relationship defined in advance based on physical common sense or behavioral patterns. Different risk categories focus on different motion features. For example, the motion component associated with the collision risk category when a vehicle is traveling straight is the vertical motion component. Its physical basis is that when a vehicle approaches from the front, it appears as upward motion (v<0) in the image, and the faster the speed, the more dangerous it is. The motion component associated with the risk category of a pedestrian crossing the road while a vehicle is moving is the horizontal motion component. Its physical basis is that when a pedestrian crosses the lane laterally, strong horizontal motion is generated; large positive or negative values ​​of the horizontal motion component can indicate the risk of a pedestrian crossing the road. The risk category associated with a driver looking down at their phone...The motion component associated with the risk category is the vertical component, and its corresponding physical basis is that the driver's head is lowered, causing the head image to shift downward in the image.

[0046] The above association transforms the abstract risk semantics (such as vehicle collision, pedestrian crossing the road, driver looking down, etc.) into quantifiable optical flow features, thereby realizing a bridge from the underlying visual signal to the upper-level risk judgment.

[0047] Figure 2 is a flowchart of the risk initial detection result determination method according to an embodiment of the present invention. As shown in Figure 2, the risk initial detection result determination method includes the following steps: Step S310, determine the target area from each of the optical flow fields.

[0048] Wherein, the target area refers to the local spatial area that is strongly correlated with the current risk category to be detected, which is extracted from the optical flow field corresponding to the video frame. This area is usually preset or dynamically detected according to the physical characteristics or occurrence location of the risk behavior, and is used to focus on analyzing key motion clues and eliminate irrelevant background interference. In vehicle collision risk detection, the target area can be the lower half of the image, such as 40% to 100% of the height, which corresponds to the area in front of the road and near the vehicle. In alcohol consumption behavior detection, the target region can be the mouth area of ​​a face, the area of ​​hand movement, or the area near the cup holder, which is obtained by cropping after being located by the face / hand detection model.

[0049] The introduction of the target region achieves the technical effects of focusing on key points, suppressing interference, and improving efficiency and accuracy. It is an important intermediate link connecting the underlying optical flow features and the high-level risk semantics. Taking vehicle collision risk detection as an example, extracting the target region from the entire optical flow field can eliminate motion noise from irrelevant or interfering areas such as the sky, distant background, and dashboard reflection, making the risk-related motion components more prominent and enhancing the feature signal-to-noise ratio. Focusing the analysis on high-risk areas such as the near lane can avoid signal dilution caused by global averaging during subsequent speed calculation, which helps to capture weak but key behavioral clues and improve detection accuracy. The position and range of the target region can also be flexibly configured according to the risk category to be detected, so that the same optical flow analysis framework can be quickly transferred to detection tasks of different risk categories.

[0050] In one possible implementation, the target area can be further divided into functional sub-regions, such as near / middle / far layers, or hand / mouth / cup holder areas, etc., to support more refined risk assessment logic and facilitate hierarchical or semantic modeling.

[0051] In the method corresponding to step S310 above, the complete dense optical flow field is first calculated for adjacent video frames, and then the target area is extracted from the optical flow field. This method ensures that the optical flow estimation fully utilizes the context information of the entire image and avoids boundary distortion caused by local cropping, thereby improving the accuracy of subsequent initial risk detection.

[0052] In one possible implementation, the extraction of the target area can also be placed in step S200 above, that is, the target area is first extracted from the video frame, and then the dense optical flow field corresponding to the target area is calculated. This method is suitable for scenarios where computing resources are limited and the target area is far from the image boundary.

[0053] In one possible implementation, the target area can be extracted from the optical flow field based on a preset spatial location or target detection result.

[0054] Optionally, a preset location is determined according to the risk category to be detected, and the area in the video frame at the preset location is determined as the target area.

[0055] Specifically, the position and size of the target area are fixed before system deployment, such as 40%~100% of the lower half of the screen or 200×200 pixels in the center. Its position does not change with the video content and is suitable for scenarios with stable viewing angles. There is a mapping relationship between the risk category to be detected and the corresponding preset location. For example, the preset location corresponding to vehicle collision risk detection is 40%~100% of the lower half of the screen because the lower half of the screen represents the near-vehicle part. The preset location corresponding to vehicle rear-end collision risk detection is 40%~100% of the upper half of the screen because for the rear of the vehicle, the lower half of the screen represents the near-vehicle part.

[0056] The method for determining the target area based on a preset spatial location does not rely on specific visual content, is sensitive to posture changes, requires no additional AI model, has extremely low computational overhead, predictable behavior, and is easy to debug and verify for compliance. It is suitable for fixed-view scenarios, such as vehicle collision detection. For details, please refer to the following embodiments, which will not be described further here.

[0057] Optionally, image recognition is first performed on video frames to determine the corresponding detection target, and then the target area is determined based on the detection target.

[0058] Specifically, key objects (i.e., detection targets) are located by calling face detection, hand detection, or object detection models, and then their bounding boxes or key point regions are used as the target area for dynamic cropping. This is suitable for scenarios requiring high-precision behavior recognition.

[0059] The method for determining the target area based on target detection relies on visual content and is highly adaptable, for example, adapting to different driver heights, sitting postures, and lighting conditions. It supports fine-grained behavior recognition, such as specifically distinguishing between drinking water and drinking alcohol. It can also reduce false alarms and avoid misjudging dashboard reflections and passenger actions as risky driver behaviors. This is applicable to scenarios requiring high-precision behavior recognition, such as risk behavior detection like drinking / smoking. Details can be found in the following embodiments, which will not be elaborated here.

[0060] Step S320: Determine the corresponding detection direction based on the risk category to be detected.

[0061] Specifically, there is a predetermined mapping relationship between the risk category to be detected and the detection direction. For example, the detection direction corresponding to vehicle collision risk detection is vertical, and the detection direction corresponding to pedestrian crossing risk detection is horizontal. The mapping relationship between the risk category to be detected and the detection direction is determined based on the behavioral logic of the behavior corresponding to the risk category.

[0062] Step S330: Extract the motion component in the detection direction from the target area.

[0063] For example, if the detection direction is vertical, the extracted motion component is the vertical motion component; if the detection direction is horizontal, the extracted motion component is the horizontal motion component.

[0064] Step S340: Determine the initial risk detection result of the corresponding video frame based on the motion component.

[0065] The initial risk detection result includes information such as risk level, time point of the corresponding video frame in the video to be detected, and confidence level.

[0066] For different risk categories, the method for determining the risk detection result based on the motion component is also different.

[0067] Figure 3 is a flowchart of the method for determining the initial risk detection result corresponding to vehicle collision risk according to an embodiment of the present invention. As shown in Figure 3, the method for determining the initial risk detection result corresponding to vehicle collision risk includes the following steps: Step S341: Divide the target area into layers in the detection direction to obtain multiple sub-regions.

[0068] Further layering of the target area is to simulate spatial depth. Layering in the vertical direction can simulate spatial depth, distinguish the motion characteristics of different distance areas, and thus more accurately assess the risk. Specifically, the target area is divided into multiple sub-regions such as near, middle, and far along the vertical direction. Each sub-region corresponds to a different distance range in the scene, such as a nearby lane or a distant road. The motion components and time to collision (TTC) of each layer are calculated separately to avoid mutual interference between motion signals of near and far objects. It can also improve the sensitivity and positioning ability of emergency approach events based on the principle of the most dangerous layer. For example, a rapid approach (high risk) in the near layer should not be diluted by a slow movement (low risk) in the far layer. Layering can preserve local high-risk signals.

[0069] The target area can be represented as: Where, represents the target area, represents the dense optical flow field of the whole frame, with a shape of H×W×2, H represents the total height of the frame 10 CN 121725397 A in the video manual 7 / 15 page 10 CN 121725397 A, W represents the total width of the video frame, and represents the starting line of the target area, for example, = 0.4×H, that is, only the lower half 60% of the screen area is considered.

[0070] In one possible implementation, when layering the target area, the layering parameters are first determined, including the number of layers, the pixel height of each layer, etc. Assuming the number of layers is 3, that is, L=3, the corresponding pixel height of each layer is: The pixel range of the i-th layer is [ ], , indicating the near layer, which is closest to the vehicle and prioritizes high risk, , indicating the far layer.

[0071] Step S342, determine the corresponding motion speed according to the motion components in each sub-region.

[0072] Wherein, the motion component is the component in the detection direction. The motion speed corresponding to the sub-region refers to the average value of the motion speeds of all pixels in the sub-region. When the risk category to be detected is vehicle collision risk, the above motion component is the vertical motion component.

[0073] The extraction formula of the vertical motion component is as follows: Wherein,The optical flow field corresponding to the target area is characterized by the third dimension of the optical flow field as [u, v], where 0 represents the horizontal motion component and 1 represents the vertical motion component.

[0074] The motion speed of the i-th layer is: where represents the motion speed of the i-th layer, represents the total number of pixels contained in the i-th layer, and represents the spatial region corresponding to the i-th layer.

[0075] The motion speeds of different layers represent the overall approach trend of objects in different longitudinal distance bands. The near layer reflects the emergency close-range risk, the middle layer reflects the normal following state, and the far layer reflects the long-distance traffic flow. Layering enables the system to have a risk perception hierarchy from near to far. If the motion speed of the entire target area is directly calculated without layering, the motion speed of the near layer will be lowered by the motion speed of the far layer, and the risk situation of the near layer will be missed.

[0076] Step S343: Determine the initial risk detection result of the corresponding video frame according to each motion speed.

[0077] The initial risk detection result includes at least the risk level corresponding to the video frame. The risk level can be divided according to the actual situation, such as high risk, medium risk, low risk, etc.

[0078] In one possible implementation, the initial risk detection result can be directly determined based on the motion speed and the speed range corresponding to the risk level. For example, if the motion speed is within the speed range corresponding to low risk, then the risk level of the video frame is low risk. The mapping relationship between the risk level and the corresponding speed range is also predetermined.

[0079] In one possible implementation, the initial risk detection result can be determined based on both motion speed and TTC (Time To Collision). Specifically, based on the motion speed corresponding to the sub-region and the relative position of the sub-region within the target region, the corresponding time to collision is determined. Based on the motion speed and the time to collision, the risk level of the corresponding video frame is determined.

[0080] The TTC calculation formula is as follows: (Instruction manual, page 8 / 15, CN 121725397 A) Where, represents the estimated distance-collision time of the i-th layer, represents the distance level weight corresponding to the i-th layer, reflecting the actual distance of the corresponding layer. Example settings: near layer, middle layer, far layer. In actual use, it can be set by calibration or experience to compensate for the difference in the actual speed corresponding to the same pixel speed at different distances, represents the absolute value of the i-th layer's motion speed, and ε is a very small constant set to prevent division by zero errors, for example, it can be 10⁶.

[0081] Optionally, a speed risk score is determined based on the motion speed, a TTC risk score is determined based on TTC, then a single-layer risk score for each sub-region is determined based on the speed risk score and the TTC risk score, and then the corresponding whole-frame risk score is determined based on the single-layer risk scores of all sub-regions.

[0082] Wherein, the mapping relationship between motion speed and the corresponding motion speed score, and the mapping relationship between TTC and the corresponding TTC risk score are all preset.

[0083] For example, the motion speed scoring formula is: where, refers to the first...Motion speed score of sub-region.

[0084] The TTC risk score formula is: Where, refers to the TTC risk score of the i-th sub-region.

[0085] The single-layer risk score corresponding to the i-th sub-region is: The higher the score, the higher the risk level.

[0086] The whole frame risk score can be the score corresponding to the most dangerous layer, that is: It is easy to understand that the method shown in Figure 3 can be applied to the initial detection of any risk category with the detection direction being vertical. If the risk category is not a collision risk, the corresponding initial risk detection result can be determined only based on the motion speed, without calculating TTC. It is worth noting that motion speed can reflect the motion change pattern. For example, if the above-mentioned risk category to be detected is the risk of drinking behavior, then the target area takes video frames from 20% to 80%, and the layers can be divided according to the corresponding height of the head, shoulders, and hands to obtain the corresponding sub-regions. The speed of each sub-region is calculated separately. If the sub-region speed of multiple consecutive video frames indicates that the sub-region corresponding to the hand moves first, then the sub-region corresponding to the shoulder moves, and finally the sub-region corresponding to the head moves, such a motion pattern may be drinking. If the detection direction is horizontal, the method shown in Figure 3 can also be used for initial risk detection, only requiring a change in direction, which will not be elaborated here.

[0087] In one possible implementation, the risk category to be detected is the risk of drinking behavior, and the detection direction is vertical. The corresponding method for determining the initial risk detection result is to determine the risk level of the corresponding video frame based on the movement speed corresponding to the sub-region. Specification 9 / 15 pages 12 CN 121725397 A

[0088] In one possible implementation, after determining the risk level of the video frame according to the above method, the video frame is added to the risk video frame sub-sequence corresponding to the risk level. For example, if the risk level of the video frame is high risk, the video frame can be submitted to the high risk video frame sub-sequence.

[0089] In one possible implementation, in response to the existence of a number of consecutive data frames with risk levels higher than a preset risk level, each of the video frames is added to the risk video frame sub-sequence corresponding to the risk level.

[0090] Specifically, if a predetermined number of consecutive video frames have a risk level higher than a preset risk level, these consecutive video frames are added to the risk video frame subsequence corresponding to the risk level; otherwise, these video frames are discarded. By introducing temporal continuity constraints and a structured event aggregation mechanism, the accuracy, interpretability, and practicality of risk detection are significantly improved without increasing the complexity of the perception module.

[0091] High risk in a single frame may be caused by image jitter, sudden changes in illumination, occlusion, or optical flow calculation errors, and does not have behavioral continuity. By requiring a predetermined number of consecutive frames (e.g., ≥3 frames) to exceed the risk threshold, isolated false alarms can be filtered out, significantly improving the stability and reliability of the detection results. Real risk events (such as emergency vehicle approach or drinking behavior) usually have several consecutive frames.The above motion characteristics. This mechanism ensures that risky behavior is recorded only when it has temporal continuity, which is more in line with the behavior laws of the physical world and improves the semantic rationality of detection. Continuous high-risk frames are organized into risk video frame subsequences to form traceable, replayable, and interpretable event units, which facilitates subsequent review, accident evidence collection, or contextual understanding of large models, and avoids fragmented judgment caused by scattered frames. Different risk levels can generate corresponding risk video frame subsequences, so that not only can it be determined whether there is a risk, but also the duration of the risk, whether the risk level has been upgraded / downgraded, etc., providing a data foundation for refined risk assessment and alarm strategies.

[0092] For example, a predetermined number of 3 is set, and the preset risk level is high risk. If the risk level of three consecutive frames, 100, 101, and 102, is high risk, then these three frames are added to the high-risk video frame subsequence. If the risk level of frame 103 is downgraded to medium risk, then the subsequence terminates here. This subsequence is then used as a complete event input to the visual large model, which can be used to determine whether it constitutes a real dangerous driving behavior.

[0093] Step S400: Determine the risk statistics information corresponding to the video to be detected based on the initial risk detection results.

[0094] Wherein, the risk statistics information includes at least one risk video frame subsequence corresponding to a risk level, and / or, the time point corresponding to each video frame in the risk video frame subsequence. In addition, the risk statistics information may also include the analysis period, the number of detections, the number of frames corresponding to each risk level, the risk distribution, the detection confidence level, etc.

[0095] For example, the risk statistics are as follows: "Analysis period: 00:00.00 to 00:34.00 (duration: 52.89 seconds) Total number of assessments: 341 Average processing time: 100.19 milliseconds per frame Severe risk: 0 times (0.0%) High risk: 1 time (0.3%) Medium risk: 5 times (1.5%) Low risk: 334 times (97.9%) Mean time to collision (TTC): 41.8 seconds Risk branch distribution: Approach risk (APPROACHING_RISK): 1 time (0.3%) Safe distance (SAFE_DISTANCE): 253 times (74.2%) Unknown (UNKNOWN): 1 time (0.3%) Specification 10 / 15 pages 13 CN 121725397 A Unreliable detection (UNRELIABLE_DETECTION): 86 times (25.2%)" High-risk frames: frames 480 to 500” Step S500: Fill the risk statistics information into the prompt template corresponding to the risk category to be detected, and generate the corresponding risk inspection prompt statement.

[0096] The prompt template may include information such as the risk category to be detected and output requirements.

[0097] Optionally, the prompt template may also include visual description text. Specifically, based on the inter-frame motion of the risk video frame sequence.Features and time sequence generate brief visual description text, such as a vehicle rapidly approaching from the front, lasting for 3 frames, etc.

[0098] Taking vehicle collision risk as an example, the prompt template can be: "Understanding vehicle dynamic collision risk: Please analyze potential collision events according to the [Risk Statistics Information Fill-in Location], and output a structured risk report in combination with the TTC (Time-To-Collision) context.

[0099] Please strictly follow the following steps to analyze the input information and images: 1. Perform image sequence description: Provide an objective text description of the input frame sequence, including key visual elements (e.g., vehicle type, position, direction of movement, environmental features, distance changes).

[0100] 2. Collision risk analysis 3. Conclusion output: A unique conclusion must be selected from the following three categories: Collision risk: If the image clearly shows that a collision is about to occur No collision risk: If the image shows a safe state More information is needed: If the information is incomplete, supplementary information is needed for judgment" It is easy to understand that the above prompt template is only an example, and the specific settings of the prompt template are not limited in this application embodiment.

[0101] By filling the risk statistics and other data into the prompt template corresponding to the risk category to be detected, the corresponding risk inspection prompt statement can be constructed.

[0102] Step S600: Input the risk inspection prompt statement into the visual big model to determine the corresponding risk detection result.

[0103] In one possible way, if the risk statistics include risk video frames, then only the risk inspection prompt statement is input into the visual big model.

[0104] Optionally, the video frames in the risk video frame sequence can also be preprocessed and encoded. The risk video frame sequence is a collection of multiple risk video frame subsequences. The video frames in the risk video frame sequence are preprocessed by scaling, normalization, etc., and encoded into a format that can be input into the visual big model, such as an image sequence tensor.

[0105] In one possible way, the risk statistics do not include risk video frames, but only the time points corresponding to each video frame of the risk video frame. Then the risk inspection prompt statement and the video to be detected are input into the visual big model.

[0106] Figure 4 is a schematic diagram of the video frame in an embodiment of the present invention. As shown in Figure 4, the video frame is the video frame corresponding to vehicle collision detection. Based on this video frame, the output of the large visual model is: Scene description: This scene is from the perspective of a bicycle dashcam, showing a car driving on the road. A large, cartoon-style "horse head" decoration is installed on the rear window of the vehicle. The horse head is white with brown mane, and its exaggerated shape sways in the wind. The background is a clear sky and buildings on both sides of the road, and other vehicles are driving normally on the road. The top of the screen has text such as "Introduction Price" and "Starting from 1 cent", and the bottom has the subtitle "Unexpectedly, I tried it out." The overall style is obviously advertising.

[0107] Collision Risk Analysis: Although the distance or relative speed to the vehicle in front is not directly displayed in the image, based on the information provided by the professional optical flow processing model, it can be concluded that "the video vehicle is at risk of colliding with the vehicle in front: high risk: approach risk specification 11 / 15 pages 14 CN 121725397 A (APPROACHING_RISK); TTC has exceeded the threshold.

[0108] It can be determined that: TTC has exceeded the safety threshold, which means that the relative distance between the current vehicle and the target in front is decreasing rapidly, and there is a high risk of an impending collision.

[0109] "Approach risk" means that the vehicle is detected approaching the object in front at a dangerous speed, and is in a critical state even if it has not yet made contact.

[0110] Conclusion: There is or is about to be a car collision event in this image.

[0111] It should be understood that the above risk detection method is not limited to a single risk category. In practical applications, the selection method of the associated motion components in the optical flow field and the content structure of the prompt template can be adjusted according to the characteristics of the risk category to be detected. For example, in vehicle collision risk monitoring scenarios, the vertical negative motion component in the lower half of the image can be focused on; while in drinking behavior detection scenarios, the coordinated motion characteristics of the hand and mouth areas can be focused on. Therefore, this method can be applied to video intelligent analysis tasks in multiple fields such as traffic, security, and content review.

[0112] The method of this embodiment first acquires the video to be detected, calculates the optical flow field between adjacent frames in the video to be detected, then extracts the motion components (such as vertical or horizontal displacement) related to the preset risk category, generates the initial risk detection result for each frame, then statistically analyzes all the initial detection results to form structured risk statistics, fills the risk statistics into the prompt template corresponding to the preset risk category, automatically generates risk inspection prompt statements, and finally inputs the statements into the visual big model to output the final risk detection result. By integrating optical flow field analysis, vertical layering of target areas, construction of continuous high-risk frame subsequences, structured prompt templates and visual big model inference, efficient, robust and interpretable video risk detection is achieved: On the one hand, it can capture dynamic clues related to risk without relying on object detection or a large amount of labeled data, based solely on pixel-level motion features. On the other hand, it effectively distinguishes the motion trends of near, middle, and far regions by simulating scene depth through layered target regions, avoiding the dilution of key risk signals by background noise. Simultaneously, it introduces a continuous frame verification mechanism, forming a risk video frame subsequence only when a predetermined number of adjacent frames consistently exceed a preset risk level, significantly suppressing instantaneous false detections. Furthermore, the statistical results are filled into the natural language prompt template corresponding to the risk category to generate standardized risk verification statements, which are then used by a large visual model for semantic-level judgment, balancing low-level evidence with high-level understanding, and greatly improving the generalizability of complex and ambiguous behaviors.The overall solution is highly versatile and can be flexibly adapted to various application scenarios such as vehicle collision warning and recognition of drinking or smoking behavior by configuring motion components and prompt templates associated with different risk categories.

[0113] Figure 5 is a flowchart of the video detection method corresponding to vehicle collision risk. As shown in Figure 5, the video detection method corresponding to vehicle collision risk includes the following steps: Step S501, read the video to be detected frame by frame.

[0114] Wherein, the video to be detected is a video containing the road scene in front of the vehicle during driving.

[0115] Step S502, determine whether the current video frame read is the first frame.

[0116] If yes, go to step S501; if not, go to step S503.

[0117] Step S503, determine the optical flow field corresponding to the current video frame and the previous video frame.

[0118] Step S504, determine the target area from the optical flow field.

[0119] Step S505, extract the vertical motion component from the target area.

[0120] Step S506: Vertically divide the target area into multiple sub-regions.

[0121] Step S507: Determine the corresponding motion speed according to the motion components within each sub-region.

[0122] Step S508: Determine the corresponding time to collision based on the motion speed of the sub-region and its relative position within the target area.

[0123] Step S509: Determine the risk level of the corresponding video frame based on the motion speed and the time to collision.

[0124] Step S510: Determine whether the corresponding video frame meets the addition conditions.

[0125] The addition condition is that there are at least a predetermined number of consecutive data frames before and after the video frame with risk levels higher than a preset risk level.

[0126] If satisfied, proceed to step S511; otherwise, proceed to step S501.

[0127] Step S511: Add the video frame to the risk video frame sub-sequence corresponding to the risk level.

[0128] Step S512: Preprocess and encode the video frames in the risk video frame sequence.

[0129] Step S513: Generate a brief visual description text based on inter-frame motion features and temporal order.

[0130] Step S514: Construct a corresponding risk inspection prompt statement based on the prompt template.

[0131] Specifically, the risk statistics, visual description text, and other information are filled into the prompt template corresponding to the risk category to be detected to generate a corresponding risk inspection prompt statement.

[0132] Step S515: Input the risk inspection prompt statement into the visual large model to determine the corresponding risk detection result.

[0133] Figure 6 is a flowchart of the data processing method of the visual large model according to an embodiment of the present invention. As shown in Figure 6, the data processing method of the visual large model includes the following steps:Step S601: Read video frames from the risk video frame sequence.

[0134] Step S602: Perform single-frame and / or continuous frame information analysis on the video frame.

[0135] Specifically, perform single-frame video frame detection on the video frame. Perform joint semantic understanding of the temporal context of the video frame and the multiple video frames before and after it to extract high-level, interpretable risk behavior features.

[0136] Step S603: Generate risk inspection results.

[0137] Specifically, the risk inspection results include risk behavior description, confidence level, contextual explanation, etc.

[0138] Step S604: Confirm whether the confidence level of the risk inspection results is greater than the preset confidence level.

[0139] If yes, proceed to step S605; otherwise, proceed to step S606.

[0140] Step S605: Determine the risk inspection information corresponding to the video frame.

[0141] Step S606: Confirm whether the confidence level of the risk inspection results is less than the preset confidence level.

[0142] If yes, proceed to step S607; otherwise, proceed to step S608.

[0143] Step S607: Delete the risk inspection information corresponding to the video frame.

[0144] Step S608: Expand the analysis scope and supplement more preceding and following video frames.

[0145] Step S609: Determine whether there are any uninspected video frames in the risk video frame sequence.

[0146] If yes, proceed to step S601; otherwise, proceed to step S610.

[0147] Step S610: Generate risk inspection results based on the risk inspection information corresponding to each video frame.

[0148] The specific implementation methods of each step in this embodiment can be found in the above embodiments, and will not be repeated here.

[0149] The method of this embodiment of the invention does not rely on complex modules such as target detection. It can capture key dynamic features and complete the initial risk detection by optical flow alone. Then, it combines the prompt template with the visual large model to realize risk verification and achieve high-precision semantic understanding. It can improve the efficiency and semantic accuracy of risk identification while reducing computational overhead. It also has good generalization ability and can be adapted to a variety of risk scenarios, combining practicality and scalability.

[0150] Figure 7 is a schematic diagram of the video detection device of this embodiment of the invention. As shown in Figure 7, the video detection device includes: an acquisition module 701, used to acquire the video to be detected.

[0151] A first determination module 702, used to determine the optical flow field corresponding to each adjacent video frame in the video to be detected. Specification 13 / 15 pages 16 CN 121725397 A

[0152] A second determination module 703, used to determine the initial risk detection result corresponding to each video frame according to the motion component in the optical flow field associated with the risk category to be detected.

[0153] The third determining module 704 is used to determine the risk statistics information corresponding to the video to be detected based on the preliminary risk detection results.

[0154] The generation module 705 is used to fill the risk statistics information into the prompt template corresponding to the risk category to be detected, and generate the corresponding risk inspection prompt statement.

[0155] The detection module 706 is used to input the risk inspection prompt statement into the visual large model and determine the corresponding risk detection result.

[0156] The device of this embodiment of the invention is used to acquire the video to be detected, calculate the optical flow field between each adjacent frame in the video to be detected, then extract the motion components (such as vertical or horizontal displacement) related to the preset risk category, generate the initial risk detection result of each frame, then statistically analyze all the initial detection results to form structured risk statistics information, fill the risk statistics information into the prompt template corresponding to the preset risk category, automatically generate the risk inspection prompt statement, and finally input the statement into the visual large model to output the final risk detection result. The above-mentioned device does not rely on complex modules such as target detection. It can capture key dynamic features and complete the initial risk detection by optical flow alone. Then, it combines the prompt template with the visual large model to realize risk verification and achieve high-precision semantic understanding. It can improve the efficiency and semantic accuracy of risk identification while reducing the computational overhead. It also has good generalization ability and can be adapted to a variety of risk scenarios, combining practicality and scalability.

[0157] Figure 8 is a schematic diagram of the electronic device of the present invention. In this embodiment, the electronic device 800 includes a server, a terminal, etc. As shown in Figure 8, the electronic device 800 includes at least one processor 801; and a memory 802 that is communicatively connected to at least one processor 801; and a communication component 803 that is communicatively connected to a scanning device. The communication component 803 receives and sends data under the control of the processor 801. The memory 802 stores instructions that can be executed by at least one processor 801. The instructions are executed by at least one processor 801 to implement the above-mentioned video detection method.

[0158] Specifically, the electronic device includes one or more processors 801 and a memory 802. Figure 8 uses one processor 801 as an example. The processor 801 and memory 802 can be connected via a bus or other means. Figure 8 shows an example of connection via a bus. Memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The processor 801 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 802, thereby implementing the aforementioned video detection method.

[0159] Memory 802 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store an option list, etc. Furthermore, memory 802 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device or flash memory.Devices, or other non-volatile solid-state storage devices. In some embodiments, memory 802 may optionally include memory remotely located relative to processor 801, and these remote memories can be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0160] One or more modules are stored in memory 802, and when executed by one or more processors 801, perform the video detection method in any of the above method embodiments.

[0161] The above products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0162] The technical solution of this embodiment first acquires the video to be detected, calculates the optical flow field between adjacent frames in the video to be detected, then extracts motion components (such as vertical or horizontal displacement) related to the preset risk category, generates the initial risk detection result for each frame, then statistically analyzes all the initial detection results to form structured risk statistics, fills the corresponding preset risk category prompt template with the risk statistics, automatically generates risk inspection prompt statements, and finally inputs the statements into the visual big model to output the final risk detection result. The above technical solution does not rely on complex modules such as target detection, but can capture key dynamic features and complete the initial risk detection by optical flow alone. Then, by combining the prompt template with the visual big model, risk inspection is realized, achieving high-precision semantic understanding. It can improve the efficiency and semantic accuracy of risk identification while reducing computational overhead, and has good generalization ability, which can be adapted to various risk scenarios, and has both practicality and scalability.

[0163] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which, when executed by a processor, implements some or all of the above-described method embodiments.

[0164] Another embodiment of the present invention relates to a computer program product, including a computer program / instructions, which, when executed by a processor, implements some or all of the above-described method embodiments.

[0165] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a read-only memory (ROM), and a random access memory (RAM).Memory), magnetic disks, optical disks, and other media capable of storing program code.

[0166] The above-described product can execute the method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of this application.

[0167] The above descriptions are merely preferred embodiments of this application and are not intended to limit this application. For those skilled in the art, this application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the protection scope of this application. Instruction Manual 15 / 15 Page 18 CN 121725397 A Figure 1 Instruction Manual Appendix 1 / 7 Page 19 CN 121725397 A Figure 2 Figure 3 Instruction Manual Appendix 2 / 7 Page 20 CN 121725397 A Figure 4 Instruction Manual Appendix 3 / 7 Page 21 CN 121725397 A Figure 5 Instruction Manual Appendix 4 / 7 Page 22 CN 121725397 A Figure 6 Instruction Manual Appendix 5 / 7 Page 23 CN 121725397 A Figure 7 Instruction Manual Appendix 6 / 7 Page 24 CN 121725397 A Figure 8 Instruction Manual Appendix 7 / 7 Page 25 CN 121725397 A Abstract Embodiments of the present application disclose a video detection method and device. The method of the embodiments of the present application first acquires a video to be detected, calculates the optical flow field between each adjacent frame in the video, then extracts the motion component (such as vertical or horizontal displacement). related to the preset risk category, generates the risk preliminary detection result of each frame, then carries out statistics on allpreliminary detection results to form the structured risk statistical information, fills the information into the prompt template corresponding to the preset risk category, automatically generates the risk inspection prompt sentence, and finally inputs the sentence into the visual large model to output the final risk detection result. The above method does not need to rely on complex modules such as target detection, but can capture key dynamic features through optical flow alone to complete preliminary detection, then combine the prompt template with the visual large model to realize risk inspection, which can reduce the computing overhead while improving the risk identification efficiency and semantic accuracy, and has good generalization ability, can adapt to various risk scenarios, and has expansibility.

Claims

1. A video detection method, characterized in that, The method includes: Obtain the video to be tested; Determine the optical flow field corresponding to each adjacent video frame in the video to be detected; Based on the motion components in each optical flow field that are associated with the risk category to be detected, the initial risk detection result corresponding to each video frame is determined; Based on the preliminary risk detection results, determine the risk statistics information corresponding to the video to be detected; Fill the risk statistics information into the prompt template corresponding to the risk category to be detected, and generate the corresponding risk inspection prompt statement; Input the risk inspection prompt statement into the visual large model to determine the corresponding risk detection result.

2. The method according to claim 1, characterized in that, The step of determining the initial risk detection result corresponding to each video frame based on the motion components in each optical flow field associated with the risk category to be detected includes: The target region is determined from each of the optical flow fields; Determine the corresponding detection direction based on the risk category to be detected; Extract the motion component in the detection direction from the target region; The initial risk detection result of the corresponding video frame is determined based on the motion components.

3. The method according to claim 2, characterized in that, Determining the target region from each of the optical flow fields includes: The video frames are used to perform image recognition to determine the corresponding detection targets; The target area is determined based on the detection target.

4. The method according to claim 2, characterized in that, Determining the target region from each of the optical flow fields includes: Determine the corresponding preset location based on the risk category to be detected; The region in the video frame located at the preset position is determined as the target region.

5. The method according to claim 2, characterized in that, The step of determining the initial risk detection result of the corresponding video frame based on the motion components includes: The target region is divided into multiple sub-regions in the detection direction; The corresponding motion speed is determined based on the motion components within each sub-region; The initial risk assessment result for the corresponding video frame is determined based on the speed of each movement.

6. The method according to claim 5, characterized in that, The risk category to be detected is vehicle collision risk, and the detection direction is the vertical direction; The determination of the initial risk detection result for the corresponding video frame based on each of the motion speeds includes: The corresponding time from collision is determined based on the movement speed of the sub-region and the relative position of the sub-region within the target region. The risk level of the corresponding video frame is determined based on the motion speed and the time from collision. The video frame is added to the risk video frame subsequence corresponding to the risk level.

7. The method according to claim 5, characterized in that, The risk category to be detected is the risk of drinking behavior, and the detection direction is vertical. The determination of the initial risk detection result for the corresponding video frame based on each of the motion speeds includes: The risk level of the corresponding video frame is determined based on the motion speed corresponding to the sub-region. The video frame is added to the risk video frame subsequence corresponding to the risk level.

8. The method according to claim 6 or 7, characterized in that, Adding the video frame to the risk video frame subsequence corresponding to the risk level includes: In response to the presence of a number of consecutive data frames with a risk level higher than a preset risk level, each of the video frames is added to the risk video frame subsequence corresponding to the risk level.

9. The method according to claim 1, characterized in that, The risk statistics information includes at least one risk video frame subsequence corresponding to a risk level, and / or the time point corresponding to each video frame in the risk video frame subsequence.

10. The method according to claim 1, characterized in that, After acquiring the video to be detected, the method further includes: Determine the type of risk to be detected corresponding to the video to be detected.

11. A video detection device, characterized in that, The device includes: The acquisition module is used to acquire the video to be detected; The first determining module is used to determine the optical flow field corresponding to each adjacent video frame in the video to be detected; The second determining module is used to determine the initial risk detection result corresponding to each video frame based on the motion components in the optical flow field that are associated with the risk category to be detected. The third determining module is used to determine the risk statistics information corresponding to the video to be detected based on the initial risk detection results of each of the above-mentioned risks. The generation module is used to fill the risk statistics information into the prompt template corresponding to the risk category to be detected, and generate the corresponding risk inspection prompt statement. The detection module is used to input the risk inspection prompt statement into the visual large model and determine the corresponding risk detection result.

12. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-10.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method as described in any one of claims 1-10.

Images