Video identification method and device, medium and program product

Abstract

The embodiment of the invention discloses a video identification method and device, a medium and a program product. The method comprises the following steps: aiming at a target video acquired in real time, performing sliding traversal on the target video from a starting frame based on a sliding window; wherein the sliding window at least comprises a first preset number of first image frames; the first image frame is an image frame of which a detection result does not meet an abnormal condition; in the traversing process, counting the number of second image frames in the sliding window; the second image frame is an image frame with a detection result meeting an abnormal condition; and determining an actual identification result of the second image frames according to the number of the second image frames in the sliding window. According to the scheme, the abnormal frame can be detected in real time in a local range through traversal of the sliding window, so that the credibility of the abnormal frame is accurately judged according to the assistance of the local video in the latest period of time, and the situation that the real video recognition result is influenced by the accidental abnormal frame is avoided.

Description

Technical Field

[0001] This application relates to the field of video recognition technology, and in particular to a video recognition method, device, medium and program product. Background Technology

[0002] In recent years, artificial intelligence video analytics technology has developed rapidly, and multi-frame detection in real-time video stream processing has become a key technology in fields such as security tracking and industrial quality inspection. However, in practical applications, unstable detection results sometimes occur. This is mainly due to sudden changes in lighting, weather conditions, or occasional instability in the detection algorithm, leading to abnormal detection results, such as missed detections, false detections, or data mutations. Although these problems are usually temporary, if they occur during continuous detection processes, they can significantly impact the final detection results. For example, in security monitoring, a brief loss of detection may lead to target loss, affecting the continuity of tracking; in industrial quality inspection, a false detection may trigger an error alarm, causing production line shutdown; and in operational statistics, missing a target may skew the statistical results.

[0003] Therefore, we need a method to automatically identify and process these anomalous data without affecting real-time performance. This requirement is particularly prominent in scenarios with stringent real-time requirements (such as autonomous driving perception systems), where traditional batch processing data analysis methods are unsuitable due to high computational latency. Existing technologies mainly address recognition jitter in two ways: front-end optimization: such as improving the robustness of the target detection model (e.g., adding data augmentation, optimizing the loss function) or introducing physical stabilization devices (e.g., a stabilizing gimbal), but these methods cannot completely eliminate random errors at the algorithm level, and the hardware costs are high. Post-processing smoothing: some solutions use simple mean filtering or fixed window averaging to smooth the detection results, but these methods lack the ability to dynamically distinguish anomalous data, which may lead to over-smoothing of valid signals (e.g., real target movement being misjudged as noise). Summary of the Invention

[0004] This application provides a video recognition method, device, medium, and program product that takes into account the dynamic adaptability of local time windows and accurately detects abnormal frames based on current scene changes.

[0005] According to one aspect of this application, a video recognition method is provided, the method comprising:

[0006] For the target video acquired in real time, the target video is traversed by sliding window starting from the starting frame; wherein, the sliding window contains at least a first preset number of first image frames; the first image frames are image frames whose detection results do not meet the abnormal conditions;

[0007] During the traversal, the number of second image frames in the sliding window is counted; wherein, the second image frame is the image frame whose detection result meets the abnormal condition;

[0008] The actual recognition result of the second image frame is determined based on the number of second image frames in the sliding window.

[0009] According to one aspect of this application, a video recognition device is provided, the device comprising:

[0010] The sliding traversal module is used to perform sliding traversal of the target video acquired in real time, starting from the starting frame, based on a sliding window; wherein the sliding window contains at least a first preset number of first image frames; the first image frames are image frames whose detection results do not meet the abnormal conditions;

[0011] The statistics module is used to count the number of second image frames in the sliding window during the traversal process; wherein, the second image frame is the image frame whose detection result meets the abnormal condition;

[0012] The actual recognition result determination module is used to determine the actual recognition result of the second image frame based on the number of second image frames in the sliding window.

[0013] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory that is communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor can perform the video recognition method of any embodiment of this application.

[0017] According to another aspect of this application, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the video recognition method of any embodiment of this application.

[0018] According to another aspect of this application, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the video recognition method of any embodiment of this application.

[0019] The technical solution of this application embodiment, for a target video acquired in real time, traverses the target video starting from the initial frame using a sliding window; wherein the sliding window contains at least a first preset number of first image frames; the first image frames are image frames whose detection results do not meet the abnormal conditions; during the traversal process, the number of second image frames in the sliding window is counted; wherein the second image frames are image frames whose detection results meet the abnormal conditions; based on the number of second image frames in the sliding window, the actual recognition result of the second image frame is determined. The above solution can, for a target video acquired in real time, update the latest local recognition data based on the real-time sliding of the sliding window, thereby detecting the authenticity of abnormal frames in the local scene, avoiding the problem of incorrect actual recognition results due to single-frame false detection, and reducing time complexity by using local data detection based on the sliding window.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a video recognition method provided in an embodiment of this application;

[0023] Figure 2 A flowchart illustrating a video recognition method provided in another embodiment of this application;

[0024] Figure 3 This is a first schematic diagram of a sliding window provided in another embodiment of this application;

[0025] Figure 4 This is a second schematic diagram of a sliding window provided in another embodiment of this application;

[0026] Figure 5 A third schematic diagram of a sliding window provided in another embodiment of this application;

[0027] Figure 6 A fourth schematic diagram of a sliding window provided in another embodiment of this application;

[0028] Figure 7 A fifth schematic diagram of a sliding window provided in another embodiment of this application;

[0029] Figure 8 A sixth schematic diagram of a sliding window provided in another embodiment of this application;

[0030] Figure 9 A seventh schematic diagram of a sliding window provided in another embodiment of this application;

[0031] Figure 10 An eighth schematic diagram of a sliding window provided in another embodiment of this application;

[0032] Figure 11 A flowchart illustrating a video recognition method provided in yet another embodiment of this application;

[0033] Figure 12 This is a schematic diagram of the structure of a video recognition device provided in an embodiment of this application;

[0034] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," "third," "fourth," "actual," "preset," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Figure 1This is a flowchart illustrating a video recognition method provided in an embodiment of this application. This embodiment is applicable to situations where robotic process automation (RPA) programs are executed when elements on a webpage change. The method can be executed by a video recognition device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0038] S110. For the target video acquired in real time, the target video is traversed by sliding window starting from the starting frame; wherein the sliding window contains at least a first preset number of first image frames; the first image frames are image frames whose detection results do not meet the abnormal conditions.

[0039] The target video can be a video captured for a target scene, containing a preset object to be monitored. In some applications, certain conditions are set, and the monitoring is based on whether the preset object in the video meets these conditions. For example, the condition might be that the preset object must remain in the target video frame for a preset duration. In this scenario, the target video is identified frame by frame to determine whether the preset object continuously appears in each image frame of the target video. The sliding window is a pre-defined window that can encompass a certain number of consecutive image frames. The number of image frames contained in the sliding window is not limited, but the number of the first image frames in the sliding window must be a first preset number. In addition to the first image frame, the sliding window can also contain other image frames. The first image frame is the image frame whose detection result does not meet the abnormal condition. The abnormal condition can be set according to the actual situation. For example, in the scenario mentioned above, the abnormal condition could be that the preset object disappears from the target video. If the recognition result of the image frame is that the preset object is detected, then the detection result of the image frame is determined to not meet the abnormal condition. The abnormal condition can also be set to other conditions in other scenarios.

[0040] In this embodiment, for the target video acquired in real time, starting from the starting frame, a sliding window is used to traverse the target video. For example, consecutive image frames are selected from the starting frame and placed within the sliding window. The total number of image frames contained within the sliding window is not limited, only a first preset number of first image frames are required. Generally, the last frame in the sliding window is the first image frame. In the sliding window starting from the starting frame, since it is not yet possible to determine whether each image frame is the first image frame, the first preset number of image frames starting from the starting frame are directly selected as the first image frames and placed in the sliding window. For subsequent image frames, it is further determined whether they are first image frames based on the statistical data of the image frames in the sliding window. The sliding step size used for sliding the sliding window can be determined according to the actual situation. After the sliding window slides according to the sliding step size, it is checked whether the number of first image frames contained in the sliding window reaches the first preset number. If it does, the sliding is completed. If it does not reach the first preset number, the image frames are traversed and added to the sliding window until the number of first image frames in the sliding window reaches the first preset number.

[0041] S120. During the traversal process, the number of second image frames in the sliding window is counted; wherein, the second image frame is the image frame whose detection result meets the abnormal condition.

[0042] In this context, the second image frame is the one whose detection result does not meet the abnormal conditions, meaning the detection result is the opposite of the first image frame. The abnormal conditions are as described in the above embodiment and can be set according to actual circumstances.

[0043] In this embodiment, second image frames that meet abnormal conditions generally affect the video recognition result. For example, if the detection results of all image frames in the target video do not meet the abnormal conditions, the preset object in the image frame is considered normal. If the detection results of an image frame in the target video meet the abnormal conditions, the preset object in the image frame is considered abnormal, and an abnormality alarm is generally triggered in this case. However, there are some cases where the abnormality of the preset object in the image frame is accidental and not normal, or it is caused by an image frame recognition error. Therefore, if this is determined as the actual recognition result of the target video, it will lead to a misjudgment of the target video recognition result. Therefore, in this embodiment, for image frames in a local sliding window, based on the behavioral characteristics of the preset object within the local time range contained in the sliding window, the number of second image frames in the sliding window is counted, and the number of second image frames determines whether the abnormality of the preset object reflected by the second image frame is credible or a misjudgment.

[0044] S130. Determine the actual recognition result of the second image frame based on the number of second image frames in the sliding window.

[0045] For example, typically, one image frame in a sliding window corresponds to the movement of a preset object within 1 / n (seconds), where n is generally 20-30. This means that the movement time of the preset object reflected in one image frame is very short. If the preset object exhibits a certain behavior, this behavior will generally appear continuously in multiple image frames, rather than just a few isolated frames. Based on this, in this embodiment, the actual recognition result of the second image frame is determined according to the number of second image frames in the sliding window, thereby accurately determining whether the anomaly of the preset object reflected in the second image frame is a credible, actual anomaly, or a misjudgment.

[0046] The technical solution of this application embodiment, for a target video acquired in real time, traverses the target video starting from the initial frame using a sliding window; wherein the sliding window contains at least a first preset number of first image frames; the first image frames are image frames whose detection results do not meet the abnormal conditions; during the traversal process, the number of second image frames in the sliding window is counted; wherein the second image frames are image frames whose detection results meet the abnormal conditions; based on the number of second image frames in the sliding window, the actual recognition result of the second image frame is determined. The above solution can, for a target video acquired in real time, update the latest local recognition data based on the real-time sliding of the sliding window, thereby detecting the authenticity of abnormal frames in the local scene, avoiding the problem of incorrect actual recognition results due to single-frame false detection, and reducing time complexity by using local data detection based on the sliding window.

[0047] Figure 2 This is a flowchart illustrating a video recognition method according to another embodiment of this application. This embodiment is an optimization based on the above embodiment; solutions not described in detail in this embodiment are found in the above embodiment. Figure 2 As shown, the method in this embodiment of the application specifically includes the following steps:

[0048] S210. For the target video acquired in real time, the sliding window is made to slide according to the sliding step size, and the image frames entering the sliding window this time are detected.

[0049] For example, regarding the current sliding process of the sliding window, specifically, the sliding window is first slid according to the sliding step size, and the image frames entering the sliding window this time are detected. For example, assuming the first preset number is 3 frames and the sliding step size is 1 frame, as follows... Figure 3 As shown, assuming the sliding window includes three image frames (frame 1, frame 2, and frame 3) after the previous slide, and the three image frames with checkmarks are all the first image frame, the sliding step size of this slide of 1 frame results in the sliding window as shown. Figure 4As shown, it includes frames 2, 3, and 4. Among them, frame 4 is the image frame that was slid into the sliding window this time. Therefore, frame 4 is detected to determine whether frame 4 is the first image frame.

[0050] S220. If the number of first image frames in the sliding window reaches a first preset number, then the sliding is completed.

[0051] For example, after detecting the image frame entering the sliding window, it can be determined whether the image frame entering the sliding window is the first image frame or the second image frame. The number of first image frames included in the sliding window is counted to see if it reaches a first preset number. If the number of first image frames reaches the first preset number, the sliding is considered complete, meaning the requirement that the sliding window contains at least the first preset number of first image frames is met. This ensures that the sliding window contains at least the first preset number of normally detected image frames, facilitating the determination of the distribution and number of abnormal frames based on this.

[0052] In this embodiment of the application, if the number of first image frames in the sliding window reaches a first preset number, then the sliding is completed, including:

[0053] If the number of the first image frames in the sliding window is equal to the first preset number, then the sliding is completed;

[0054] If the number of first image frames in the sliding window is greater than a first preset number, then the image sequence in the sliding window that includes the first preset number of first image frames starting from the last image frame is retained, and the remaining first image frames and second image frames in the sliding window are moved out of the sliding window.

[0055] For example, the number of first image frames within the sliding window is counted. If the number of first image frames equals a first preset number, the sliding operation is completed, and the sliding window contains the first preset number of first image frames. Figure 4 In this scenario, assuming frame 4 entering the sliding window is the first image frame, the sliding window contains 3 first image frames, which equals a first preset number, thus completing the current slide. If the number of first image frames in the sliding window is greater than the first preset number, then the image sequence containing the first preset number of first image frames, starting from the last image frame in the sliding window, is retained, and the remaining first and second image frames in the sliding window are removed from the sliding window. Figure 5 As shown, Figure 5 The middle section shows the result of the previous slide of the sliding window. Frame 2 marked with an "×" indicates it is the second image frame, while other image frames marked with a "√" indicate they are the first image frames. The sliding window is then slid further based on this result. Figure 6As shown, assuming frame 6 is detected as the first image frame, the number of first image frames within the sliding window exceeds a first preset number. Therefore, we can start from the last image frame, i.e., frame 6, and trace back to retain the image sequence containing the first preset number of first image frames, i.e., the image sequence composed of frames 4, 5, and 6. Frame 3 is then moved out of the sliding window, resulting in the sliding window as shown. Figure 7 As shown. The beneficial effect of the above scheme is that by extracting a first preset number of first image frames, a certain number of first image frames can be uniformly retained in the sliding window, so as to unify the standard for comparing the number of abnormal frames and improve the confidence of the abnormal frame identification results.

[0056] S230. If the number of first image frames in the sliding window is less than the first preset number, then the image frames located after the sliding window along the sliding direction of the sliding window are added to the sliding window in sequence, and the image frames entering the sliding window this time are detected until the number of first image frames in the sliding window reaches the first preset number.

[0057] For example, if the number of first image frames in the sliding window is less than a first preset number, then image frames following the sliding window along the sliding direction are added to the sliding window, and the image frames entering the sliding window this time are detected to determine whether the number of first image frames in the current sliding window reaches the first preset number. In the target video, the earlier the image frame, the earlier its corresponding timestamp; the later the image frame, the later its corresponding timestamp. The backward direction in the target video frame is also the sliding direction of the sliding window. Figure 8 As shown, the sliding window after the last slide contained a first preset number of first image frames. After this slide, the sliding window is as follows: Figure 9 As shown, it contains two first image frames, which is less than the first preset number. At this point, frame 8, following frame 7, is added to the sliding window, as shown. Figure 10 As shown, it is then checked whether frame 8 is the first image frame. If frame 8 is the first image frame, then the sliding window contains 3 first image frames, reaching the first preset number, and the current sliding ends. If frame 8 is the second image frame, then frame 9 is added to the sliding window, and it is checked whether frame 9 is the first image frame, and so on, until the sliding window contains the first preset number of first image frames.

[0058] In this embodiment of the application, the detection of the image frame entering the sliding window includes:

[0059] The preset objects in the image frame are identified, and the identification result is determined.

[0060] The detection result of the image frame is determined based on the recognition result; wherein the detection result includes whether the image frame is a first image frame or the image frame is a second image frame.

[0061] During the detection of image frames entering the sliding window, a preset object is identified in each newly entered image frame. The identification result is determined, including whether the preset object was identified, the confidence level, the position of the preset object in the image frame, and the number of preset objects of the same type. Based on the identification result, the detection result of the image frame is determined, thereby determining whether the image frame is the first image frame or the second image frame.

[0062] In this embodiment of the application, identifying a preset object in the image frame and determining the identification result includes at least one of the following:

[0063] Calculate the distance between the position of the preset object in the image frame and the position of the preset object in the previous frame;

[0064] Identify the number of preset objects of the same type in the image frame, and count the mode of the number of preset objects of the same type in each image frame of the sliding window obtained in the last sliding;

[0065] Determine the confidence level of the preset object identified in the image frame, and calculate the mean and variance of the confidence levels of the preset object identified in each image frame of the sliding window obtained in the previous sliding.

[0066] Accordingly, determining the detection result of the image frame based on the recognition result includes:

[0067] If at least one of the following conditions is met, the detection result of the image frame is determined to be the second image frame; otherwise, the detection result of the image frame is determined to be the first image frame:

[0068] The distance is greater than a preset distance threshold;

[0069] The difference between the quantity and the mode is greater than a preset difference threshold;

[0070] The confidence level is less than a preset confidence level; wherein the preset confidence level is the difference between the mean and twice the variance.

[0071] For example, in the process of detecting an image frame to determine whether it is the first image frame or the second image frame, specifically, during the process of recognizing the image frame, at least one of the following can be specifically identified: calculating the distance between the position of the preset object in the image frame and the position of the preset object in the previous frame, expressed as... Where D is the distance, x tand y t x is the position coordinate of the preset object in this image frame. t-1 and y t-1 This refers to the position coordinates of the preset object in the previous frame. For example, in... Figure 10 In the process, for frame 8 within the sliding window, the position of the preset object in frame 8 and the distance between the preset object's position in frame 7 can be identified. The number of preset objects of the same type in the image frames is identified, and the mode of the number of preset objects of the same type in each image frame of the previously obtained sliding window is counted. For example, for frame 8 within the sliding window, if the preset object is a red car, then the number of red cars in frame 8 is identified. The mode of the number of preset objects of the same type in each image frame of the previously obtained sliding window (e.g., ...) is also counted. Figure 8 The mode of the number of preset objects of the same type in each image frame of a sliding window (in a sliding window). For example, suppose... Figure 8 In the sliding window, the number of red cars in frame 3 is 3, in frame 4 it is 2, in frame 5 it is 3, and in frame 6 it is 3. Therefore, the mode of the number of red cars in each image frame within the previously obtained sliding window is determined to be 3. The confidence level of identifying the preset object in the image frame is determined; for example, the confidence level of identifying the red car in frame 8 is 0.88. The mean and variance of the confidence levels of the preset object identified in each image frame within the previously obtained sliding window are calculated. Figure 8 In frame 3, the confidence level of the preset object is identified as P1; in frame 4, the confidence level of the preset object is identified as P2; in frame 5, the confidence level of the preset object is identified as P3; and in frame 6, the confidence level of the preset object is identified as P4. The mean and variance of P1-P4 are calculated. Accordingly, after identifying the above identification results, the specific process of determining the detection result of the image frame based on the identification results is as follows: if the detection result satisfies at least one of the following, the image frame is determined to be the second image frame: the distance is greater than a preset distance threshold; the difference between the quantity and the mode is greater than a preset difference threshold; the confidence level is less than a preset confidence level; wherein, the preset confidence level is the difference between twice the mean and the variance, i.e., μ-2σ, where μ is the mean and σ is the variance. If any of the above conditions are not met, the image frame is determined as the second image frame. Specifically, if the distance between the position of the preset object in the current image frame and its position in the previous frame is greater than a preset distance threshold, the difference between the number of preset objects of the same type in the current image frame and the mode of the number of preset objects of the same type in each image frame of the previously obtained sliding window is greater than a preset difference threshold, and the confidence level of identifying the preset object in the image frame is less than at least one of the preset confidence levels, then the image frame is determined as the first image frame.

[0072] S240. During the traversal process, the number of second image frames in the sliding window is counted; wherein, the second image frame is the image frame whose detection result meets the abnormal condition.

[0073] S250. Determine the actual recognition result of the second image frame based on the number of second image frames in the sliding window.

[0074] In this embodiment, the sliding window slides according to a sliding step size, and the image frames entering the sliding window are detected. If the number of first image frames in the sliding window reaches a first preset number, the sliding is completed. If the number of first image frames in the sliding window is less than the first preset number, image frames following the sliding window along the sliding direction are added to the sliding window sequentially, and the image frames entering the sliding window are detected again until the number of first image frames in the sliding window reaches the first preset number. By adaptively adding first image frames during the sliding window's movement, a first preset number of first image frames are maintained in the sliding window, i.e., a first preset number of normal image frames are maintained, so that accurate detection of abnormal frames can be achieved based on a unified standard in subsequent abnormal frame detection processes.

[0075] Figure 11 This is a flowchart illustrating a video recognition method according to another embodiment of this application. This embodiment is an optimization based on the above embodiments; solutions not described in detail in this embodiment are found in the above embodiments. Figure 11 As shown, the method in this embodiment of the application specifically includes the following steps:

[0076] S310. For the target video acquired in real time, the target video is traversed by sliding window starting from the starting frame; wherein the sliding window contains at least a first preset number of first image frames; the first image frames are image frames whose detection results do not meet the abnormal conditions.

[0077] S320. During the traversal process, the number of second image frames in the sliding window is counted; wherein, the second image frame is the image frame whose detection result meets the abnormal condition.

[0078] S330. If the number of second image frames in the sliding window is less than or equal to the second preset number, the actual recognition result of the second image frame is determined during the continued sliding of the sliding window.

[0079] For example, if the number of second image frames in the sliding window is less than or equal to the second preset number, it reflects that the number of second image frames in the sliding window is small. The situation where the second image frame meets the abnormal condition should not be caused by the abnormal behavior of the preset object. Therefore, the actual recognition result of the target video is not determined based on the recognition result of the second image frame at this time, but the actual recognition result of the second image frame is determined during the sliding window process.

[0080] S340. If the number of second image frames in the sliding window is greater than the second preset number, then the recognition result of the second image frame shall be taken as the actual recognition result.

[0081] The second preset number can be adaptively adjusted according to actual conditions. Since the total number of image frames in the sliding window is not fixed, the second preset number can be adaptively adjusted based on the total number of image frames in the sliding window. For example, it can be the product of the total number of image frames in the sliding window and a preset ratio, such as 60%. For example, if the number of second image frames in the sliding window is greater than the second preset number, it reflects that there are a large number of second image frames. The appearance of second image frames is not an accidental event, but rather a result of abnormal behavior of the preset object. Therefore, the recognition result of the second image frames is reliable, and the recognition result of the second image frames is taken as the actual recognition result.

[0082] In this embodiment of the application, determining the actual recognition result of the second image frame during the continued sliding of the sliding window includes:

[0083] When the sliding window is slid until the second image frame moves out of the sliding window, the actual recognition result of the second image frame is determined to be the recognition result of the adjacent first image frame;

[0084] If the second image frame remains within the sliding window during the sliding process, the actual recognition result of the second image frame continues to be determined during the traversal of the sliding window.

[0085] For example, during the sliding window's movement, if the second image frame is moved out of the sliding window, the recognition result of the first image frame adjacent to the second image frame is taken as the actual recognition result of the second image frame, and the second image frame is determined to be misidentified. If the second image frame remains within the sliding window during the sliding window's movement, the recognition result of the second image frame continues to be determined during the sliding window traversal process. That is, the actual recognition result of the second image frame is not determined at first, but the second image frame continues to participate in the statistical process of the second image frame in the current sliding window, which leads to S320 continuing to execute.

[0086] In this embodiment of the application, if the number of second image frames in the sliding window is less than or equal to a second preset number, the actual recognition result of the second image frame is determined as the sliding window continues to slide; if the number of second image frames in the sliding window is greater than the second preset number, the recognition result of the second image frame is taken as the actual recognition result. This solution can adaptively determine whether a second image frame is a misidentified image frame or a genuine case where the recognition result of the second image frame meets an abnormal condition due to the abnormal behavior of the preset object, based on the number of second image frames in the sliding window and the persistence of the behavioral characteristics of the preset object in consecutive image frames. This allows for a more accurate distinction between the two situations and solves the problem of false alarms in abnormal situations.

[0087] This application provides specific implementation schemes for two specific application scenarios, as follows:

[0088] Example Scenario 1: When handling business remotely, customers need to appear in the video for a specified duration.

[0089] Solution: Employ high-fault-tolerant AI real-time video analysis using a sliding window to identify customer presence duration. For example, in remote bank account opening scenarios, the system scans the video stream frame by frame using a sliding window, dynamically adjusting the detection threshold with a high-fault-tolerant algorithm. When a customer's face is not detected in some frames due to a brief moment of looking down to sign or adjusting their posture, the sliding window mechanism combines features from previous frames to make a continuity judgment, avoiding misjudgment of departure due to momentary occlusion. Simultaneously, the system can set a sliding window (e.g., a 30-second image frame sliding window). As long as the customer's cumulative presence time within this sliding window exceeds a first preset time threshold (e.g., 25 seconds) for the first image frame, or is less than or equal to a second preset threshold (e.g., 5 seconds) for the second image frame, it is considered a valid stay. This ensures recognition accuracy while improving fault tolerance in complex scenarios. This method prevents business risks caused by customers leaving midway and reduces erroneous interruptions caused by normal actions, significantly improving the smoothness and security of remote business processing.

[0090] Example Scenario 2: When handling business at a bank's smart counter, it is necessary to ensure that the customer is always within the monitoring screen during the operation to prevent others from operating on their behalf or changing personnel midway.

[0091] Solution: Employing high-fault-tolerant AI real-time video analytics based on a sliding window, the system dynamically monitors the customer's continuous presence at the smart teller. For example, when opening a bank account at a smart teller, the system scans the video stream frame by frame using a sliding window, combining a high-fault-tolerant algorithm to handle partial frame occlusion caused by the customer briefly turning to sign or looking down at documents. When the customer temporarily leaves the center of the screen due to normal actions, the sliding window mechanism continuously tracks the customer's body features (such as clothing, body shape, and movement trajectory) in the preceding and following frames, avoiding misjudgment of departure due to momentary occlusion. Simultaneously, the system can set a sliding window (e.g., a 45-second sliding window), requiring the customer's cumulative time in the frame within this window to exceed a preset threshold (e.g., 40 seconds). Otherwise, an alert is triggered. If the customer's cumulative time in the sliding window corresponding to the first image frame exceeds the first preset time threshold (e.g., 40 seconds), meaning the time corresponding to the second image frame is less than or equal to the second preset threshold (e.g., 5 seconds), it is considered a valid presence. This method can effectively identify whether customers personally handle the entire business process, reduce false alarms caused by normal operations, ensure the security and compliance of business processing, and improve the smoothness of customer experience.

[0092] Figure 12 This is a schematic diagram of a video recognition device provided in an embodiment of this application. This device can execute the video recognition method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. For example... Figure 12 As shown, the device includes:

[0093] The sliding traversal module 410 is used to perform sliding traversal of the target video acquired in real time, starting from the starting frame, based on a sliding window; wherein the sliding window contains at least a first preset number of first image frames; the first image frames are image frames whose detection results do not meet the abnormal conditions;

[0094] The statistics module 420 is used to count the number of second image frames in the sliding window during the traversal process; wherein, the second image frame is the image frame whose detection result meets the abnormal condition;

[0095] The actual recognition result determination module 430 is used to determine the actual recognition result of the second image frame based on the number of second image frames in the sliding window.

[0096] In this embodiment of the application, the sliding traversal module 410 performs a sliding traversal of the target video starting from the starting frame based on a sliding window, including:

[0097] The sliding window is made to slide according to the sliding step size, and the image frames that enter the sliding window this time are detected;

[0098] If the number of first image frames in the sliding window reaches the first preset number, then the sliding is completed;

[0099] If the number of first image frames in the sliding window is less than the first preset number, then image frames located after the sliding window along the sliding direction of the sliding window are added to the sliding window in sequence, and the image frames entering the sliding window this time are detected until the number of first image frames in the sliding window reaches the first preset number.

[0100] In this embodiment of the application, if the number of first image frames in the sliding window reaches a first preset number, the sliding traversal module 410 completes the current sliding, including:

[0101] If the number of the first image frames in the sliding window is equal to the first preset number, then the sliding is completed;

[0102] If the number of first image frames in the sliding window is greater than a first preset number, then the image sequence in the sliding window that includes the first preset number of first image frames starting from the last image frame is retained, and the remaining first image frames and second image frames in the sliding window are moved out of the sliding window.

[0103] In this embodiment of the application, the sliding traversal module 410 detects the image frames that have entered the sliding window, including:

[0104] The preset objects in the image frame are identified, and the identification result is determined.

[0105] The detection result of the image frame is determined based on the recognition result; wherein the detection result includes whether the image frame is a first image frame or the image frame is a second image frame.

[0106] In this embodiment of the application, the sliding traversal module 410 identifies a preset object in the image frame and determines the identification result, including at least one of the following:

[0107] Calculate the distance between the position of the preset object in the image frame and the position of the preset object in the previous frame;

[0108] Identify the number of preset objects of the same type in the image frame, and count the mode of the number of preset objects of the same type in each image frame of the sliding window obtained in the last sliding;

[0109] Determine the confidence level of the preset object identified in the image frame, and calculate the mean and variance of the confidence levels of the preset object identified in each image frame of the sliding window obtained in the previous sliding.

[0110] Accordingly, determining the detection result of the image frame based on the recognition result includes:

[0111] If at least one of the following conditions is met, the detection result of the image frame is determined to be the second image frame; otherwise, the detection result of the image frame is determined to be the first image frame:

[0112] The distance is greater than a preset distance threshold;

[0113] The difference between the quantity and the mode is greater than a preset difference threshold;

[0114] The confidence level is less than a preset confidence level; wherein the preset confidence level is the difference between the mean and twice the variance.

[0115] In this embodiment, the actual recognition result determination module 430 determines the actual recognition result of the second image frame based on the number of second image frames in the sliding window, including:

[0116] If the number of second image frames in the sliding window is less than or equal to the second preset number, the actual recognition result of the second image frame is determined during the continued sliding of the sliding window.

[0117] If the number of second image frames in the sliding window is greater than the second preset number, then the recognition result of the second image frame is taken as the actual recognition result.

[0118] In this embodiment of the application, the actual recognition result determination module 430 determines the actual recognition result of the second image frame during the continued sliding of the sliding window, including:

[0119] When the sliding window is slid until the second image frame moves out of the sliding window, the actual recognition result of the second image frame is determined to be the recognition result of the adjacent first image frame;

[0120] If the second image frame remains within the sliding window during the sliding process, the actual recognition result of the second image frame continues to be determined during the traversal of the sliding window.

[0121] The video recognition device provided in this application embodiment can execute a video recognition method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0122] Figure 13A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0123] like Figure 13 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0124] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of monitors, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless video identification transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0125] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video recognition methods.

[0126] In some embodiments, the video recognition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video recognition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video recognition method by any other suitable means (e.g., by means of firmware).

[0127] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable video recognition device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0129] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0131] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0132] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0133] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video recognition method provided in any embodiment of this application.

[0134] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired information of the technical solution of this application can be achieved, and this is not limited herein.

[0136] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video recognition method, characterized in that, The method includes: For the target video acquired in real time, the target video is traversed by sliding window starting from the starting frame; wherein, the sliding window contains at least a first preset number of first image frames; the first image frames are image frames whose detection results do not meet the abnormal conditions; During the traversal, the number of second image frames in the sliding window is counted; wherein, the second image frame is the image frame whose detection result meets the abnormal condition; The actual recognition result of the second image frame is determined based on the number of second image frames in the sliding window.

2. The method according to claim 1, characterized in that, The target video is traversed using a sliding window, starting from the first frame, including: The sliding window is made to slide according to the sliding step size, and the image frames that enter the sliding window this time are detected; If the number of first image frames in the sliding window reaches the first preset number, then the sliding is completed; If the number of first image frames in the sliding window is less than the first preset number, then image frames located after the sliding window along the sliding direction of the sliding window are added to the sliding window in sequence, and the image frames entering the sliding window this time are detected until the number of first image frames in the sliding window reaches the first preset number.

3. The method according to claim 2, characterized in that, If the number of first image frames in the sliding window reaches a first preset number, then the sliding is completed, including: If the number of the first image frames in the sliding window is equal to the first preset number, then the sliding is completed; If the number of first image frames in the sliding window is greater than a first preset number, then the image sequence in the sliding window that includes the first preset number of first image frames starting from the last image frame is retained, and the remaining first image frames and second image frames in the sliding window are moved out of the sliding window.

4. The method according to claim 2, characterized in that, The detection of image frames entering the sliding window includes: The preset objects in the image frame are identified, and the identification result is determined. The detection result of the image frame is determined based on the recognition result; wherein the detection result includes whether the image frame is a first image frame or the image frame is a second image frame.

5. The method according to claim 4, characterized in that, Identifying a preset object in the image frame and determining the identification result includes at least one of the following: Calculate the distance between the position of the preset object in the image frame and the position of the preset object in the previous frame; Identify the number of preset objects of the same type in the image frame, and count the mode of the number of preset objects of the same type in each image frame of the sliding window obtained in the last sliding; Determine the confidence level of the preset object identified in the image frame, and calculate the mean and variance of the confidence levels of the preset object identified in each image frame of the sliding window obtained in the previous sliding. Accordingly, determining the detection result of the image frame based on the recognition result includes: If at least one of the following conditions is met, the detection result of the image frame is determined to be the second image frame; otherwise, the detection result of the image frame is determined to be the first image frame: The distance is greater than a preset distance threshold; The difference between the quantity and the mode is greater than a preset difference threshold; The confidence level is less than a preset confidence level; wherein the preset confidence level is the difference between the mean and twice the variance.

6. The method according to claim 1, characterized in that, Based on the number of second image frames in the sliding window, the actual recognition result of the second image frame is determined, including: If the number of second image frames in the sliding window is less than or equal to the second preset number, the actual recognition result of the second image frame is determined during the continued sliding of the sliding window. If the number of second image frames in the sliding window is greater than the second preset number, then the recognition result of the second image frame is taken as the actual recognition result.

7. The method according to claim 6, characterized in that, Determining the actual recognition result of the second image frame as the sliding window continues to slide includes: When the sliding window is slid until the second image frame moves out of the sliding window, the actual recognition result of the second image frame is determined to be the recognition result of the adjacent first image frame; If the second image frame remains within the sliding window during the sliding process, the actual recognition result of the second image frame continues to be determined during the traversal of the sliding window.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video recognition method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video recognition method according to any one of claims 1-7.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the video recognition method as described in any one of claims 1-7.

Images