Video monitoring method and device, electronic equipment and medium

Abstract

The embodiment of the invention discloses a video monitoring method and device, electronic equipment and a storage medium. The method comprises the following steps: receiving a monitoring demand instruction sent by a user terminal in real time; screening at least one key video frame matched with the target attention dimension from a real-time video stream acquired by a camera; inputting the monitoring demand instruction and the at least one key video frame into a pre-trained analysis large model, extracting feature information of the target object under the target attention dimension through the analysis large model, and generating a structured description text containing the feature information; the structured description text and a thumbnail of the at least one key video frame are packaged into a monitoring result package, the monitoring result package is output to a terminal corresponding to the user, and the thumbnail is an image obtained after resolution compression processing is conducted on the key video frame. According to the scheme, the demand pertinence, the result structuring degree and the data security of video monitoring can be improved.

Description

Technical Field

[0001] This application relates to the field of security technology, specifically to a video surveillance method, device, electronic device, and storage medium. Background Technology

[0002] As the core front-end equipment of a security system, cameras typically integrate artificial intelligence algorithms and have functions such as target object behavior recognition and anomaly detection (such as intrusion, loitering, and object leaving). Some devices also support license plate recognition, face comparison, and can be linked to the alarm system to trigger real-time alarms.

[0003] However, existing technologies have several shortcomings: First, after a camera identifies a target object, it usually sends the complete video stream containing the target object to the backend device. Users need to review the video recordings segment by segment to obtain the required information. If users only focus on a specific dimension, they still need to process a large amount of irrelevant video data, which is not targeted enough. Second, existing monitoring results are mostly presented as video or simple identification tags (such as the presence of people or the passage of vehicles), lacking structured descriptions. Users find it difficult to quickly extract key features, resulting in low efficiency in information understanding and retrieval. Third, some monitoring systems do not have fine-grained control over user permissions. Unauthorized users may obtain sensitive monitoring data, and there is a lack of encryption mechanisms during data transmission, posing a risk of information leakage. Summary of the Invention

[0004] This application provides a video surveillance method, electronic device, apparatus, and storage medium, which can improve the demand-specificity of video surveillance, the degree of result structuring, and data security, and solve the problems of insufficient convenience for users to obtain monitoring information and data security risks.

[0005] In a first aspect, embodiments of this application provide a video surveillance method, including: The system receives monitoring request instructions sent by user terminals in real time. The monitoring request instructions include target attention dimensions, which include at least one of the target object's behavior category and environmental association information. From the real-time video stream captured by the camera, at least one key video frame that matches the target focus dimension is selected, and the target object exists in the key video frame; The monitoring requirement instruction and the at least one key video frame are input into a pre-trained analysis model. The analysis model extracts the feature information of the target object under the target attention dimension and generates a structured description text containing the feature information. The structured description text includes a feature label field and a natural language interpretation field. The structured description text and the thumbnail of at least one key video frame are packaged into a monitoring result package and output to the user's terminal. The thumbnail is an image obtained after resolution compression processing of the key video frame.

[0006] Optionally, in some embodiments of this application, the step of filtering at least one key video frame from the real-time video stream captured by the camera that matches the target focus dimension includes: Extract frame features from each original video frame in the real-time video stream, and the frame features correspond to the target attention dimension; Preset feature thresholds for each target focus dimension, and determine the original video frames whose frame features satisfy the feature thresholds as candidate video frames; Redundant frames are removed from the candidate video frames to obtain at least one key video frame.

[0007] Optionally, in some embodiments of this application, the step of extracting frame features of each original video frame in the real-time video stream includes: If the target focus dimension is the behavior category of the target object, then extract the dynamic trajectory features and limb movement features of the target object; If the target focus dimension is environmental association information, then the illumination intensity features and background object state features in the video frame are extracted.

[0008] Optionally, in some embodiments of this application, the large-scale analysis model is a cross-modal large-scale model that integrates image features and textual requirements. The training data of the cross-modal large-scale model includes video frame samples from different scenarios, requirement instruction samples corresponding to the scenarios, and structured description samples labeled with feature information. The cross-modal large-scale model learns the association mapping relationship between video frames and requirement instructions through the training data.

[0009] Optionally, in some embodiments of this application, after generating the structured description text containing the feature information, the method further includes: The risk level of the feature information is assessed according to the preset risk assessment rules; Generate a risk warning label corresponding to the risk level, and add the risk warning label to the monitoring result package.

[0010] Optionally, in some embodiments of this application, when the target object's behavior category is a first category and the duration exceeds a preset duration, or when the target object is in a first state in the environmental association information, it is determined to be of medium risk; when the target object's behavior category is a second category, or when the target object is in a second state in the environmental association information, it is determined to be of high risk.

[0011] Optionally, in some embodiments of this application, it further includes: Verify the user permissions corresponding to the user terminal; If the user is an authorized user, then the step of filtering key video frames is performed; If the user is an unauthorized user, a prompt message is returned to the terminal corresponding to the user, and the current process is terminated.

[0012] Optionally, in some embodiments of this application, the output to the user's corresponding terminal includes: The resolution of the thumbnail is adjusted according to the terminal type of the user's terminal; If the terminal type is the target terminal type, the thumbnail is encrypted based on a preset key, and the encrypted thumbnail is transmitted to the terminal corresponding to the user.

[0013] Secondly, embodiments of this application provide a video surveillance device, comprising: The receiving module is used to receive monitoring request instructions sent by the user terminal in real time. The monitoring request instructions include target attention dimensions, which include at least one of the target object's behavior category and environmental association information. The filtering module is used to filter at least one key video frame from the real-time video stream captured by the camera that matches the target attention dimension, wherein the target object exists in the key video frame; The generation module is used to input the monitoring requirement instruction and the at least one key video frame into a pre-trained analysis model, extract the feature information of the target object under the target attention dimension through the analysis model, and generate a structured description text containing the feature information. The structured description text includes a feature label field and a natural language interpretation field. The output module is used to package the structured description text and the thumbnails of at least one key video frame into a monitoring result package and output it to the terminal corresponding to the user. The thumbnails are images obtained after resolution compression processing of the key video frames.

[0014] Accordingly, this application also provides an electronic device, including a memory, a processor, and a processor program stored in the memory and executable on the processor, wherein the processor executes the program as described in any of the methods above.

[0015] This application also provides a storage medium storing a processor program that, when executed by a processor, implements any of the methods described above.

[0016] This application provides a video surveillance method, apparatus, electronic device, and storage medium. After receiving a monitoring request instruction sent by a user terminal in real time, the monitoring request instruction includes a target attention dimension, which includes at least one of the target object's behavior category and environmental association information. From the real-time video stream captured by the camera, at least one key video frame matching the target attention dimension is selected, and the target object is present in the key video frame. Then, the monitoring request instruction and the at least one key video frame are input into a pre-trained large-scale analysis model. The large-scale analysis model extracts feature information of the target object under the target attention dimension and generates a structured description text containing the feature information. The structured description text includes feature label fields and natural language interpretation fields. Finally, the structured description text and thumbnails of the at least one key video frame are packaged into a monitoring result package and output to the user's corresponding terminal. The thumbnails are images obtained after resolution compression processing of the key video frames. In the video surveillance solution provided in this application, key video frames are selected from the real-time video stream of the camera to avoid transmitting the complete video stream and reduce the user's processing of irrelevant data. Furthermore, by analyzing a large model, structured descriptive text containing feature label fields and natural language interpretation fields is generated to help users quickly extract key features, improve information understanding and retrieval efficiency, and package the structured text and compressed thumbnails for output, balancing information integrity and transmission convenience, and reducing the risk of leakage during data transmission. As a result, the relevance of video surveillance to the user's needs, the degree of structure of the results, and data security can be improved, solving the problems of insufficient convenience for users to obtain monitoring information and data security risks. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the video surveillance method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the interface of the video surveillance method provided in the embodiments of this application. Figure 3 This is a schematic diagram of the structure of the video surveillance device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0020] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0021] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0022] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0023] The following describes in detail the embodiments involved in this application. It should be noted that the order of description of the embodiments in this application is not intended to limit the priority of the embodiments.

[0024] This application provides a video surveillance method, apparatus, storage medium, and smart terminal. Specifically, the video surveillance method of this application can be executed by a smart terminal or a server, wherein the smart terminal can be a terminal. The terminal can be a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other smart terminal. The terminal may also include a client, which can be a media playback client or a real-time video surveillance client, etc.

[0025] This application provides a video surveillance method, which can be executed by an electronic device or a server. This application describes the video surveillance method executed by an electronic device as an example. The electronic device includes a touch screen and a processor. The touch screen is used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. When the user operates the GUI through the touch screen, the GUI can control the local content of the electronic device in response to the received operation commands, or it can control the content on the server side in response to the received operation commands.

[0026] The video surveillance solution provided in this application filters and matches key video frames from the camera's real-time video stream, avoiding the transmission of the complete video stream and reducing the need for users to process irrelevant data. Furthermore, it generates structured descriptive text containing feature label fields and natural language interpretation fields by analyzing a large model, helping users quickly extract key features and improving information understanding and retrieval efficiency. At the same time, it packages and outputs the structured text and compressed thumbnails, balancing information integrity and ease of transmission, and reducing the risk of leakage during data transmission. Thus, it can improve the relevance of video surveillance to user needs, the structure of results, and data security, solving the problems of insufficient convenience for users to obtain monitoring information and data security risks.

[0027] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0028] A video surveillance method includes: receiving a monitoring request command sent by a user terminal in real time; filtering at least one key video frame that matches the target's focus dimension from a real-time video stream captured by a camera; inputting the monitoring request command and the at least one key video frame into a pre-trained analysis model, extracting feature information of the target object under the target's focus dimension through the analysis model, and generating a structured descriptive text containing the feature information; packaging the structured descriptive text and thumbnails of the at least one key video frame into a monitoring result package, and outputting it to the user's corresponding terminal.

[0029] Please see Figure 1 , Figure 1 This application provides a flowchart illustrating the video surveillance method. The specific flow of this video surveillance method is as follows: 101. Receive monitoring request commands sent by user terminals in real time.

[0030] Specifically, users can input monitoring needs through the terminal's monitoring application (such as an APP or webpage), generate monitoring request instructions, and send them to the front-end device or back-end monitoring platform. The monitoring request instructions include target focus dimensions, which can be selected or customized by the user according to actual needs, including at least one of the target object's behavior category and environmental related information.

[0031] Optionally, in some embodiments of this application, in a factory security scenario, a security administrator needs to monitor whether personnel in the workshop are wearing safety helmets as required, so the target focus dimension of the monitoring request instruction can be the behavior category and the helmet wearing status; in a residential security scenario, residents need to check whether there are items piled up under their balconies, so the target focus dimension of the monitoring request instruction can be environmental association information and the status of objects under the balconies; in a school security scenario, teachers need to monitor whether students are running and playing on the playground, so the target focus dimension of the monitoring request instruction can be the behavior category and the running and playing behavior.

[0032] Furthermore, upon receiving a monitoring request instruction, user permissions must be verified first. For example, the front-end device or back-end monitoring platform obtains the user's identity information (such as account ID or biometric data) sent by the user terminal and matches it with the preset authorized user database. If the match is successful, step 102 is executed. If the match fails, a text message indicating insufficient permissions to provide monitoring services is returned to the user terminal, and the illegal access attempt is recorded on the back-end platform, terminating the current process.

[0033] Optionally, in some embodiments of this application, it may further include: Verify the user permissions corresponding to the user terminal; If the user is an authorized user, then proceed with the step of filtering key video frames; If the user is an unauthorized user, a prompt message will be returned to the user's corresponding terminal, and the current process will be terminated.

[0034] 102. From the real-time video stream captured by the camera, filter at least one key video frame that matches the target dimension of interest.

[0035] Among them, the key video frames contain target objects. The target focus dimension refers to the type of information that needs to be monitored in detail, as specified by the user through monitoring request instructions. This includes the target object's behavior category and environmental association information. The target object's behavior category refers to the specific actions or states of the target object (such as people, vehicles, animals, etc.) in the monitoring scene. The environmental association information refers to the background state, environmental parameters, or surrounding objects of the monitoring scene in which the target object is located.

[0036] Key video frames refer to video frames selected from real-time video streams captured by cameras that contain target objects and highly match the dimensions of interest to the target audience, while removing redundancy. They can accurately reflect the monitoring information that users are concerned about and avoid interference from irrelevant video data.

[0037] For the target focus dimension in the monitoring requirement command, the corresponding frame features of each original video frame in the real-time video stream are extracted. For example, if the target focus dimension is a behavior category, the dynamic trajectory features of the target object (such as the displacement speed of a person or the driving direction of a vehicle) and limb movement features (such as the swing amplitude of a person's arm) are extracted through behavior recognition algorithms; if it is environmental related information, the light intensity features (such as brightness value) and background object state features (such as the gap of door and window outlines and the color ratio of equipment indicator lights) are extracted through image segmentation algorithms.

[0038] Then, based on the feature thresholds corresponding to each target focus dimension, the original video frames whose frame features meet the feature thresholds are identified as candidate video frames. Next, an inter-frame difference calculation algorithm is used to determine whether the candidate video frames are redundant frames; if they are redundant frames, only the first frame in a series of redundant frames is retained; if they are not redundant frames, they are retained directly, ultimately yielding at least one key video frame.

[0039] Optionally, in some embodiments of this application, the step of "filtering at least one key video frame that matches the target dimension of interest from the real-time video stream captured by the camera" may specifically include: Extract frame features from each original video frame in a real-time video stream. Preset feature thresholds for each target focus dimension, and determine the original video frames whose frame features meet the feature thresholds as candidate video frames; Redundant frames are removed from the candidate video frames to obtain at least one key video frame.

[0040] Frame features correspond to target attention dimensions. Specifically, for the target attention dimension in the monitoring requirement command, the corresponding frame features of each original video frame in the real-time video stream are extracted. If the target attention dimension is a behavior category, then the dynamic trajectory features and limb movement features of the target object are extracted using behavior recognition algorithms, such as a behavior recognition model based on 3D CNN. For example, when recognizing running behavior, the displacement speed of the person is extracted. If the speed exceeds 1.5m / s, it is determined to be a running-related feature, and if the arm swing angle exceeds 30°, it is also determined to be a running-related feature. If the target attention dimension is environmental information, then environmental parameter features can be extracted using image segmentation and feature extraction algorithms, such as an image segmentation model based on Mask R-CNN. For example, when recognizing the opening and closing status of doors and windows, the contour features of the door and window area are extracted. If the gap between the door and window contours exceeds 5cm, it can be determined to be an open state feature. When recognizing light intensity, the average brightness value of the video frame is extracted. If the brightness value is lower than 50cd / m², it is determined to be a low-light feature.

[0041] Preset feature thresholds for each target focus dimension, and determine original video frames whose frame features meet the thresholds as candidate video frames. Optionally, in some embodiments of this application, for "behavior category - loitering", the preset feature threshold is "displacement speed less than 0.3m / s and dwell time more than 3 minutes". If the displacement speed of a person in an original video frame is 0.2m / s and the person has stayed for 4 minutes, then the frame is determined as a candidate video frame; for "environmental association information - fire smoke", the preset feature threshold is "the proportion of smoke area in the video frame exceeds 10%". If the proportion of smoke area in an original video frame is 15%, then the frame is determined as a candidate video frame.

[0042] The feature threshold can be set by the system default or customized by the user. For example, the monitoring threshold for "equipment operating status" in a factory workshop can be set to "the proportion of red indicator lights exceeds 50%" (red represents a fault), while the threshold for "equipment operating status" in an office building can be set to "the proportion of red indicator lights exceeds 30%", so as to adapt to the monitoring accuracy requirements of different scenarios.

[0043] Using an inter-frame difference calculation algorithm (such as frame similarity calculation based on SSIM), if the feature difference of 5 or more consecutive candidate video frames is less than a preset difference threshold (e.g., the difference is less than 5%), they are determined to be redundant frames, and only the first frame in the consecutive frames is retained as the key video frame; if the feature difference of a candidate video frame exceeds 5%, the frame is retained as the key video frame.

[0044] Optionally, in some embodiments of this application, the step "extracting frame features of each original video frame in the real-time video stream" specifically includes: If the target focus dimension is the behavior category of the target object, then extract the dynamic trajectory features and body movement features of the target object; If the target focus dimension is environmental information, then extract the light intensity features and background object state features from the video frame.

[0045] For example, if a person is continuously loitering in a certain area, and the displacement speed of 10 consecutive candidate video frames is 0.2 m / s and the dwell time is 4 minutes, with a feature difference of 3%, then only the first frame is retained as the key video frame; if the displacement speed of the person in subsequent frames changes to 0.5 m / s, then that frame is retained as the new key video frame.

[0046] By removing redundant frames, the number of candidate video frames can be reduced, significantly decreasing the amount of data required for subsequent large-scale model analysis and improving processing efficiency.

[0047] 103. Input the monitoring requirement command and at least one key video frame into the pre-trained analysis model, extract the feature information of the target object in the target attention dimension through the analysis model, and generate a structured descriptive text containing the feature information.

[0048] Structured descriptive text is text generated by analyzing key video frames that match the target's focus dimension using a pre-trained large-scale analysis model (such as a cross-modal large-scale model). Its core function is to accurately present the feature information of the target object within the target focus dimension, combining machine searchability with human readability. Feature tag fields are the core component of structured descriptive text; they are standardized, formatted combinations of fields, typically using a fixed format such as object type-behavioral state-environmental parameters, with fields separated by specific symbols (such as "-"). This field directly corresponds to the target focus dimension.

[0049] The Natural Language Explanation field is a supplementary component of the structured descriptive text. It provides a simplified and detailed explanation of the feature label field, using natural language that is easily understood by humans. It is primarily used by users to quickly understand the behavioral scenarios and specific characteristics of the target object. This field expands upon the details of the feature label field.

[0050] Optionally, in some embodiments of this application, the large analysis model is a cross-modal large model that integrates image features and textual requirements. The training data of the cross-modal large model includes video frame samples from different scenarios, requirement instruction samples corresponding to the scenarios, and structured description samples labeled with feature information. The cross-modal large model learns the association mapping relationship between video frames and requirement instructions through the training data.

[0051] It should be noted that the large-scale analysis model is a cross-modal large-scale model that integrates image features and textual requirements, and its training process includes: Construct a training dataset: Collect video frame samples from different security scenarios such as parks, campuses, factories, and residential communities, and generate corresponding scenario requirement instruction samples; then, label the feature information of each video frame sample and generate structured description samples.

[0052] Model Training: The cross-modal large model is trained by taking video frame samples and demand instruction samples as input and structured description samples as output. Specifically, the large model first converts video frame samples into image feature vectors and demand instruction samples into text feature vectors. The two types of vectors are fused through an attention mechanism, and then the structured description text is output through a decoder. During training, the cross-entropy loss function can be used. The model converges after multiple iterations (e.g., 100 iterations). This large model can accurately identify the correlation between demand instructions and video frames and avoid outputting irrelevant feature information. For example, when the demand instruction is to detect whether a person is wearing a safety helmet, the large model only extracts the feature of whether the person's head is wearing a safety helmet and does not output irrelevant information such as the color of the person's clothing.

[0053] After the large analysis model performs fusion analysis on key video frames and monitoring requirements, it outputs structured description text, which includes feature label fields and natural language interpretation fields. For the feature label fields, a fixed format of object type-behavioral state-environmental parameters can be adopted. For example, if the key video frame shows a person in a factory workshop not wearing a safety helmet and the ambient temperature is 28℃, then the feature label fields would be person-not wearing a safety helmet-workshop / 28℃.

[0054] Natural language explanation fields provide simplified and detailed descriptions of feature label fields, facilitating quick user understanding. For example, the natural language explanation field for "Personnel - Not Wearing a Safety Helmet - Workshop / 28℃" above is: "In Workshop No. 3 of the factory, a person wearing blue overalls is not wearing a safety helmet, and the current ambient temperature in the workshop is 28℃." Figure 2 As shown, the display interface is divided into a feature label area S1 and a natural language interpretation area S2. Users can quickly switch between viewing the two types of information. It also supports searching by feature label field. For example, entering "not wearing a safety helmet" can quickly filter out all relevant monitoring results. Optionally, in some embodiments of this application, after the step "generating structured description text containing feature information", the following may be further included: The risk level of the feature information is assessed according to the preset risk assessment rules; Generate risk warning labels corresponding to the risk level and add the risk warning labels to the monitoring results package.

[0055] Specifically, key information, including the target object's behavior category, duration of behavior, and environmental context, is extracted from the feature label fields and natural language interpretation fields of the structured descriptive text. This information serves as input data for risk assessment, outputting different risk levels and generating corresponding semantic prompts for each level. For example, if the feature information is "person - climbing over fence - north wall," a high-risk rule is matched, generating a high-risk level and labeling it as requiring immediate action. This label is then packaged with the structured text and thumbnail for output.

[0056] Optionally, in some embodiments of this application, when the target object's behavior category is the first category and the duration exceeds a preset duration, or when the target object is in the first state in the environmental association information, it is determined to be of medium risk; when the target object's behavior category is the second category, or when the target object is in the second state in the environmental association information, it is determined to be of high risk.

[0057] It should be noted that the first category refers to "abnormal but non-urgent" behaviors exhibited by the target, which need to last for a certain period of time to constitute a risk, such as: people loitering, not wearing helmets as required, vehicles temporarily stopping, and animals briefly staying. The second category refers to urgent and high-threat behaviors exhibited by the target, which constitute a high risk as soon as they occur, without waiting for a certain duration, such as people climbing over fences, carrying unidentified large items, violent conflicts, vehicles driving against traffic, crashing into guardrails, and animals attacking humans.

[0058] The first state refers to an abnormal but not high-risk background state in the environment where the target object is located, such as doors and windows being open outside of working hours, and low light in the monitored area. The second state refers to a high-risk and urgent background state in the environment where the target object is located, such as fire smoke, equipment smoke, gas leak alarms, and abnormal noises in the monitored area. The preset duration is a risk triggering duration threshold set for the first category of behavior, which needs to be customized according to the scenario. Optionally, in some embodiments of this application, the preset duration for personnel loitering can be set to 15 minutes, the preset duration for not wearing a safety helmet can be set to 5 minutes, and the preset duration for vehicle parking can be set to 30 minutes.

[0059] Specifically, the behavior categories of the target object are extracted from the structured description text. If the behavior category of the target object belongs to the first category, the specific duration corresponding to the behavior of the target object needs to be extracted. At the same time, the associated state of the environment in which the target object is located is determined.

[0060] Next, the extracted feature information is matched with high-risk rules to determine whether either of the following conditions is met: the target object's behavior category belongs to the second category, or the associated information of the target object's environment belongs to the second state. If condition A or condition B is met, it is directly determined as high-risk, and a high-risk label is generated; If the conditions are not met, it is then determined whether the target object's behavior falls under the first category and the duration of that behavior exceeds a preset duration; or, the associated information of the target object's environment falls under the first state. If any of the medium-risk conditions are met, the object is classified as medium-risk, and a medium-risk label is generated.

[0061] If the conditions are not met, the risk level is determined to be low. That is, if the target object's behavior category does not belong to either the first or second category, and the environmental association information does not belong to either the first or second state, the risk level is determined to be low by default. In this case, no special risk warning label is required, or a normal monitoring status label is generated.

[0062] 104. Package the structured description text and a thumbnail of at least one key video frame into a monitoring result package and output it to the user's corresponding terminal.

[0063] Specifically, key video frames are compressed to generate thumbnails. If the user terminal receiving the results is a mobile terminal such as a mobile phone or tablet, the thumbnail resolution is adjusted to below 720P (e.g., 480P), and the size of a single frame image is controlled within 50KB to adapt to mobile network bandwidth and screen size. If the user terminal is a desktop terminal such as a computer, the thumbnail resolution is adjusted to 1080P, and the size of a single frame image is controlled within 200KB to ensure image clarity.

[0064] The compression process can employ a lossy compression algorithm (such as JPEG), with a compression quality factor set to 0.8, reducing data volume while maintaining image quality. Adding the aforementioned risk warning labels to the monitoring result packet facilitates quick risk priority identification by users, preventing the omission of important security risks. If the monitoring result packet is output to a user terminal, it is sent directly through the communication link between the user terminal and the front-end device and / or back-end platform; if output to a back-end monitoring platform, the monitoring result packet must first be encrypted before transmission via wired or wireless network.

[0065] Optionally, in some embodiments of this application, the step "output to the user's corresponding terminal" may specifically include: The resolution of the thumbnail is adjusted according to the type of the user's terminal. If the terminal type is the target terminal type, the thumbnail is encrypted based on the preset key, and the encrypted thumbnail is transmitted to the user's corresponding terminal.

[0066] The encrypted transmission process includes key generation and updating, data encryption, and decryption and reception, as detailed below: During initial deployment, the front-end camera and the back-end monitoring platform negotiate and generate a preset key using the Diffie-Hellman key exchange algorithm. The preset key is updated every 7 days, for example, at 2 AM every 7 days. During updates, the front-end device and the back-end platform automatically renegotiate and generate a new key, rendering the old key immediately invalid, thus avoiding the risk of cracking due to long-term use of the same key. Then, the monitoring result packets are encrypted using the SM4 symmetric encryption algorithm. This involves converting the monitoring result packets into binary data streams, encrypting the data stream in blocks using the preset key, and generating encrypted data packets. Upon receiving the encrypted data packets, the back-end monitoring platform decrypts them using the currently valid preset key, restoring the original monitoring result packets, and stores them in the database. When a user terminal accesses the back-end platform with authorization, the platform sends the decrypted monitoring result packets to the user terminal, or directly sends encrypted data packets to the user terminal (which the user terminal decrypts using a locally stored key).

[0067] The video surveillance method provided in this application embodiment receives a monitoring request instruction sent by a user terminal in real time. The monitoring request instruction includes a target attention dimension, which includes at least one of the target object's behavior category and environmental association information. From the real-time video stream captured by the camera, at least one key video frame matching the target attention dimension is selected. The target object is present in the key video frame. Then, the monitoring request instruction and at least one key video frame are input into a pre-trained analysis model. The analysis model extracts the feature information of the target object under the target attention dimension and generates a structured description text containing the feature information. The structured description text includes feature label fields and natural language interpretation fields. Finally, the structured description text and the thumbnails of at least one key video frame are packaged into a monitoring result package and output to the user's corresponding terminal. The thumbnail is an image obtained after resolution compression processing of the key video frame. In the video surveillance solution provided in this application, key video frames are selected from the real-time video stream of the camera to avoid transmitting the complete video stream and reduce the user's processing of irrelevant data. Furthermore, by analyzing a large model, structured descriptive text containing feature label fields and natural language interpretation fields is generated to help users quickly extract key features, improve information understanding and retrieval efficiency, and package the structured text and compressed thumbnails for output, balancing information integrity and transmission convenience, and reducing the risk of leakage during data transmission. As a result, the relevance of video surveillance to the user's needs, the degree of structure of the results, and data security can be improved, solving the problems of insufficient convenience for users to obtain monitoring information and data security risks.

[0068] To facilitate better implementation of the video surveillance method of this application, this application also provides a video surveillance device. The meanings of the terms used are the same as in the video surveillance system described above, and specific implementation details can be found in the descriptions of the system embodiments.

[0069] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a video surveillance device provided in an embodiment of this application. Specifically, the video surveillance device may include a receiving module 201, a filtering module 202, a generating module 203, and an output module 204, as follows: The receiving module 201 is used to receive monitoring request instructions sent by the user terminal in real time. The monitoring request instructions include target attention dimensions, which include at least one of the target object's behavior category and environmental association information. The filtering module 202 is used to filter at least one key video frame that matches the target focus dimension from the real-time video stream captured by the camera, wherein the target object exists in the key video frame; The generation module 203 is used to input the monitoring requirement instruction and the at least one key video frame into a pre-trained analysis model, extract the feature information of the target object under the target attention dimension through the analysis model, and generate a structured description text containing the feature information. The structured description text includes a feature label field and a natural language interpretation field. The output module 204 is used to package the structured description text and the thumbnail of at least one key video frame into a monitoring result package and output it to the user's corresponding terminal. The thumbnail is an image obtained after the key video frame has been compressed to a higher resolution.

[0070] Optionally, in some embodiments of this application, the filtering module 202 may specifically be used for: Extract frame features from each original video frame in the real-time video stream, and associate the frame features with the target focus dimension; Preset feature thresholds for each target focus dimension, and determine the original video frames whose frame features satisfy the feature thresholds as candidate video frames; Redundant frames are removed from the candidate video frames to obtain at least one key video frame.

[0071] Optionally, in some embodiments of this application, the filtering module 202 is specifically used for: If the target focus dimension is the behavior category of the target object, then extract the dynamic trajectory features and limb movement features of the target object; If the target focus dimension is environmental association information, then the illumination intensity features and background object state features in the video frame are extracted.

[0072] Optionally, in some embodiments of this application, the generation module 203 may also be used for: The risk level of the feature information is assessed according to the preset risk assessment rules; Generate risk warning labels corresponding to the risk level and add the risk warning labels to the monitoring result package.

[0073] Optionally, in some embodiments of this application, the generation module 203 may also be used for: Verify the user permissions corresponding to the user terminal; If the user is an authorized user, then proceed with the step of filtering key video frames; If the user is an unauthorized user, a prompt message will be returned to the user's corresponding terminal, and the current process will be terminated.

[0074] Optionally, in some embodiments of this application, the output module 204 may specifically be used for: Adjust the thumbnail resolution according to the user's terminal type; If the terminal type is the target terminal type, the thumbnail is encrypted based on the preset key, and the encrypted thumbnail is transmitted to the user's corresponding terminal.

[0075] This application embodiment provides a video surveillance device. After receiving the monitoring request instruction sent by the user terminal in real time, the monitoring request instruction includes a target attention dimension, which includes at least one of the target object's behavior category and environmental association information. The filtering module 202 filters at least one key video frame that matches the target attention dimension from the real-time video stream captured by the camera. The key video frame contains the target object. Then, the generation module 203 inputs the monitoring request instruction and at least one key video frame into a pre-trained analysis model. The analysis model extracts the feature information of the target object under the target attention dimension and generates a structured description text containing the feature information. The structured description text includes feature label fields and natural language interpretation fields. Finally, the output module 204 packages the structured description text and the thumbnail of at least one key video frame into a monitoring result package and outputs it to the user's corresponding terminal. The thumbnail is an image obtained after resolution compression processing of the key video frame. In the video surveillance solution provided in this application, key video frames are selected from the real-time video stream of the camera to avoid transmitting the complete video stream and reduce the user's processing of irrelevant data. Furthermore, by analyzing a large model, structured descriptive text containing feature label fields and natural language interpretation fields is generated to help users quickly extract key features, improve information understanding and retrieval efficiency, and package the structured text and compressed thumbnails for output, balancing information integrity and transmission convenience, and reducing the risk of leakage during data transmission. As a result, the relevance of video surveillance to the user's needs, the degree of structure of the results, and data security can be improved, solving the problems of insufficient convenience for users to obtain monitoring information and data security risks.

[0076] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 4As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more processor-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 301 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 302, and by calling data stored in the memory 302, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless video surveillance. It is understood that the modem processor may also not be integrated into the processor 301.

[0077] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and video surveillance methods by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0078] The electronic device also includes a power supply 303 that supplies power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 303 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0079] The electronic device may also include an input unit 304, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0080] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302 to realize various functions, as follows: The system receives monitoring request commands from user terminals in real time; it filters at least one key video frame from the real-time video stream captured by the camera that matches the target's focus dimension; it inputs the monitoring request command and at least one key video frame into a pre-trained analysis model, extracts the feature information of the target object under the target focus dimension through the analysis model, and generates a structured description text containing the feature information; it packages the structured description text and the thumbnail of at least one key video frame into a monitoring result package and outputs it to the user's corresponding terminal.

[0081] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0082] In this embodiment, after receiving a monitoring request instruction sent by a user terminal in real time, the monitoring request instruction includes a target attention dimension, which includes at least one of the target object's behavior category and environmental association information. From the real-time video stream captured by the camera, at least one key video frame matching the target attention dimension is selected, and the target object is present in the key video frame. Then, the monitoring request instruction and at least one key video frame are input into a pre-trained analysis model. The analysis model extracts the feature information of the target object under the target attention dimension and generates a structured description text containing the feature information. The structured description text includes feature label fields and natural language interpretation fields. Finally, the structured description text and the thumbnails of at least one key video frame are packaged into a monitoring result package and output to the user's corresponding terminal. The thumbnail is an image obtained after resolution compression processing of the key video frame. In the video surveillance solution provided in this application, key video frames are selected from the real-time video stream of the camera to avoid transmitting the complete video stream and reduce the user's processing of irrelevant data. Furthermore, by analyzing a large model, structured descriptive text containing feature label fields and natural language interpretation fields is generated to help users quickly extract key features, improve information understanding and retrieval efficiency, and package the structured text and compressed thumbnails for output, balancing information integrity and transmission convenience, and reducing the risk of leakage during data transmission. As a result, the relevance of video surveillance to the user's needs, the degree of structure of the results, and data security can be improved, solving the problems of insufficient convenience for users to obtain monitoring information and data security risks.

[0083] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a processor-readable storage medium and loaded and executed by a processor.

[0084] Therefore, embodiments of this application provide a storage medium storing multiple instructions that can be loaded by a processor to execute steps in any of the video surveillance methods provided in this application. For example, the instructions can execute the following steps: The system receives monitoring request commands from user terminals in real time; it filters at least one key video frame from the real-time video stream captured by the camera that matches the target's focus dimension; it inputs the monitoring request command and at least one key video frame into a pre-trained analysis model, extracts the feature information of the target object under the target focus dimension through the analysis model, and generates a structured description text containing the feature information; it packages the structured description text and the thumbnail of at least one key video frame into a monitoring result package and outputs it to the user's corresponding terminal.

[0085] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0086] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0087] Since the instructions stored in the storage medium can execute the steps of any of the video surveillance methods provided in the embodiments of this application, the beneficial effects that any of the video surveillance methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0088] The above provides a detailed description of a video surveillance method, apparatus, electronic device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video surveillance method, characterized in that, include: The system receives monitoring request instructions sent by user terminals in real time. The monitoring request instructions include target attention dimensions, which include at least one of the target object's behavior category and environmental association information. From the real-time video stream captured by the camera, at least one key video frame that matches the target focus dimension is selected, and the target object exists in the key video frame; The monitoring requirement instruction and the at least one key video frame are input into a pre-trained analysis model. The analysis model extracts the feature information of the target object under the target attention dimension and generates a structured description text containing the feature information. The structured description text includes a feature label field and a natural language interpretation field. The structured description text and the thumbnail of at least one key video frame are packaged into a monitoring result package and output to the user's terminal. The thumbnail is an image obtained after resolution compression processing of the key video frame.

2. The video surveillance method according to claim 1, characterized in that, The step of filtering at least one key video frame from the real-time video stream captured by the camera that matches the target dimension of interest includes: Extract frame features from each original video frame in the real-time video stream, and the frame features correspond to the target attention dimension; Preset feature thresholds for each target focus dimension, and determine the original video frames whose frame features satisfy the feature thresholds as candidate video frames; Redundant frames are removed from the candidate video frames to obtain at least one key video frame.

3. The video surveillance method according to claim 2, characterized in that, The extraction of frame features from each original video frame in the real-time video stream includes: If the target focus dimension is the behavior category of the target object, then extract the dynamic trajectory features and limb movement features of the target object; If the target focus dimension is environmental association information, then the illumination intensity features and background object state features in the video frame are extracted.

4. The video surveillance method according to claim 1, characterized in that, The analytical big model is a cross-modal big model that integrates image features and textual requirements. The training data of the cross-modal big model includes video frame samples from different scenarios, requirement instruction samples for the corresponding scenarios, and structured description samples labeled with feature information. The cross-modal big model learns the association mapping relationship between video frames and requirement instructions through the training data.

5. The video surveillance method according to claim 1, characterized in that, After generating the structured description text containing the feature information, the method further includes: The risk level of the feature information is assessed according to the preset risk assessment rules; Generate a risk warning label corresponding to the risk level, and add the risk warning label to the monitoring result package.

6. The video surveillance method according to claim 5, characterized in that, When the target object's behavior category is the first category and the duration exceeds the preset duration, or when the target object is in the first state in the environmental association information, it is determined to be of medium risk; when the target object's behavior category is the second category, or when the target object is in the second state in the environmental association information, it is determined to be of high risk.

7. The video surveillance method according to claim 1, characterized in that, Also includes: Verify the user permissions corresponding to the user terminal; If the user is an authorized user, then the step of filtering key video frames is performed; If the user is an unauthorized user, a prompt message is returned to the terminal corresponding to the user, and the current process is terminated.

8. The video surveillance method according to claim 1, characterized in that, The output to the user's corresponding terminal includes: The resolution of the thumbnail is adjusted according to the terminal type of the user's terminal; If the terminal type is the target terminal type, the thumbnail is encrypted based on a preset key, and the encrypted thumbnail is transmitted to the terminal corresponding to the user.

9. A video surveillance device, characterized in that, include: The receiving module is used to receive monitoring request instructions sent by the user terminal in real time. The monitoring request instructions include target attention dimensions, which include at least one of the target object's behavior category and environmental association information. The filtering module is used to filter at least one key video frame from the real-time video stream captured by the camera that matches the target attention dimension, wherein the target object exists in the key video frame; The generation module is used to input the monitoring requirement instruction and the at least one key video frame into a pre-trained analysis model, extract the feature information of the target object under the target attention dimension through the analysis model, and generate a structured description text containing the feature information. The structured description text includes a feature label field and a natural language interpretation field. The output module is used to package the structured description text and the thumbnails of at least one key video frame into a monitoring result package and output it to the terminal corresponding to the user. The thumbnails are images obtained after resolution compression processing of the key video frames.

10. An electronic device, characterized in that, include: A memory, a processor, and a processor program stored in the memory and executable on the processor, wherein the processor executes the program as steps of the video surveillance method as described in any one of claims 1 to 8.

11. A storage medium, characterized in that, The computer processing program is stored and can be loaded by a processor to execute the video surveillance method as described in any one of claims 1 to 8.

Images