Security video description information generation method and device, electronic equipment and readable medium

Abstract

The embodiment of the invention discloses a security video description information generation method and device, electronic equipment and a readable medium. A specific embodiment of the method comprises the following steps: acquiring a monitoring video of a security scene; performing frame extraction processing on the monitoring video to obtain each frame of monitoring image; performing coding processing on each frame of monitoring image to obtain each visual feature; performing feature compression processing on each visual feature to obtain compressed visual feature information; and generating security and protection video description information based on the compressed visual feature information and a pre-trained security and protection video description information generation model. According to the embodiment, the accuracy and effect of video description in the security scene are improved, and reasoning of the security video description information generation model can be lighter.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, electronic device, and readable medium for generating security video description information. Background Art

[0002] Video surveillance plays a vital role in the security field. It can not only prevent crimes and ensure the safety of security scenes, but also provide strong support for investigations after incidents. The security field mainly analyzes the video streams recorded by cameras. The commonly used method is to generate video descriptions using computer vision algorithms or large models of general image modalities.

[0003] However, when using the above method, the following technical problems often arise: computer vision algorithms have poor understanding of complex scenes, resulting in poor accuracy of the generated video description; general large models are mostly for image modalities, and have poor description effects in the video field; and their application in the video field requires solving the problem of lightweight model reasoning.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a method, device, electronic device and computer-readable medium for generating security video description information to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a method for generating security video description information, the method comprising: acquiring a surveillance video of a security scene; performing frame extraction processing on the surveillance video to obtain surveillance images of each frame; performing encoding processing on the surveillance images of each frame to obtain visual features; performing feature compression processing on the visual features to obtain compressed visual feature information; and generating security video description information based on the compressed visual feature information and a pre-trained security video description information generation model.

[0008] Optionally, the method further includes: sending the security video description information to at least one terminal device associated with the security scene.

[0009] Optionally, the method also includes: in response to detecting feedback information corresponding to the above-mentioned security video description information, generating an updated sample corresponding to the above-mentioned feedback information according to the above-mentioned feedback information; adding the above-mentioned updated sample to an updated sample set; in response to determining that the model update conditions of the above-mentioned security video description information generation model are met, updating the above-mentioned security video description information generation model based on the updated sample set.

[0010] Optionally, the encoding process is performed on each of the surveillance image frames to obtain various visual features, including: for each of the surveillance image frames, performing the following steps: performing block processing on the surveillance image to obtain surveillance image blocks; inputting the surveillance image blocks into the visual feature encoding layer included in the security video description information generation model to obtain visual features corresponding to the surveillance image.

[0011] Optionally, the above-mentioned feature compression processing is performed on each of the above-mentioned visual features to obtain compressed visual feature information, including: performing dimensionality transformation processing on each of the above-mentioned visual features to obtain each transformed visual feature; performing pooling processing on each of the above-mentioned transformed visual features to obtain compressed visual feature information.

[0012] Optionally, the above-mentioned pooling processing of each of the above-mentioned transformed visual features to obtain compressed visual feature information includes: branching processing of each of the above-mentioned transformed visual features to obtain a first transformed visual feature group and a second transformed visual feature group, wherein the number of each first transformed visual features included in the above-mentioned first transformed visual feature group is less than the number of each second transformed visual features included in the above-mentioned second transformed visual feature group; pooling processing of the above-mentioned first transformed visual feature group to obtain a first compressed visual feature group; pooling processing of the above-mentioned second transformed visual feature group to obtain a second compressed visual feature group; flattening processing of the above-mentioned first compressed visual feature group and the above-mentioned second compressed visual feature group respectively to obtain first compressed visual feature information and second compressed visual feature information; splicing the above-mentioned first compressed visual feature information and the above-mentioned second compressed visual feature information into compressed visual feature information.

[0013] Optionally, the above-mentioned feature compression processing is performed on the above-mentioned each visual feature to obtain compressed visual feature information, including: based on the third dimension, the above-mentioned each visual feature is aggregated to obtain each aggregated visual feature, wherein the above-mentioned each visual feature is a tensor corresponding to the first dimension, the second dimension and the third dimension, and the above-mentioned each aggregated visual feature is a tensor corresponding to the above-mentioned first dimension and the above-mentioned second dimension; for each of the above-mentioned aggregated visual features, determining each feature position in the above-mentioned aggregated visual feature that meets a preset filtering condition in the above-mentioned second dimension, wherein each feature position corresponds to a first dimension value and a second dimension value; for each of the above-mentioned each visual feature, based on the determined feature positions, generating a compressed visual feature corresponding to the above-mentioned visual feature; and determining each generated compressed visual feature as compressed visual feature information.

[0014] Optionally, based on the determined feature positions, a compressed visual feature corresponding to the visual feature is generated, including: determining the dimension value of the first dimension corresponding to the visual feature as the current first dimension value; selecting from the determined feature positions each feature position whose corresponding first dimension value is equal to the current first dimension value as each target feature position; determining each second dimension value corresponding to each target feature position as each target second dimension value; determining each visual coding unit in the visual feature whose dimension value corresponding to the second dimension is equal to each target second dimension value as a compressed visual feature, wherein the compressed visual feature corresponds to the visual feature.

[0015] Optionally, the training step of the security video description information generation model includes: obtaining a set of surveillance videos corresponding to at least one security scene as an initial sample set; for each initial sample in the initial sample set, performing the following steps: preprocessing the initial sample to obtain a preprocessed sample; generating annotation information based on the preprocessed sample and a pre-trained annotation model; determining the preprocessed sample and the annotation information as samples; combining the determined samples into a sample set; and training a pre-trained multimodal large model based on the sample set to obtain a security video description information generation model.

[0016] In a second aspect, some embodiments of the present disclosure provide a security video description information generation device, the device comprising: an acquisition unit, configured to acquire a surveillance video of a security scene; a frame extraction unit, configured to perform frame extraction processing on the surveillance video to obtain surveillance images of each frame; an encoding unit, configured to perform encoding processing on the surveillance images of each frame to obtain visual features; a compression unit, configured to perform feature compression processing on the visual features to obtain compressed visual feature information; and a generation unit, configured to generate security video description information based on the compressed visual feature information and a pre-trained security video description information generation model.

[0017] Optionally, the security video description information generating device further includes: a sending unit configured to send the security video description information to at least one terminal device associated with the security scene.

[0018] Optionally, the security video description information generation device further includes: an update sample generation unit, an adding unit and an updating unit. The update sample generation unit is configured to generate an update sample corresponding to the feedback information in response to detecting the feedback information corresponding to the security video description information. The adding unit is configured to add the update sample to the update sample set. The updating unit is configured to perform a model update on the security video description information generation model based on the update sample set in response to determining that the model update condition of the security video description information generation model is met.

[0019] Optionally, the encoding unit is further configured to: for each of the above-mentioned surveillance image frames, perform the following steps: block-process the above-mentioned surveillance image to obtain a surveillance image block; input the above-mentioned surveillance image block into the visual feature encoding layer included in the above-mentioned security video description information generation model to obtain the visual features corresponding to the above-mentioned surveillance image.

[0020] Optionally, the compression unit is further configured to: perform dimensionality transformation processing on each of the above-mentioned visual features to obtain each transformed visual feature; and perform pooling processing on each of the above-mentioned transformed visual features to obtain compressed visual feature information.

[0021] Optionally, the compression unit is further configured to: perform branch processing on the above-mentioned each transformed visual feature to obtain a first transformed visual feature group and a second transformed visual feature group, wherein the number of each first transformed visual feature included in the above-mentioned first transformed visual feature group is less than the number of each second transformed visual feature included in the above-mentioned second transformed visual feature group; perform pooling processing on the above-mentioned first transformed visual feature group to obtain a first compressed visual feature group; perform pooling processing on the above-mentioned second transformed visual feature group to obtain a second compressed visual feature group; perform flattening processing on the above-mentioned first compressed visual feature group and the above-mentioned second compressed visual feature group respectively to obtain first compressed visual feature information and second compressed visual feature information; and splice the above-mentioned first compressed visual feature information and the above-mentioned second compressed visual feature information into compressed visual feature information.

[0022] Optionally, the compression unit is further configured to: based on the third dimension, aggregate the above-mentioned visual features to obtain aggregated visual features, wherein the above-mentioned visual features are tensors corresponding to the first dimension, the second dimension and the third dimension, and the above-mentioned aggregated visual features are tensors corresponding to the above-mentioned first dimension and the above-mentioned second dimension; for each of the above-mentioned aggregated visual features, determine each feature position in the above-mentioned aggregated visual features that meets the preset filtering conditions in the above-mentioned second dimension, wherein each feature position corresponds to a first dimension value and a second dimension value; for each of the above-mentioned visual features, based on the determined feature positions, generate a compressed visual feature corresponding to the above-mentioned visual feature; and determine the generated compressed visual features as compressed visual feature information.

[0023] Optionally, the compression unit is further configured to: determine the dimension value of the first dimension corresponding to the above-mentioned visual feature as the current first dimension value; select from the determined feature positions each feature position whose corresponding first dimension value is equal to the above-mentioned current first dimension value as each target feature position; determine each second dimension value corresponding to the above-mentioned each target feature position as each target second dimension value; determine each visual encoding unit whose dimension value corresponding to the second dimension in the above-mentioned visual feature is equal to the above-mentioned each target second dimension value as a compressed visual feature, wherein the above-mentioned compressed visual feature corresponds to the above-mentioned visual feature.

[0024] Optionally, the training step of the security video description information generation model includes: obtaining a set of surveillance videos corresponding to at least one security scene as an initial sample set; for each initial sample in the initial sample set, performing the following steps: preprocessing the initial sample to obtain a preprocessed sample; generating annotation information based on the preprocessed sample and a pre-trained annotation model; determining the preprocessed sample and the annotation information as samples; combining the determined samples into a sample set; and training a pre-trained multimodal large model based on the sample set to obtain a security video description information generation model.

[0025] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0026] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0027] In a fifth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the method described in any implementation manner of the above-mentioned first aspect when executed by a processor.

[0028] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the security video description information generation method of some embodiments of the present disclosure, the accuracy and effect of the video description in the security scene are improved, and the reasoning of the security video description information generation model can be made more lightweight. Specifically, the reason why the accuracy and effect of the video description in the security scene are poor and the problem of model reasoning lightweight needs to be solved is that the computer vision algorithm has poor understanding ability of complex scenes, resulting in poor accuracy of the generated video description. Most of the general large models are for image modalities, and the description effect in the video field is poor, and the problem of model lightweight needs to be solved when applied in the video field. Based on this, the security video description information generation method of some embodiments of the present disclosure first obtains the surveillance video of the security scene. Then, the above-mentioned surveillance video is subjected to frame extraction processing to obtain each frame of surveillance image. Then, the above-mentioned each frame of surveillance image is encoded to obtain each visual feature. Thus, the visual features of each frame of surveillance image in the overall surveillance video can be obtained. Secondly, the above-mentioned each visual feature is subjected to feature compression processing to obtain compressed visual feature information. Thus, the visual features of each frame of surveillance images can be compressed to reduce the number of visual features of each frame of surveillance images, thereby reducing the amount of information that needs to be processed by the neural network model, and achieving the effect of making the reasoning deployment of the security video description information generation model more lightweight. Finally, based on the above-mentioned compressed visual feature information and the pre-trained security video description information generation model, the security video description information is generated. Thus, the compressed visual features can be used to generate security video description information using a neural network model pre-trained for the security scene video field, thereby improving the model's understanding ability in the security scene, improving the accuracy of generating security video description information and the description effect in the video field. Thus, the accuracy and effect of the video description in the security scene are improved, and the reasoning of the security video description information generation model can be made more lightweight. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0030] Figure 1 is an architectural diagram of an exemplary system in which some embodiments of the present disclosure may be applied;

[0031] Figure 2 is a flow chart of some embodiments of the method for generating security video description information according to the present disclosure;

[0032] Figure 3is a training flow chart of some embodiments of a security video description information generation model according to the present disclosure;

[0033] Figure 4 It is a structural schematic diagram of some embodiments of the security video description information generating device according to the present disclosure;

[0034] Figure 5 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0035] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0036] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0037] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0038] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0039] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0040] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0041] Figure 1 An exemplary system architecture 100 is shown to which a method or device for generating security video description information according to some embodiments of the present disclosure may be applied.

[0042] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, a server 105, and a monitoring device 106. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105, between the server 105 and the monitoring device 106, and between the terminal devices 101, 102, 103 and the monitoring device 106. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0043] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The user can also use the terminal devices 101, 102, 103 to obtain the monitoring video collected by the monitoring device 106 through the network 104. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, security monitoring applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0044] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens and supporting information display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules for providing distributed services, for example, or they can be implemented as a single software or software module. No specific limitation is made here.

[0045] The server 105 may be a server that provides various services, such as a background server that provides support for the information displayed on the terminal devices 101, 102, and 103. The background server may analyze and process the received request and other data, and feed back the processing results (such as security video description information) to the terminal device. The background server may also analyze and process the acquired surveillance video and other data, and feed back the processing results (such as security video description information) to the terminal device.

[0046] It should be noted that the security video description information generation method provided in the embodiment of the present disclosure may be executed by the server 105. Accordingly, the security video description information generation device may be disposed in the server 105.

[0047] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules for providing distributed services, or it can be implemented as a single software or software module. No specific limitation is made here.

[0048] It should be understood that Figure 1 The number of terminal devices, monitoring devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, monitoring devices, networks and servers may be provided according to implementation requirements.

[0049] Continue to refer Figure 2 , shows a process 200 of some embodiments of the method for generating security video description information according to the present disclosure. The method for generating security video description information comprises the following steps:

[0050] Step 201, obtaining surveillance video of the security scene.

[0051] In some embodiments, the execution subject of the security video description information generation method (for example Figure 1 The server shown in the figure) can obtain the monitoring video of the security scene from the monitoring terminal through a wired connection or a wireless connection. Among them, the above-mentioned monitoring terminal can be a monitoring camera in the above-mentioned security scene. The security scene can be a real scene where security needs to be monitored. For example, the security scene can include but is not limited to: parking lots, stations, airports, banks, shopping malls, residential areas, schools, factory workshops, warehouses, and roads. The above-mentioned monitoring video can be a real-time monitoring video of the security scene collected by the above-mentioned monitoring terminal. It should be pointed out that the above-mentioned wireless connection method can include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0052] Step 202: extract frames from the surveillance video to obtain surveillance images.

[0053] In some embodiments, the execution subject may perform frame extraction processing on the surveillance video to obtain each frame of surveillance image. In practice, the execution subject may perform frame extraction processing on the surveillance video at a preset frame rate to obtain each frame of surveillance image. For example, the preset frame rate may be 1 frame per second. Here, the specific setting of the preset frame rate is not limited.

[0054] Step 203: Encode each frame of the monitoring image to obtain various visual features.

[0055] In some embodiments, the execution subject may encode the surveillance images to obtain visual features. In practice, for each of the surveillance images, the execution subject may input the surveillance image into a visual feature encoder (generally a vision transformer, ViT) pre-trained based on a contrastive learning method to obtain initial visual features corresponding to the surveillance image. The contrastive learning method may be SigLIP (Sigmoid Loss for Language Image Pre-Training) or CLIP (Contrastive Language-Image Pre-training). Then, the initial visual features obtained may be input into a projector to obtain visual features. The projector may include, but is not limited to, a linear projector, an MLP, and cross attention. Thus, the initial visual features may be mapped to the text feature space of the security video description information generation model.

[0056] In some optional implementations of some embodiments, for each of the above-mentioned surveillance image frames, the execution subject may perform the following steps:

[0057] The first step is to process the monitoring image into blocks to obtain monitoring image blocks. In practice, the execution entity may divide the monitoring image into monitoring image blocks according to a preset size. The preset size may be determined by the type of visual feature encoder used. For example, when the visual feature encoder is SigLIP's ViT, the preset size of each monitoring image block may be 14*14.

[0058] In the second step, the monitoring image block is input into the visual feature coding layer included in the security video description information generation model to obtain the visual features corresponding to the monitoring image. The visual feature coding layer can be used to extract the visual features of the image. The visual feature coding layer can include a visual feature encoder and a projector. The dimension of the visual features corresponding to the monitoring image can be N*C. N can be the number of divided monitoring image blocks. C can be the feature dimension corresponding to each image block. Thus, the visual features of the monitoring image can be represented in blocks from the dimension of the image block. The dimensions of each visual feature obtained can be T*N*C. T can be the total number of frames of each monitoring image obtained after extracting frames from the monitoring video.

[0059] Step 204: perform feature compression processing on each visual feature to obtain compressed visual feature information.

[0060] In some embodiments, the execution subject may perform feature compression processing on the visual features to obtain compressed visual feature information. In practice, the execution subject may perform feature compression processing on the visual features to obtain compressed visual feature information by principal component analysis or linear discriminant analysis.

[0061] In some optional implementations of some embodiments, the above execution subject may perform the following steps:

[0062] The first step is to perform dimension transformation processing on each of the above-mentioned visual features to obtain each transformed visual feature. In practice, the above-mentioned execution subject can perform dimension transformation processing on each of the above-mentioned visual features according to the preset dimension to obtain each transformed visual feature. Among them, the granularity of the above-mentioned preset dimension is finer than the original dimension of each of the above-mentioned visual features. The overall dimension of each transformed visual feature can be the above-mentioned preset dimension. The above-mentioned each transformed visual feature can be a tensor corresponding to the first dimension, the first transformation dimension, the second transformation dimension and the third dimension. The above-mentioned first dimension can correspond to the total number of frames of the above-mentioned monitoring images. The above-mentioned first transformation dimension can correspond to the height of each frame of the monitoring image after the block is divided. The above-mentioned second transformation dimension can correspond to the width of each frame of the monitoring image after the block is divided. The above-mentioned third dimension can correspond to the feature dimension of each monitoring image block. For example, the above-mentioned preset dimension can be T*H*W*C. H can be the height of the divided monitoring image block. W can be the width of the divided monitoring image block. The dimension transformation processing can be an overall reshape processing of each visual feature.

[0063] The second step is to perform pooling processing on each of the above-mentioned transformed visual features to obtain compressed visual feature information. In practice, the above-mentioned execution subject can perform two-dimensional average pooling processing on each of the above-mentioned transformed visual features to obtain a compressed visual feature set. Among them, kernel=stride=2. Then, the above-mentioned compressed visual feature set can be flattened to obtain compressed visual feature information. Here, the flattening processing can be a processing of converting multi-dimensional data into a one-dimensional array (feature flatten).

[0064] In some optional implementations of some embodiments, the execution subject may perform pooling processing on the above-mentioned transformed visual features through the following steps to obtain compressed visual feature information, including:

[0065] In the first step, each of the above-mentioned transformed visual features is subjected to branch processing to obtain a first transformed visual feature group and a second transformed visual feature group. The number of each first transformed visual feature included in the above-mentioned first transformed visual feature group is less than the number of each second transformed visual feature included in the above-mentioned second transformed visual feature group. In practice, the above-mentioned execution subject may divide a random first number of transformed visual features into the first transformed visual feature group, and divide the remaining second number of transformed visual features into the second transformed visual feature group. The above-mentioned first number is less than the above-mentioned second number.

[0066] In the second step, pooling is performed on the first transformed visual feature group to obtain a first compressed visual feature group.

[0067] The third step is to perform pooling processing on the second transformed visual feature group to obtain a second compressed visual feature group. Here, the pooling processing of the first transformed visual feature group and the second transformed visual feature group can also be a two-dimensional average pooling processing. The kernel and stride used in the pooling processing of the first transformed visual feature group are smaller than the kernel and stride used in the pooling processing of the second transformed visual feature group. Also because the kernel and stride used in the pooling processing of the second transformed visual feature group are larger, the information of interest can be made more general and more inclined to motion information. When the kernel / stride is smaller, the information of interest is more detailed, so it is more inclined to semantic information. As a result, the obtained first compressed visual feature group can pay more attention to semantic information. The obtained second compressed visual feature group can pay more attention to motion information.

[0068] In the fourth step, the first compressed visual feature group and the second compressed visual feature group are flattened to obtain first compressed visual feature information and second compressed visual feature information. Here, the flattening process may be a process of converting multi-dimensional data into a one-dimensional array (feature flatten).

[0069] In the fifth step, the first compressed visual feature information and the second compressed visual feature information are concatenated into compressed visual feature information. Thus, two branches can be used to perform visual feature compression respectively, which is more suitable for surveillance videos with a longer duration. One branch focuses on semantic information, and the other branch focuses more on motion information, so that most of the important information can be further retained to reduce information redundancy and improve the comprehension ability of multimodal large models.

[0070] Optionally, each of the visual features is a tensor corresponding to a first dimension, a second dimension, and a third dimension. The first dimension may correspond to the total number of frames of each surveillance image. The second dimension may correspond to the number of divided surveillance image blocks. The third dimension may correspond to the feature dimension of each surveillance image block.

[0071] In some optional implementations of some embodiments, the execution subject may perform feature compression processing on each of the above-mentioned visual features through the following steps to obtain compressed visual feature information:

[0072] The first step is to summarize the above-mentioned visual features based on the third dimension to obtain each summarized visual feature. Among them, each summarized visual feature is a tensor corresponding to the first dimension and the second dimension. In practice, the execution entity may use a summation function to summarize the above-mentioned visual features in the third dimension to obtain each summarized visual feature. The summation function may be F=sum(V,dim=-1). F may represent each summarized visual feature obtained. V may represent each visual feature. dim=-1 may represent the third dimension.

[0073] In the second step, for each of the above-mentioned summarized visual features, determine each feature position in the above-mentioned summarized visual features that meets the preset filtering condition in the above-mentioned second dimension. Among them, each feature position corresponds to a first dimension value and a second dimension value. The preset filtering condition may be that the feature value corresponding to the feature position in the summarized visual feature is ranked TopK in the above-mentioned second dimension. The feature position may be the dimensional position corresponding to each feature value in the summarized visual feature. The first dimension value may be the dimensional value of the feature position in the first dimension. The second dimension value may be the dimensional value of the feature position in the second dimension. For example, when the summarized visual feature is the summarized visual feature corresponding to the first frame and the feature position is the third position in the summarized visual feature, the first dimension value corresponding to the feature position may be 1, and the corresponding second dimension value may be 3.

[0074] The third step is to generate a compressed visual feature corresponding to each of the above-mentioned visual features based on the determined feature positions. In practice, the execution subject may first determine the dimension value of the first dimension corresponding to the above-mentioned visual feature as the current first dimension value. Then, the feature positions whose corresponding first dimension values ​​are equal to the current first dimension values ​​may be selected from the determined feature positions as target feature positions. Next, the second dimension values ​​corresponding to the target feature positions may be determined as target second dimension values. Finally, the visual coding units whose dimension values ​​corresponding to the second dimension in the above-mentioned visual features are equal to the target second dimension values ​​may be determined as compressed visual features. Among them, the compressed visual feature corresponds to the above-mentioned visual feature. The visual coding unit may be a single feature value, that is, the smallest visual feature coding unit (token embedding).

[0075] In the fourth step, the generated compressed visual features are determined as compressed visual feature information. Thus, the compressed visual features of each frame of the image can be screened by the TopK values ​​of the summarized visual features in the second dimension to achieve the effect of compressing the visual features of each frame of the monitoring image.

[0076] Step 205 , generating security video description information based on the compressed visual feature information and a pre-trained security video description information generation model.

[0077] In some embodiments, the execution subject may generate security video description information based on the compressed visual feature information and the pre-trained security video description information generation model. In practice, the execution subject may input the compressed visual feature information into the pre-trained security video description information generation model to obtain the security video description information. The security video description information generation model may be a pre-trained neural network model with data including but not limited to compressed visual feature information as input data and security video description information as output data. The input data may also include questions. For example, the question may be "Please provide a short text description for the video". The question may also be a preset prompt for the security video description information generation model. The basic neural network model for training the security video description information generation model may be a pre-trained general multimodal large model. Thus, description information of the surveillance video may be generated for the security scene video field based on the general multimodal large model. The security video description information may be information describing security events occurring in the surveillance video. For example, the security video description information may be "Someone is breaking into a parked car to steal something".

[0078] Optionally, the execution subject may also send the security video description information to at least one terminal device associated with the security scene. The at least one terminal device associated with the security scene may be at least one terminal device pre-bound to the security scene. The security scene may be uniquely represented by a scene identifier. The at least one terminal device may include but is not limited to: a mobile phone, a tablet computer, and a desktop computer. Thus, the security video description information may be notified to the user side.

[0079] Optionally, the above execution entity may further perform the following steps:

[0080] The first step is to generate an update sample corresponding to the feedback information according to the feedback information in response to detecting the feedback information corresponding to the security video description information. In practice, the execution subject may determine whether the feedback information satisfies the positive feedback condition. The feedback information may be the user's feedback on the accuracy of the security video description information. The feedback information may represent positive feedback or negative feedback. For example, the feedback information representing positive feedback may be "correct". The feedback information representing negative feedback may be "wrong" or "irrelevant". The positive feedback condition may represent positive feedback for the feedback information. Then, in response to determining that the feedback information satisfies the positive feedback condition, the surveillance video and the security video description information may be determined as update samples. Specifically, the execution subject may combine the surveillance video and the security video description information into an update sample. The surveillance video may be used as sample data, and the security video description information may be used as a sample label.

[0081] In practice, the execution subject may also determine whether the feedback information satisfies the negative feedback condition. The positive feedback condition may be a negative feedback characterization for the feedback information. Then, in response to determining that the feedback information satisfies the negative feedback condition, the surveillance video may be input into a pre-trained annotation model to obtain annotation information. Next, the surveillance video may be input into a pre-trained visual information generation model to obtain visual information. Secondly, the annotation information and the visual information may be determined as fused annotation information. Finally, the surveillance video and the fused annotation information may be determined as update samples. Optionally, the execution subject may also send the surveillance video to an annotation terminal so that the annotation personnel may annotate the security video description information for the surveillance video, and then the surveillance video and the annotated security video description information may be combined into an update sample. It should be noted that the surveillance video may be a surveillance video that has been authorized by the user. Thus, when negative feedback is received, fused annotation information or annotated security video description information may be generated for the surveillance video with negative feedback, so that it may be used to generate update samples for model update.

[0082] The third step is to add the above update samples to the update sample set. The above update sample set may be a set of update samples. In the initial state, the update sample set may be an empty set.

[0083] In the fourth step, in response to determining that the model update condition of the security video description information generation model is met, the security video description information generation model is updated based on the update sample set. The model update condition may be that the number of updated samples included in the update sample set reaches a preset number. The model update condition may also be that the time interval between the current time and the last model update is a preset duration. After the security video description information generation model is trained, the initial training completion time of the model may be recorded, and the model update condition may be that the time interval between the current time and the initial training completion time is a preset duration. In practice, the execution subject may use the monitoring videos included in the update sample set as input and the security video description information included in the update sample set as the expected output to update the security video description information generation model. The model update method may refer to the model training method. Thus, the pre-trained security video description information generation model may be optimized and updated based on the user's feedback on the generated security video description information.

[0084] Optionally, the above execution entity may further perform the following steps:

[0085] The first step is to obtain the ambient audio, vibration signal sequence and temperature information sequence corresponding to the acquisition time of the above-mentioned security scene and surveillance video. Among them, the acquisition time of the above-mentioned ambient audio, vibration signal sequence and temperature information sequence is the same as the acquisition time of the above-mentioned surveillance video. The above-mentioned ambient audio can be collected by a microphone array deployed in the above-mentioned security scene. The above-mentioned vibration signal sequence can be detected by a vibration sensor installed on the ground. The above-mentioned temperature information sequence can be detected by an infrared camera. Each temperature information can be represented in the form of an image. The pixel value in the image can represent the high and low temperature. The ambient audio, vibration signal sequence and temperature information sequence can all be time series data.

[0086] The second step is to perform frame processing on the above-mentioned ambient audio to obtain an ambient audio frame sequence. In practice, the above-mentioned execution subject can divide the above-mentioned ambient audio into individual ambient audio frames according to a preset time interval to obtain an ambient audio frame sequence. For example, the preset time interval can be 20 milliseconds. Here, the specific setting of the preset time interval is not limited.

[0087] The third step is to generate a frequency domain representation of each ambient audio frame in the ambient audio frame sequence to obtain spectrum feature information. In practice, the execution subject may use fast Fourier transform to generate a frequency domain representation of each ambient audio frame in the ambient audio frame sequence to obtain a spectrum graph as spectrum feature information.

[0088] The fourth step is to normalize the above-mentioned spectrum feature information to obtain normalized spectrum feature information, so that the features of the audio mode and the features of other modes can have a similar scale range.

[0089] The fifth step is to filter the vibration signal sequence to obtain a denoised vibration signal sequence. In practice, the execution subject may use a low-pass, high-pass or band-pass filter to filter the vibration signal sequence to obtain a denoised vibration signal sequence.

[0090] The sixth step is to generate the time domain feature information and frequency domain feature information corresponding to the denoised vibration signal sequence. In practice, the execution entity may generate the average amplitude, root mean square (RMS), peak value, maximum value, and minimum value corresponding to the denoised vibration signal sequence as the time domain feature information. The time domain feature information may reflect the intensity of the vibration. The execution entity may obtain the frequency domain feature information by performing Fourier transform (FFT) on the denoised vibration signal sequence. The frequency domain feature information may include the main frequency, harmonic components, and frequency band energy.

[0091] Step 7: Generate vibration characteristic information corresponding to the vibration signal sequence based on the time domain characteristic information and the frequency domain characteristic information. In practice, the execution subject may combine the time domain characteristic information and the frequency domain characteristic information into vibration characteristic information corresponding to the vibration signal sequence.

[0092] In the eighth step, the vibration characteristic information is normalized to obtain normalized vibration characteristic information, so that the vibration mode characteristics and other mode characteristics have a similar scale range.

[0093] In the ninth step, the temperature information sequence is normalized to obtain a normalized temperature sequence, so that the temperature mode features and other mode features have a similar scale range.

[0094] The tenth step is to smooth the normalized temperature information sequence to obtain a smoothed temperature sequence. In practice, the execution subject may use a moving average method to smooth the normalized temperature information sequence to obtain a smoothed temperature sequence.

[0095] In the eleventh step, the smoothed temperature information sequence is spatially mapped to obtain a heat map sequence. In practice, a heat map can be constructed for each smoothed temperature information in the smoothed temperature information sequence. The color of each pixel in the heat map represents a different temperature level.

[0096] Step 12: For each heat map in the above heat map sequence, perform the following steps:

[0097] First, generate global feature information corresponding to the above heat map. In practice, the above execution entity can generate the average temperature, standard deviation, maximum temperature, and minimum temperature of the entire image corresponding to the above heat map as global feature information. Here, the temperature can be a normalized temperature value.

[0098] Second, generate regional characteristic information of the target area corresponding to the above-mentioned heat map. The above-mentioned target area can be a pre-set area of ​​interest. In practice, the above-mentioned execution subject can generate the average temperature, standard deviation, maximum temperature, and minimum temperature of the above-mentioned heat map in the above-mentioned target area as regional characteristic information.

[0099] Third, edge detection is performed on the above heat map to obtain edge information. In practice, the execution subject may use a Canny or Sobel operator to perform edge detection on the above heat map to obtain edge information. The edge information may be represented by various image coordinates.

[0100] Fourth, the thermal map is segmented into regions to obtain segmented thermal images. In practice, the execution subject can divide the thermal map into regional images of different temperature levels by K-means clustering to obtain segmented thermal images.

[0101] Fifth, based on the temperature difference between adjacent pixels in the thermal map, a temperature gradient map is generated. In practice, the execution subject may calculate the temperature difference between adjacent pixels in the thermal map to form a temperature gradient map to emphasize areas with drastic temperature changes.

[0102] Sixth, the global feature information, the regional feature information, the edge information, the segmented thermal images and the temperature gradient map are combined into temperature feature information.

[0103] In the thirteenth step, the combined temperature feature information is determined as a temperature feature information sequence. In this way, rich features can be systematically extracted from the temperature data in the form of images, which is not limited to simple temperature readings, but also includes knowledge about temperature distribution patterns, change trends, etc., providing strong support for intelligent security.

[0104] In some optional implementations of some embodiments, the above-mentioned execution entity may input the above-mentioned compressed visual feature information, the above-mentioned normalized spectrum feature information, the above-mentioned normalized vibration feature information and the above-mentioned temperature feature information sequence into the above-mentioned security video description information generation model to obtain the security video description information.

[0105] The above-mentioned first step to the thirteenth step and their related contents, as an inventive point of an embodiment of the present disclosure, solve the technical problem of "relying only on the visual information provided by the video data to identify security events, the data type is relatively single, and the camera may cause false alarms or missed reports due to factors such as lighting conditions and occlusions, resulting in poor accuracy in security event identification". The factors that lead to poor accuracy in security event identification are often as follows: relying only on the visual information provided by the video data to identify security events, the data type is relatively single, and the camera may cause false alarms or missed reports due to factors such as lighting conditions and occlusions. If the above factors are solved, the effect of improving the accuracy of security event identification can be achieved. In order to achieve this effect, the present disclosure introduces multimodal sensor data, including environmental audio, vibration signals, and temperature information. Visual features combined with multiple sensor data such as audio, vibration, and temperature can verify each other and reduce misjudgments. At the same time, the fusion of multiple sensor data can also enhance the model's environmental perception ability and improve the model's response efficiency. In summary, the introduction of multiple sensor data improves the accuracy of security event identification.

[0106] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the security video description information generation method of some embodiments of the present disclosure, the accuracy and effect of the video description in the security scene are improved, and the reasoning of the security video description information generation model can be made more lightweight. Specifically, the reason why the accuracy and effect of the video description in the security scene are poor and the problem of model reasoning lightweight needs to be solved is that the computer vision algorithm has poor understanding ability of complex scenes, resulting in poor accuracy of the generated video description. Most of the general large models are for image modalities, and the description effect in the video field is poor, and the problem of model lightweight needs to be solved when applied in the video field. Based on this, the security video description information generation method of some embodiments of the present disclosure first obtains the surveillance video of the security scene. Then, the above-mentioned surveillance video is subjected to frame extraction processing to obtain each frame of surveillance image. Then, the above-mentioned each frame of surveillance image is encoded to obtain each visual feature. Thus, the visual features of each frame of surveillance image in the overall surveillance video can be obtained. Secondly, the above-mentioned each visual feature is subjected to feature compression processing to obtain compressed visual feature information. Thus, the visual features of each frame of surveillance images can be compressed to reduce the number of visual features of each frame of surveillance images, thereby reducing the amount of information that needs to be processed by the neural network model, and achieving the effect of making the reasoning deployment of the security video description information generation model more lightweight. Finally, based on the above-mentioned compressed visual feature information and the pre-trained security video description information generation model, the security video description information is generated. Thus, the compressed visual features can be used to generate security video description information using a neural network model pre-trained for the security scene video field, thereby improving the model's understanding ability in the security scene, improving the accuracy of generating security video description information and the description effect in the video field. Thus, the accuracy and effect of the video description in the security scene are improved, and the reasoning of the security video description information generation model can be made more lightweight.

[0107] Further references Figure 3 , which shows the training process 300 of some embodiments of the security video description information generation model. The training process 300 of the security video description information generation model includes the following steps:

[0108] Step 301: Acquire a surveillance video set corresponding to at least one security scene as an initial sample set.

[0109] In some embodiments, the execution subject (e.g. Figure 1 The server shown in the figure can obtain a surveillance video set corresponding to at least one security scene as an initial sample set. Thus, surveillance videos of various security scenes can be obtained for model training.

[0110] Step 302: for each initial sample in the initial sample set, perform the following steps:

[0111] Step 3021, preprocessing the initial sample to obtain a preprocessed sample.

[0112] In some embodiments, the execution subject may pre-process the initial sample to obtain a pre-processed sample. Here, the pre-processing may include but is not limited to: image size adjustment, definition adjustment, and denoising.

[0113] Step 3022, generating annotation information based on the preprocessed samples and the pre-trained annotation model.

[0114] In some embodiments, the execution subject may generate annotation information based on the preprocessed sample and the pretrained annotation model. The pretrained annotation model may include but is not limited to at least one of the following: an open source multimodal large model, a previously trained multimodal large model. The open source large model can identify objects, object attributes, and object motion information from the preprocessed sample. The annotation information may include object name, object attributes, and motion information. The motion information may be a description of the object motion process. For example, the annotation information may be "a masked man is picking a lock." Here, the object name is man. The object attributes include one, masked. The motion information includes picking a lock. In practice, the execution subject may input the preprocessed sample into the open source multimodal large model to obtain the annotation information. Optionally, the execution subject may also input the preprocessed sample into the previously trained multimodal large model to obtain supplementary annotation information. Then, the supplementary annotation information may be integrated into the annotation information to update the annotation information.

[0115] Optionally, the above-mentioned execution entity can also input the above-mentioned preprocessed samples into a pre-trained visual information generation model to obtain visual information. Among them, the above-mentioned visual information generation model can be a computer vision model. For example, the visual information generation model can include but is not limited to: YOLO (can be used to identify objects), RAM++ (can be used as an identification label), Unmasked Teacher: Towards Training-Efficient Video Foundation Models (can be used to identify motion information). The visual information generation model can identify objects, object attributes and object motion information from the above-mentioned preprocessed samples. The above-mentioned visual information may include object names, object attributes and motion information. Then, the above-mentioned visual information and the above-mentioned preprocessed samples can be input into the above-mentioned pre-trained annotation model to obtain annotation information.

[0116] Step 3023: determine the preprocessed sample and the annotation information as a sample.

[0117] In some embodiments, the execution entity may determine the preprocessed sample and the fused annotation information as a sample.

[0118] Step 303: combine the determined samples into a sample set.

[0119] In some embodiments, the execution entity may combine the determined samples into a sample set.

[0120] Step 304: Based on the sample set, the pre-trained multimodal large model is trained to obtain a security video description information generation model.

[0121] In some embodiments, the execution subject may train a pre-trained multimodal large model based on the sample set to obtain a security video description information generation model. The pre-trained multimodal large model may be a general multimodal large model. In practice, the execution subject may train a pre-trained multimodal large model based on the sample set in batches to obtain a security video description information generation model.

[0122] from Figure 3 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 3 The process 300 of the security video description information generation method in some corresponding embodiments embodies the steps of training the security video description information generation model. Therefore, the schemes described in these embodiments can be based on relatively accurate fusion annotation information, and can be trained on the security scene video field on the basis of a multimodal large model, thereby improving the accuracy of the security video description information generated by the security video description information generation model and reducing the difficulty of model construction.

[0123] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a security video description information generation device. These device embodiments are similar to Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0124] like Figure 4As shown, the security video description information generating device 400 of some embodiments includes: an acquisition unit 401, a frame extraction unit 402, an encoding unit 403, a compression unit 404 and a generation unit 405. The acquisition unit 401 is configured to acquire a surveillance video of a security scene; the frame extraction unit 402 is configured to perform frame extraction processing on the surveillance video to obtain each frame of surveillance image; the encoding unit 403 is configured to perform encoding processing on each frame of surveillance image to obtain each visual feature; the compression unit 404 is configured to perform feature compression processing on each visual feature to obtain compressed visual feature information; the generation unit 405 is configured to generate the security video description information based on the compressed visual feature information and a pre-trained security video description information generation model.

[0125] Optionally, the security video description information generating device 400 may further include: a sending unit (not shown in the figure), configured to send the security video description information to at least one terminal device associated with the security scene.

[0126] Optionally, the security video description information generating device 400 may further include: an update sample generating unit, an adding unit and an updating unit (not shown in the figure). The update sample generating unit is configured to generate an update sample corresponding to the feedback information according to the feedback information in response to detecting the feedback information corresponding to the security video description information. The adding unit is configured to add the update sample to the update sample set. The updating unit is configured to perform a model update on the security video description information generating model based on the update sample set in response to determining that the model update condition of the security video description information generating model is met.

[0127] Optionally, the encoding unit 403 can be further configured to: for each of the above-mentioned surveillance image frames, perform the following steps: block-process the above-mentioned surveillance image to obtain a surveillance image block; input the above-mentioned surveillance image block into the visual feature encoding layer included in the above-mentioned security video description information generation model to obtain the visual features corresponding to the above-mentioned surveillance image.

[0128] Optionally, the compression unit 404 may be further configured to: perform dimensional transformation processing on each of the above-mentioned visual features to obtain each transformed visual feature; and perform pooling processing on each of the above-mentioned transformed visual features to obtain compressed visual feature information.

[0129] Optionally, the compression unit 404 can be further configured to: perform branch processing on the above-mentioned each transformed visual feature to obtain a first transformed visual feature group and a second transformed visual feature group, wherein the number of each first transformed visual feature included in the above-mentioned first transformed visual feature group is less than the number of each second transformed visual feature included in the above-mentioned second transformed visual feature group; perform pooling processing on the above-mentioned first transformed visual feature group to obtain a first compressed visual feature group; perform pooling processing on the above-mentioned second transformed visual feature group to obtain a second compressed visual feature group; perform flattening processing on the above-mentioned first compressed visual feature group and the above-mentioned second compressed visual feature group respectively to obtain first compressed visual feature information and second compressed visual feature information; and splice the above-mentioned first compressed visual feature information and the above-mentioned second compressed visual feature information into compressed visual feature information.

[0130] Optionally, the compression unit 404 can be further configured to: based on the third dimension, aggregate the above-mentioned visual features to obtain aggregated visual features, wherein the above-mentioned visual features are tensors corresponding to the first dimension, the second dimension and the third dimension, and the above-mentioned aggregated visual features are tensors corresponding to the above-mentioned first dimension and the above-mentioned second dimension; for each of the above-mentioned aggregated visual features, determine each feature position in the above-mentioned aggregated visual features that meets the preset filtering conditions in the above-mentioned second dimension, wherein each feature position corresponds to a first dimension value and a second dimension value; for each of the above-mentioned visual features, based on the determined feature positions, generate a compressed visual feature corresponding to the above-mentioned visual feature; and determine the generated compressed visual features as compressed visual feature information.

[0131] Optionally, the compression unit 404 can be further configured to: determine the dimension value of the first dimension corresponding to the above-mentioned visual feature as the current first dimension value; select each feature position whose corresponding first dimension value is equal to the above-mentioned current first dimension value from the determined feature positions as each target feature position; determine each second dimension value corresponding to the above-mentioned each target feature position as each target second dimension value; determine each visual encoding unit whose dimension value corresponding to the second dimension in the above-mentioned visual feature is equal to the above-mentioned each target second dimension value as a compressed visual feature, wherein the above-mentioned compressed visual feature corresponds to the above-mentioned visual feature.

[0132] Optionally, the training step of the security video description information generation model includes: obtaining a set of surveillance videos corresponding to at least one security scene as an initial sample set; for each initial sample in the initial sample set, performing the following steps: preprocessing the initial sample to obtain a preprocessed sample; generating annotation information based on the preprocessed sample and a pre-trained annotation model; determining the preprocessed sample and the annotation information as samples; combining the determined samples into a sample set; and training a pre-trained multimodal large model based on the sample set to obtain a security video description information generation model.

[0133] It is understood that the units described in the device 400 are similar to those described in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 400 and the units included therein, and will not be described in detail here.

[0134] Reference below Figure 5 , which shows an electronic device 500 (eg, Figure 1 A schematic diagram of the structure of the server in FIG. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0135] like Figure 5 As shown, the electronic device 500 may include a processing device 501 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0136] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0137] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0138] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0139] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0140] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the surveillance video of the security scene; performs frame extraction processing on the surveillance video to obtain each frame of surveillance image; performs encoding processing on each frame of surveillance image to obtain each visual feature; performs feature compression processing on each visual feature to obtain compressed visual feature information; generates security video description information based on the compressed visual feature information and a pre-trained security video description information generation model.

[0141] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, a frame extraction unit, an encoding unit, a compression unit, and a generation unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring surveillance video of a security scene".

[0144] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0145] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements any of the above-mentioned methods for generating security video description information when executed by a processor.

[0146] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for generating security video description information, comprising: Get surveillance video of security scenes; Performing frame extraction processing on the monitoring video to obtain monitoring images of each frame; Performing encoding processing on each frame of the monitoring image to obtain various visual features; Performing feature compression processing on each of the visual features to obtain compressed visual feature information; The security video description information is generated based on the compressed visual feature information and a pre-trained security video description information generation model.

2. The method according to claim 1, wherein: The method further comprises: The security video description information is sent to at least one terminal device associated with the security scene.

3. The method according to claim 1, wherein: The method further comprises: In response to detecting feedback information corresponding to the security video description information, generating an update sample corresponding to the feedback information according to the feedback information; Adding the updated sample to the updated sample set; In response to determining that a model update condition of the security video description information generation model is met, the security video description information generation model is updated based on an update sample set.

4. The method according to claim 1, wherein: The encoding process is performed on each frame of the monitoring image to obtain various visual features, including: For each of the surveillance image frames, the following steps are performed: Performing block processing on the monitoring image to obtain monitoring image blocks; The monitoring image block is input into the visual feature encoding layer included in the security video description information generation model to obtain the visual features corresponding to the monitoring image.

5. The method according to claim 1, wherein: The performing feature compression processing on each of the visual features to obtain compressed visual feature information includes: Performing dimension transformation processing on each of the visual features to obtain each transformed visual feature; Pooling is performed on each of the transformed visual features to obtain compressed visual feature information.

6. The method according to claim 5, wherein: The performing pooling processing on each of the transformed visual features to obtain compressed visual feature information includes: Performing branch processing on each of the transformed visual features to obtain a first transformed visual feature group and a second transformed visual feature group, wherein the number of each first transformed visual feature included in the first transformed visual feature group is less than the number of each second transformed visual feature included in the second transformed visual feature group; Performing pooling processing on the first transformed visual feature group to obtain a first compressed visual feature group; performing pooling processing on the second transformed visual feature group to obtain a second compressed visual feature group; Flattening the first compressed visual feature group and the second compressed visual feature group respectively to obtain first compressed visual feature information and second compressed visual feature information; The first compressed visual feature information and the second compressed visual feature information are concatenated into compressed visual feature information.

7. The method according to claim 1, wherein: The performing feature compression processing on each of the visual features to obtain compressed visual feature information includes: Based on the third dimension, the visual features are aggregated to obtain aggregated visual features, wherein the visual features are tensors corresponding to the first dimension, the second dimension, and the third dimension, and the aggregated visual features are tensors corresponding to the first dimension and the second dimension; For each of the summarized visual features, determining each feature position in the summarized visual features that satisfies a preset screening condition in the second dimension, wherein each feature position corresponds to a first dimension value and a second dimension value; For each of the visual features, based on the determined feature positions, generating a compressed visual feature corresponding to the visual feature; The generated compressed visual features are determined as compressed visual feature information.

8. The method according to claim 7, wherein: The generating, based on the determined respective feature positions, a compressed visual feature corresponding to the visual feature comprises: Determine the dimension value of the first dimension corresponding to the visual feature as the current first dimension value; Selecting, from the determined feature positions, feature positions whose corresponding first dimension values ​​are equal to the current first dimension value as target feature positions; Determine each second dimension value corresponding to each target feature position as each target second dimension value; Each visual coding unit in the visual feature whose dimension value corresponding to the second dimension is equal to the each target second dimension value is determined as a compressed visual feature, wherein the compressed visual feature corresponds to the visual feature.

9. The method according to claim 1, wherein: The training steps of the security video description information generation model include: Acquire a surveillance video set corresponding to at least one security scene as an initial sample set; For each initial sample in the initial sample set, perform the following steps: Preprocessing the initial sample to obtain a preprocessed sample; Generate annotation information based on the preprocessed samples and the pre-trained annotation model; Determine the preprocessed sample and the labeled information as a sample; Combining the determined samples into a sample set; Based on the sample set, a pre-trained multimodal large model is trained to obtain a security video description information generation model.

10. A security video description information generating device, comprising: An acquisition unit configured to acquire surveillance video of a security scene; A frame extraction unit is configured to perform frame extraction processing on the monitoring video to obtain each frame of monitoring image; An encoding unit is configured to perform encoding processing on each frame of the monitoring image to obtain various visual features; A compression unit, configured to perform feature compression processing on each of the visual features to obtain compressed visual feature information; The generating unit is configured to generate the security video description information based on the compressed visual feature information and a pre-trained security video description information generating model.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images