Information sending method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses an information sending method and device, electronic equipment and a computer readable medium. A specific embodiment of the method comprises the following steps: acquiring a security video; performing cross frame extraction processing on the security video to generate at least one frame image sequence; generating a video range according to the at least one frame image sequence; performing image sequence segmentation on each frame image sequence to generate segmented image sequences and obtain at least one segmented image sequence; obtaining video voice and a distributed model set; generating object intention information according to the model set, the at least one segmented image sequence and the video voice; in response to determining that the object intention information is in the abnormal intention information table, performing target object labeling on the security and protection sub-video corresponding to the video range to generate a labeled video; and sending the annotated video and the object intention information to an alarm terminal. According to the embodiment of the invention, by generating the object intention information, object abnormity supervision can be accurately and efficiently carried out on the security and protection video.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to an information sending method, device, electronic device, and computer-readable medium. Background Art

[0002] At present, the current security awareness has been widely rooted in people's minds. For object anomaly detection in security videos, the usual approach is to generate object intent information for security videos through a video-based object intent detection model.

[0003] However, when using the above method to generate object intent information, the following technical problems often occur:

[0004] The object intent detection model learns limited video features, resulting in inaccurate object intent information being generated.

[0005] In the process of adopting technical solutions to solve the above technical problem 1, the following technical problem often arises: the training of each model in the model set is also crucial. For the above technical problem, the conventional solution is generally to generate a model set by training each model in the model set individually. However, the above conventional solution still has the following problem 2: the output coordination between the various models in the obtained model set is not accurate enough, resulting in the subsequent output of the object intent information is not accurate enough.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose information sending methods, devices, electronic devices and computer-readable media to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide an information sending method, comprising: acquiring a security video for a target monitoring area; performing cross-frame extraction processing on the security video to generate at least one frame image sequence; generating a video range for a target object based on the at least one frame image sequence; performing image sequence segmentation on each frame image sequence in the at least one frame image sequence according to the video range to generate a segmented image sequence, and obtaining at least one segmented image sequence; acquiring video and voice in the security video within the video range and a distributed model set, wherein the model set comprises: a main node object intent recognition information generation model based on multimodality, and a plurality of sub-node object intent recognition information generation models based on multimodality; generating object intent information for the target object based on the model set, the at least one segmented image sequence and the video and voice; in response to determining that the object intent information is in an abnormal intent information table, annotating the security sub-video corresponding to the video range with the target object to generate an annotated video; and sending the annotated video and the object intent information to an alarm terminal.

[0010] In a second aspect, some embodiments of the present disclosure provide an information sending device, comprising: a first acquisition unit, configured to acquire a security video for a target monitoring area; a cross-frame extraction processing unit, configured to perform cross-frame extraction processing on the security video to generate at least one frame image sequence; a first generation unit, configured to generate a video range for a target object based on the at least one frame image sequence; a segmentation unit, configured to perform image sequence segmentation on each frame image sequence in the at least one frame image sequence according to the video range to generate a segmented image sequence and obtain at least one segmented image sequence; a second acquisition unit, configured to acquire the above-mentioned video range and the above-mentioned image sequence. Video and voice in security video and a distributed model set, wherein the model set includes: a main node object intent recognition information generation model based on multimodality, and a plurality of sub-node object intent recognition information generation models based on multimodality; a second generation unit, configured to generate object intent information for the target object according to the model set, the at least one segmented image sequence and the video and voice; a labeling unit, configured to label the security sub-video corresponding to the video range with the target object in response to determining that the object intent information is in an abnormal intent information table, so as to generate a labeled video; a sending unit, configured to send the labeled video and the object intent information to an alarm terminal.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0013] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the information sending method of some embodiments of the present disclosure, by generating object intent information, object abnormality supervision can be accurately and efficiently performed on security videos. Specifically, the reason for the inaccuracy of the relevant object abnormality supervision is that the video features learned by the object intent detection model are limited, resulting in the inaccuracy of the generated object intent information. Based on this, the information sending method of some embodiments of the present disclosure first obtains the security video for the target monitoring area as a data basis, so as to facilitate the subsequent identification of the object intent information corresponding to the target object. Then, the above-mentioned security video is cross-frame processed to generate at least one frame image sequence, which can obtain more feature-rich at least one frame image sequence with different feature details corresponding to each frame image sequence, and the object intent information can be accurately generated subsequently. Then, according to the above-mentioned at least one frame image sequence, a video range for the target object is generated, so as to facilitate the subsequent generation of at least one segmented image sequence to determine at least one segmented image sequence associated with the target object and having only valid features. Furthermore, according to the above video range, each frame image sequence in the above at least one frame image sequence is segmented to generate a segmented image sequence, and at least one segmented image sequence is obtained, so as to facilitate the subsequent generation of accurate object intent information. Then, the video voice in the above security video within the above video range and the model set based on distribution are obtained, wherein the above model set includes: a main node object intent recognition information generation model based on multimodality, and a plurality of sub-node object intent recognition information generation models based on multimodality. Here, by using the model set based on distribution, the subsequent object intent information can be accurately generated with less calculation amount. In addition, the video voice is a modal data to assist in the subsequent generation of object intent information, which facilitates the subsequent extraction of more feature information about the target object to generate more accurate object intent information. Further, according to the above model set, the above at least one segmented image sequence and the above video voice, the object intent information for the above target object can be accurately generated. Secondly, in response to determining that the above object intent information is in the abnormal intent information table, the target object is annotated for the security sub-video corresponding to the above video range to generate an annotated video, so as to facilitate the subsequent implementation of the corresponding alarm operation for the target object. Finally, the annotated video and the object intent information are sent to the alarm terminal. In summary, by cross-frame processing of the security video and using a distributed model set, the object intent information corresponding to the target object can be accurately and efficiently generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flow chart of some embodiments of the information sending method according to the present disclosure;

[0016] Figure 2 is a schematic diagram of the structure of some embodiments of the information sending device according to the present disclosure;

[0017] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] refer to Figure 1, shows a process 100 of some embodiments of the information sending method according to the present disclosure. The information sending method comprises the following steps:

[0025] Step 101, obtaining security video for a target monitoring area.

[0026] In some embodiments, the execution subject of the above information sending method can obtain the security video for the target monitoring area through a wired connection or a wireless connection. The target monitoring area can be an area for security monitoring. The security video can be a regional shooting video.

[0027] Step 102: cross-frame extraction is performed on the security video to generate at least one frame image sequence.

[0028] In some embodiments, the execution entity may perform cross-frame processing on the security video to generate at least one frame image sequence.

[0029] In some optional implementations of some embodiments, the cross-frame extraction processing of the security video to generate at least one frame image sequence may include the following steps:

[0030] The first step is to obtain a preset frame extraction time sequence. The frame extraction time in the frame extraction time sequence decreases from left to right. For example, the frame extraction time sequence is [10, 9, 7, 5, 3, 2].

[0031] The second step is to divide the security video into video segments according to the maximum frame extraction duration in the frame extraction duration sequence to generate a security sub-video sequence, wherein each video duration corresponding to the security sub-video is the same as the maximum frame extraction duration.

[0032] As an example, the execution entity may use the maximum frame extraction duration as the video division duration to divide the security video to generate a security sub-video sequence.

[0033] Step 3: For the target frame extraction duration in the above frame extraction duration sequence, perform the following frame extraction steps:

[0034] Sub-step 1, determining a previous frame extraction duration corresponding to the target frame extraction duration, wherein the previous frame extraction duration may be a frame extraction duration in a frame extraction duration sequence and located adjacent to the left of the target frame extraction duration.

[0035] Sub-step 2: determining a plurality of frame image sub-sequences corresponding to the previous frame extraction time length as a plurality of first frame image sub-sequences, wherein the plurality of first frame image sub-sequences may be a plurality of image sub-sequences generated based on the frame extraction step.

[0036] Sub-step 3, extracting a target number of frame images from each security sub-video in the above security sub-video sequence to generate a frame image subsequence as a second frame image subsequence, and obtaining multiple second frame image subsequences. The target number is the same as the target frame extraction duration. The second frame image subsequences in the multiple second frame image subsequences and the first frame image subsequences in the multiple first frame image subsequences have a one-to-one time correspondence. The frame image overlap between each frame image included in each second frame image subsequence and each frame image included in the corresponding first frame image subsequence is greater than a preset overlap. For example, the preset overlap can be 30%.

[0037] Sub-step 4, in response to determining that the target frame extraction time is the minimum frame extraction time, determining the plurality of second frame image subsequences as at least one frame image sequence, wherein the minimum frame extraction time may be the minimum frame extraction time in the frame extraction time sequence.

[0038] In the fourth step, in response to determining that the target frame extraction duration is not the minimum frame extraction duration, the next frame extraction duration corresponding to the target frame extraction duration is used as the target frame extraction duration, and the above frame extraction steps are continued.

[0039] Step 103: Generate a video range for the target object according to the at least one frame image sequence.

[0040] In some embodiments, the execution subject may generate a video range for the target object based on the at least one frame image sequence. The video range may be a screen range in the security video that is associated with the target object. The video range may be represented by a video time range.

[0041] In some optional implementations of some embodiments, generating a video range for the target object according to the at least one frame image sequence includes:

[0042] In the first step, for each frame image sequence in the at least one frame image sequence, the following first generation step is performed:

[0043] Sub-step 1: Input each frame image in the above frame image sequence into the target detection model for the above target object to generate target detection information and obtain a target detection information sequence. The target detection information can be detection information of whether there is object information corresponding to the target object in the frame image. In practice, the target detection model can be a YOLO model.

[0044] Sub-step 2, removing frame images corresponding to the target detection information indicating that the target object does not exist from the frame image sequence, to obtain a frame image sequence after removal.

[0045] Sub-step 3, supplementing the above-mentioned removed frame image sequence with the preceding and following continuous frame images to generate a supplemented frame image sequence. The two supplemented image sequences in the supplemented frame image are respectively an image sequence corresponding to the previous position of the removed frame image and having a time-continuous relationship, and an image sequence corresponding to the subsequent position of the removed frame image and having a time-continuous relationship.

[0046] Sub-step 4: determining the initial video range corresponding to the supplemented frame image sequence.

[0047] As an example, the execution subject may determine the video time range corresponding to the supplemented frame image sequence as the initial video range.

[0048] In the second step, a union process is performed on the obtained at least one initial video range to generate a union video range as the above video range. The union process may be a video time range union process.

[0049] Step 104 , performing image sequence segmentation on each frame image sequence in the at least one frame image sequence according to the video range to generate a segmented image sequence, thereby obtaining at least one segmented image sequence.

[0050] In some embodiments, the execution subject may perform image sequence segmentation on each frame image sequence in the at least one frame image sequence according to the video range to generate a segmented image sequence, thereby obtaining at least one segmented image sequence, wherein the video time range corresponding to the segmented image sequence is the same as the video time range corresponding to the video range.

[0051] As an example, the execution subject may perform image sequence segmentation on each frame image sequence in the at least one frame image sequence according to a video time range corresponding to the video range to generate a segmented image sequence, thereby obtaining at least one segmented image sequence.

[0052] Step 105, obtaining the video and voice in the security video within the video range and a distributed model set.

[0053] In some embodiments, the execution subject may obtain the video voice in the security video within the video range and the distributed model set. The video voice may be the voice occurring in the security video within the video range. The model set includes: a main node object intention recognition information generation model based on multimodality, and a plurality of sub-node object intention recognition information generation models based on multimodality. The main node object intention recognition information generation model may be a main neural network model for generating object intention recognition information based on multimodal data as input. The plurality of sub-node object intention recognition information generation models based on multimodality may be a plurality of sub-neural network models for generating object intention recognition information based on multimodal data as input. The accuracy of the generated object intention recognition information corresponding to the main node object intention recognition information generation model is higher than the accuracy of the generated object intention recognition information corresponding to the sub-node object intention recognition information generation model. The computational amount of the generated object intention recognition information corresponding to the main node object intention recognition information generation model is much higher than the computational amount of the generated object intention recognition information corresponding to the sub-node object intention recognition information generation model. In practice, the main node object intention recognition information generation model may be a classification model that determines the actual intention recognition information from a plurality of intention recognition information based on an input data set. In practice, the main node object intent recognition information generation model can be a Transformer model.

[0054] Step 106: Generate object intent information for the target object based on the model set, the at least one segmented image sequence and the video speech.

[0055] In some embodiments, the execution subject may generate object intention information for the target object based on the model set, the at least one segmented image sequence, and the video voice. The object intention information may be intention information of the behavior intention of the target object in the security video. For example, the object intention information may be intention information of the target object's illegal intrusion.

[0056] In some optional implementations of some embodiments, the generating of the object intent information for the target object according to the model set, the at least one segmented image sequence and the video speech may include the following steps:

[0057] In the first step, for each segmented image sequence in the at least one segmented image sequence, the following second generating step is performed:

[0058] Sub-step 1: Generate an image feature information sequence for the segmented image sequence.

[0059] As an example, the execution subject may input each segmented image in the segmented image sequence into a feature extraction model to generate image feature information and obtain an image feature information sequence. The image feature information may represent the image feature semantic content corresponding to the segmented image. The feature extraction model may be a multi-layer series convolutional neural network model.

[0060] Sub-step 2, according to the image set corresponding to each image feature information in the above image feature information sequence, the above video speech is divided into speech to generate a sub-speech sequence. There is a time correspondence between the sub-speech in the sub-speech sequence and the image feature information in the image feature information sequence. That is, there is a one-to-one correspondence between the speech content and duration corresponding to the sub-speech and the content and duration corresponding to the image feature information.

[0061] As an example, for each image feature information in the above image feature information sequence, first, determine the segmented image subsequence corresponding to the image feature information. Then, determine the image time period corresponding to the segmented image subsequence. Finally, cut out the speech corresponding to the above image time period from the above video speech as the sub-speech corresponding to the above image feature information.

[0062] Sub-step 3: for each sub-speech in the above sub-speech sequence, perform the following third generation step:

[0063] The first sub-step is to perform audio preprocessing on the sub-speech to generate processed audio, wherein the audio preprocessing may be audio noise reduction processing.

[0064] The second sub-step is to perform fast Fourier transform on the processed audio to generate a frequency domain signal.

[0065] The third sub-step is to generate a processed frequency domain signal corresponding to the above frequency domain signal using a Mel filter bank.

[0066] The fourth sub-step is to perform logarithmic processing on the above processed frequency domain signal to generate a linear enhanced signal.

[0067] The fifth sub-step is to perform discrete cosine transformation on the linear enhanced signal to generate transformed audio feature information;

[0068] Sub-step 4, according to the number of segmented images corresponding to the above-mentioned segmented image sequence, select the corresponding sub-node object intention recognition information generation model from the above-mentioned multiple sub-node object intention recognition information generation models as the target sub-node object intention recognition information generation model, wherein the output weight ratio corresponding to the above-mentioned target sub-node object intention recognition information generation model corresponds to the above-mentioned number of segmented images. The number of segmented images can be the number of images of each segmented image included in the segmented image sequence. Each sub-node object intention recognition information generation model in the multiple sub-node object intention recognition information generation models has a corresponding output weight ratio. Specifically, depending on the complexity of the model structure of the corresponding sub-node object intention recognition information generation model (that is, the number of units of the network units included), the corresponding output weight ratio is also different. The higher the complexity of the model structure, the higher the corresponding output weight ratio, and the more accurate and important it is to characterize the output of the corresponding model.

[0069] Sub-step 5, based on the above image feature information sequence and the obtained changed audio feature information sequence, using the above target sub-node object intention recognition information generation model, generates initial object intention information. The initial object intention information can represent the preliminary judgment intention information of the target object in the security video.

[0070] As an example, the execution entity may directly input the image feature information sequence and the obtained changed audio feature information sequence into the target sub-node object intent recognition information generation model to generate initial object intent information.

[0071] The second step is to generate the above-mentioned object intent information based on the at least one initial object intent information obtained.

[0072] As an example, the above-mentioned execution entity can summarize the intent information of at least one initial object intent information to generate summarized intent information as the object intent information.

[0073] In some optional implementations of some embodiments, the above-mentioned target sub-node object intention recognition information generation model includes: multiple feature extraction units, multiple feature fusion units and a recognition information generation layer. Among them, the feature extraction unit can be a neural network unit for feature depth extraction. The feature fusion unit can be a unit for multimodal feature information fusion. The recognition information generation layer can be a network layer that generates recognition information. In practice, the feature fusion unit can be an attention mechanism model. The recognition information generation layer can be a fully connected layer with multiple layers connected in series.

[0074] It should be noted that, according to the different numbers of feature extraction units and feature fusion units included in the object intent recognition information generation model of each sub-node, the corresponding output weight ratio is set for the corresponding model.

[0075] Optionally, the generating of initial object intent information based on the image feature information sequence and the obtained changed audio feature information sequence using the target sub-node object intent recognition information generation model may include the following steps:

[0076] The first step is to determine the time step sequence corresponding to the above image feature information sequence, wherein the image feature information in the image feature information sequence and the time step in the time step sequence have a time corresponding relationship.

[0077] In the second step, the target time step is selected from the time step sequence.

[0078] As an example, the execution subject may randomly select a target time step from the time step sequence.

[0079] In the third step, for the target time step, the following third generation step is performed:

[0080] Sub-step 1, obtaining image feature information that has a temporal correspondence with the target time step from the above-mentioned image feature information sequence as the target image feature information, and obtaining changed audio feature information that has a temporal correspondence with the target time step from the above-mentioned changed audio feature information sequence as the target changed audio feature information.

[0081] Sub-step 2, determining the feature extraction unit corresponding to the target time step as the target feature extraction unit, and determining the feature fusion unit corresponding to the target time step as the target feature fusion unit.

[0082] Sub-step 3, determining the previous time step corresponding to the target time step, wherein the previous time step may be the time step in the time step sequence that is located at the left adjacent position of the target time step.

[0083] Sub-step 4, determining the fusion feature information corresponding to the previous time step as the target fusion feature information. The fusion feature information may be multi-modal feature fusion information representing each input data before the previous time step.

[0084] Sub-step 5, inputting the audio feature information after the target change into the audio feature weight information generation model included in the target feature extraction unit to generate audio feature weight information. The audio feature weight information generation model may be a neural network model for generating audio feature weight information. The audio feature weight information may characterize the importance and richness of each audio feature included in the audio feature information after the target change. The higher the corresponding numerical value of the audio feature weight information, the higher the importance and richness of the corresponding audio features included. In practice, the audio feature weight information generation model may be a multi-layer convolutional neural network model.

[0085] Sub-step 6: multiply the audio feature weight information and the target changed audio feature information to generate multiplied audio feature information.

[0086] Sub-step 7, inputting the multiplied audio feature information and the target image feature information into the attention mechanism model included in the target feature extraction unit to generate first attention weight information for the multiplied audio feature information and second attention weight information for the target image feature information. The first attention weight information can represent the importance of the multiplied audio feature information. The second attention weight information can represent the importance of the target image feature information.

[0087] Sub-step 8, inputting the first attention weight information, the second attention weight information, the multiplied audio feature information and the target image feature information into the target feature fusion unit to generate fused feature information.

[0088] Sub-step 9, performing feature information splicing on the above fused feature information and the above target fused feature information to generate spliced ​​feature information.

[0089] Sub-step 10, in response to determining that the target time step is the latest time step in the time step sequence, inputting the splicing feature information corresponding to the latest time step into the recognition information generation layer to generate initial object intent information.

[0090] In the fourth step, in response to determining that the target time step is not the latest time step in the time step sequence, the next time step corresponding to the target time step is used as the target time step, and the third generation step is continued.

[0091] Considering the problems of the above conventional solutions, the output coordination between the models in the model set obtained in the face of the above technical problem 2 is not accurate enough, resulting in the subsequent output of the object intent information is not accurate enough. Combined with the advantages / technical status, it can be decided to adopt the following solution.

[0092] In some optional implementations of some embodiments, the distributed-based model set is trained by the following steps:

[0093] The first step is to obtain a training data set, where the training data includes: target videos and intent recognition labels.

[0094] The second step is to select target training data from the above training data set.

[0095] In the third step, for the target training data, perform the following training steps:

[0096] Sub-step 1: determine the target video and intent recognition label corresponding to the above target training data.

[0097] Sub-step 2: generating at least one target segmented image sequence and target video speech for the target video.

[0098] Sub-step 3, based on the above-mentioned at least one target segmented image sequence and the above-mentioned target video speech, generate the target initial object intent information corresponding to each initial sub-node object intent recognition information generation model in the above-mentioned multiple initial sub-node object intent recognition information generation models, and obtain at least one target initial object intent information.

[0099] Sub-step 4, using the initial main node object intent recognition information generation model, generates target candidate object intent information for the above-mentioned at least one target initial object intent information.

[0100] Sub-step 5: determining the difference information between the target candidate object intention information and the intention recognition label.

[0101] Sub-step 6: Generate first loss information for the difference information.

[0102] Sub-step 7, determining an initial sub-node object intention recognition information generation model set whose corresponding output is different from the above-mentioned intention recognition label.

[0103] Sub-step 8, for each initial sub-node object intent recognition information generation model in the initial sub-node object intent recognition information generation model set, perform the following information generation steps:

[0104] The first sub-step is to determine the target initial object intent information corresponding to the above-mentioned initial sub-node object intent recognition information generation model as the first initial object intent information.

[0105] The second sub-step is to determine the second loss information between the first initial object intention information and the intention recognition label.

[0106] As an example, the execution entity may utilize a cross entropy loss function to determine second loss information between the first initial object intent information and the intent recognition label.

[0107] Sub-step 9, based on the obtained second loss information set, the model parameters of each model in the above-mentioned initial sub-node object intent recognition information generation model set are updated to generate a sub-node object intent recognition information generation model set.

[0108] Sub-step 10, in response to determining that the above-mentioned first loss information is less than the first predetermined value, the obtained sub-node object intention recognition information generation model set, the remaining sub-node object intention recognition information generation model set and the initial main node object intention recognition information generation model are combined to generate a model set. Among them, the remaining sub-node object intention recognition information generation model set can be a model set after removing the initial sub-node object intention recognition information generation model set from multiple initial sub-node object intention recognition information generation models.

[0109] The fourth step, in response to determining that the above-mentioned first loss information is greater than or equal to the above-mentioned first predetermined value, updates the model parameters of the initial main node object intention recognition information generation model according to the above-mentioned first loss information to obtain an updated main node object intention recognition information generation model.

[0110] The fifth step is to remove the target training data from the above training data set to obtain the removed training data set.

[0111] Step 6: Reselect training data from the above removed training data set as candidate training data.

[0112] In the seventh step, the updated main node object intent recognition information generation model, the sub-node object intent recognition information generation model set and the remaining sub-node object intent recognition information generation model sets are combined to generate an initial combined model set.

[0113] In the eighth step, the initial combined model set is used as the initial model set, and the candidate training data is used as the target training data, and the above training steps are continued. The initial model set includes: multiple initial sub-node object intent recognition information generation models and an initial main node object intent recognition information generation model.

[0114] The above-mentioned content of "In some optional implementation methods of some embodiments", as an inventive point of the present disclosure, solves the technical problem mentioned in the background technology that "the output coordination between the various models in the obtained model set is not accurate enough, resulting in the subsequent output of the object intent information is not accurate enough." Based on this, the present disclosure trains the models between the main node object intent recognition information generation model and the multiple sub-node object intent recognition information generation models in sequence, and under the output restriction of the corresponding model of the main node, also updates the models of the multiple sub-node object intent recognition information generation models more accurately, so as to obtain more accurate output of object intent information.

[0115] In some optional implementations of some embodiments, the generating of the object intent information according to the obtained at least one initial object intent information may include the following steps:

[0116] The first step is to determine at least one output weight ratio corresponding to the at least one initial object intention information.

[0117] The second step is to classify each of the initial object intent information in the at least one initial object intent information to generate a set of candidate object intent information. Each candidate object intent information has a corresponding object intent category. The object intent category can be the category to which the object intent belongs.

[0118] The third step is to determine the intention information of each candidate object in the candidate object intention information set, and perform the following fourth generation step:

[0119] Sub-step 1, determining the output weight ratio set corresponding to the above candidate object intention information.

[0120] As an example, first, the execution subject may determine the sub-node object intention recognition information generation model corresponding to the candidate object intention information as the target sub-node object intention recognition information generation model. Then, the output weight ratio corresponding to the target sub-node object intention recognition information generation model is determined.

[0121] Sub-step 2, summing up the output weight proportions in the above output weight proportion set to generate the total output weight proportion.

[0122] The fourth step is to determine whether there is candidate object intention information in the above candidate object intention information set whose corresponding output weight ratio is greater than the first weight ratio. The first weight ratio can be a preset ratio value. For example, the first weight ratio can be 0.8.

[0123] In step 5, in response to the determination of existence, the candidate object intention information whose corresponding output weight ratio is greater than the first weight ratio is determined as the object intention information

[0124] Optionally, the generating the object intent information according to the obtained at least one initial object intent information further includes:

[0125] In the first step, in response to determining that it does not exist, at least one candidate object intention information whose total corresponding output weight ratio is greater than the second weight ratio is determined from the above candidate object intention information set as at least one target candidate object intention information. Wherein, the above second weight ratio is greater than the above first weight ratio.

[0126] In the second step, the main node object intent recognition information generation model is used to select the above object intent information from the above at least one target candidate object intent information.

[0127] As an example, first, the execution subject may perform de-duplication fusion processing on at least one frame image sequence to generate a frame image set. Then, the frame image set and the at least one target candidate object intention information are input into the master node object intention recognition information generation model to generate object intention information.

[0128] In the third step, at least one sub-node object intent recognition information generation model whose corresponding output intent information is not the above-mentioned object intent information is determined from the above-mentioned multiple sub-node object intent recognition information generation models.

[0129] The fourth step is to perform model retraining for the at least one sub-node object intent recognition information generation model.

[0130] Step 107, in response to determining that the object intention information is in the abnormal intention information table, the security sub-video corresponding to the video range is annotated with a target object to generate an annotated video.

[0131] In some embodiments, in response to determining that the object intention information is in the abnormal intention information table, the execution subject may annotate the security sub-video corresponding to the video range with the target object to generate an annotated video. The abnormal intention information table may be a table including various abnormal intention information for objects. The abnormal intention information represents the intention information that there is a security risk.

[0132] As an example, the execution subject may use a bounding box labeling model to label the target object in the security sub-video corresponding to the video range to generate a labeled video. The bounding box labeling model may be a sub-model in the target detection model.

[0133] Step 108: Send the annotated video and the object intention information to an alarm terminal.

[0134] In some embodiments, the execution subject may send the annotated video and the object intention information to an alarm terminal, wherein the alarm terminal may be a terminal for performing alarm processing for security risks.

[0135] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the information sending method of some embodiments of the present disclosure, by generating object intent information, object abnormality supervision can be accurately and efficiently performed on security videos. Specifically, the reason for the inaccuracy of the relevant object abnormality supervision is that the video features learned by the object intent detection model are limited, resulting in the inaccuracy of the generated object intent information. Based on this, the information sending method of some embodiments of the present disclosure first obtains the security video for the target monitoring area as a data basis, so as to facilitate the subsequent identification of the object intent information corresponding to the target object. Then, the above-mentioned security video is cross-frame processed to generate at least one frame image sequence, which can obtain more feature-rich at least one frame image sequence with different feature details corresponding to each frame image sequence, and the object intent information can be accurately generated subsequently. Then, according to the above-mentioned at least one frame image sequence, a video range for the target object is generated, so as to facilitate the subsequent generation of at least one segmented image sequence to determine at least one segmented image sequence associated with the target object and having only valid features. Furthermore, according to the above video range, each frame image sequence in the above at least one frame image sequence is segmented to generate a segmented image sequence, and at least one segmented image sequence is obtained, so as to facilitate the subsequent generation of accurate object intent information. Then, the video voice in the above security video within the above video range and the model set based on distribution are obtained, wherein the above model set includes: a main node object intent recognition information generation model based on multimodality, and a plurality of sub-node object intent recognition information generation models based on multimodality. Here, by using the model set based on distribution, the subsequent object intent information can be accurately generated with less calculation amount. In addition, the video voice is a modal data to assist in the subsequent generation of object intent information, which facilitates the subsequent extraction of more feature information about the target object to generate more accurate object intent information. Further, according to the above model set, the above at least one segmented image sequence and the above video voice, the object intent information for the above target object can be accurately generated. Secondly, in response to determining that the above object intent information is in the abnormal intent information table, the target object is annotated for the security sub-video corresponding to the above video range to generate an annotated video, so as to facilitate the subsequent implementation of the corresponding alarm operation for the target object. Finally, the annotated video and the object intent information are sent to the alarm terminal. In summary, by cross-frame processing of the security video and using a distributed model set, the object intent information corresponding to the target object can be accurately and efficiently generated.

[0136] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an information sending device, and these device embodiments are Figure 1Corresponding to the method embodiments shown, the information sending device can be specifically applied to various electronic devices.

[0137] like Figure 2 As shown, an information sending device 200 includes: a first acquisition unit 201, a cross frame extraction processing unit 202, a first generation unit 203, a segmentation unit 204, a second acquisition unit 205, a second generation unit 206, a labeling unit 207 and a sending unit 208. The first acquisition unit 201 is configured to acquire a security video for a target monitoring area; the cross frame extraction processing unit 202 is configured to perform cross frame extraction processing on the security video to generate at least one frame image sequence; the first generation unit 203 is configured to generate a video range for a target object according to the at least one frame image sequence; the segmentation unit 204 is configured to perform image sequence segmentation on each frame image sequence in the at least one frame image sequence according to the video range to generate a segmented image sequence and obtain at least one segmented image sequence; the second acquisition unit 205 is configured to acquire video and voice in the security video within the video range. and a distributed-based model set, wherein the model set includes: a main node object intent recognition information generation model based on multimodality, and a plurality of sub-node object intent recognition information generation models based on multimodality; a second generation unit 206 is configured to generate object intent information for the target object according to the model set, the at least one segmented image sequence and the video voice; a labeling unit 207 is configured to label the security sub-video corresponding to the video range with the target object in response to determining that the object intent information is in the abnormal intent information table, so as to generate a labeled video; a sending unit 208 is configured to send the labeled video and the object intent information to the alarm terminal.

[0138] It is understandable that the units described in the information sending device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the information sending device 200 and the units included therein, and will not be described in detail here.

[0139] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0140] like Figure 3As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0141] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0142] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0143] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0144] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0145] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains security video for the target monitoring area; performs cross-frame extraction processing on the security video to generate at least one frame image sequence; generates a video range for the target object based on the at least one frame image sequence; performs image sequence segmentation on each frame image sequence in the at least one frame image sequence according to the video range to generate a segmented image sequence, and obtains at least one segmented image sequence; obtains video and voice in the security video within the video range and a distributed model set, wherein the model set includes: a main node object intent recognition information generation model based on multimodal, and a plurality of sub-node object intent recognition information generation models based on multimodal; generates object intent information for the target object based on the model set, the at least one segmented image sequence and the video and voice; in response to determining that the object intent information is in the abnormal intent information table, annotates the security sub-video corresponding to the video range with the target object to generate an annotated video; and sends the annotated video and the object intent information to the alarm terminal.

[0146] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor including a first acquisition unit, a cross-frame processing unit, a first generation unit, a segmentation unit, a second acquisition unit, a second generation unit, a labeling unit and a sending unit. Among them, the names of these units do not constitute a limitation on the units themselves in certain cases. For example, the sending unit may also be described as "a unit that sends the above-mentioned labeled video and the above-mentioned object intention information to the alarm terminal."

[0149] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0150] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for sending information, comprising: Obtain security video of the target monitoring area; Performing cross-frame extraction processing on the security video to generate at least one frame image sequence; Generate a video range for the target object according to the at least one frame image sequence; According to the video range, performing image sequence segmentation on each frame image sequence in the at least one frame image sequence to generate a segmented image sequence, thereby obtaining at least one segmented image sequence; Acquire the video and voice in the security video within the video range and a distributed model set, wherein the model set includes: a main node object intention recognition information generation model based on multimodality, and a plurality of sub-node object intention recognition information generation models based on multimodality; generating object intent information for the target object based on the model set, the at least one segmented image sequence, and the video speech; In response to determining that the object intention information is in the abnormal intention information table, marking the security sub-video corresponding to the video range with a target object to generate a marked video; The annotated video and the object intention information are sent to an alarm terminal.

2. The method according to claim 1, wherein: The cross-frame extraction process is performed on the security video to generate at least one frame image sequence, including: Acquire a preset frame extraction duration sequence, wherein the frame extraction durations in the frame extraction duration sequence decrease from left to right; According to the maximum frame extraction duration in the frame extraction duration sequence, the security video is divided into video segments to generate a security sub-video sequence; For the target frame extraction duration in the frame extraction duration sequence, the following frame extraction steps are performed: Determine the previous frame extraction duration corresponding to the target frame extraction duration; Determine a plurality of frame image subsequences corresponding to the previous frame extraction duration as a plurality of first frame image subsequences; Extracting a target number of frame images from each security sub-video in the security sub-video sequence to generate a frame image subsequence as a second frame image subsequence, and obtaining a plurality of second frame image subsequences, wherein the target number is the same as the numerical value of the target frame extraction duration, wherein a second frame image subsequence in the plurality of second frame image subsequences and a first frame image subsequence in the plurality of first frame image subsequences have a one-to-one time correspondence, and a frame image coincidence degree between each frame image included in each second frame image subsequence and each frame image included in the corresponding first frame image subsequence is greater than a preset coincidence degree; In response to determining that the target frame extraction time is the minimum frame extraction time, determining the plurality of second frame image subsequences as at least one frame image sequence; In response to determining that the target frame extraction duration is not the minimum frame extraction duration, the next frame extraction duration corresponding to the target frame extraction duration is used as the target frame extraction duration, and the frame extraction step is continued.

3. The method according to claim 1, wherein: The step of generating a video range for the target object according to the at least one frame image sequence comprises: For each frame image sequence in the at least one frame image sequence, the following first generating step is performed: Inputting each frame image in the frame image sequence into a target detection model for the target object to generate target detection information, thereby obtaining a target detection information sequence; Removing frame images corresponding to target detection information indicating that the target object does not exist from the frame image sequence, to obtain a frame image sequence after removal; Supplementing the removed frame image sequence with preceding and following continuous frame images to generate a supplemented frame image sequence; Determine an initial video range corresponding to the supplemented frame image sequence; A union process is performed on the obtained at least one initial video range to generate a union video range as the video range.

4. The method according to claim 1, wherein: The step of generating object intention information for the target object according to the model set, the at least one segmented image sequence and the video speech comprises: For each segmented image sequence in the at least one segmented image sequence, the following second generating step is performed: Generate an image feature information sequence for the segmented image sequence; According to the image set corresponding to each image feature information in the image feature information sequence, the video speech is divided into speech to generate a sub-speech sequence, wherein the sub-speech in the sub-speech sequence and the image feature information in the image feature information sequence have a time correspondence relationship; For each sub-speech in the sub-speech sequence, the following third generation step is performed: Performing audio preprocessing on the sub-speech to generate processed audio; Performing fast Fourier transform on the processed audio to generate a frequency domain signal; Using a Mel filter bank, generating a processed frequency domain signal corresponding to the frequency domain signal; Performing logarithmic processing on the processed frequency domain signal to generate a linear enhanced signal; Performing discrete cosine transformation on the linear enhanced signal to generate transformed audio feature information; According to the number of segmented images corresponding to the segmented image sequence, a corresponding sub-node object intention recognition information generation model is selected from the multiple sub-node object intention recognition information generation models as the target sub-node object intention recognition information generation model, wherein the output weight ratio corresponding to the target sub-node object intention recognition information generation model corresponds to the number of segmented images; Generate initial object intent information using the target sub-node object intent recognition information generation model according to the image feature information sequence and the obtained changed audio feature information sequence; The object intent information is generated according to the obtained at least one initial object intent information.

5. The method according to claim 4, wherein: The target sub-node object intention recognition information generation model includes: a plurality of feature extraction units, a plurality of feature fusion units and a recognition information generation layer; and The generating initial object intent information according to the image feature information sequence and the obtained changed audio feature information sequence by using the target sub-node object intent recognition information generation model comprises: Determine a time step sequence corresponding to the image feature information sequence; Select a target time step from the time step sequence; For the target time step, the following third generation step is performed: Acquire image feature information that has a temporal correspondence with a target time step from the image feature information sequence as target image feature information, and acquire changed audio feature information that has a temporal correspondence with a target time step from the changed audio feature information sequence as target changed audio feature information; Determine a feature extraction unit corresponding to the target time step as a target feature extraction unit, and determine a feature fusion unit corresponding to the target time step as a target feature fusion unit; Determining a previous time step corresponding to the target time step; Determine the fusion feature information corresponding to the previous time step as the target fusion feature information; Inputting the target changed audio feature information into an audio feature weight information generation model included in the target feature extraction unit to generate audio feature weight information; Multiplying the audio feature weight information and the target changed audio feature information to generate multiplied audio feature information; Inputting the multiplied audio feature information and the target image feature information into the attention mechanism model included in the target feature extraction unit to generate first attention weight information for the multiplied audio feature information and second attention weight information for the target image feature information; Inputting the first attention weight information, the second attention weight information, the multiplied audio feature information, and the target image feature information into the target feature fusion unit to generate fused feature information; Performing feature information splicing on the fused feature information and the target fused feature information to generate spliced ​​feature information; In response to determining that the target time step is the latest time step in the time step sequence, inputting the splicing feature information corresponding to the latest time step into the recognition information generation layer to generate initial object intention information; In response to determining that the target time step is not the latest time step in the time step sequence, taking the next time step corresponding to the target time step as the target time step, and continuing to execute the third generating step.

6. The method according to claim 4, wherein: The generating the object intent information according to the obtained at least one initial object intent information comprises: Determining at least one output weight ratio corresponding to the at least one initial object intention information; Classifying each of the at least one initial object intent information to generate a set of candidate object intent information, wherein each candidate object intent information has a corresponding object intent category; Determine each candidate object intention information in the candidate object intention information set, and perform the following fourth generation step: Determine an output weight ratio set corresponding to the candidate object intention information; Summing each output weight proportion in the output weight proportion set to generate a total output weight proportion; Determine whether there is candidate object intention information in the candidate object intention information set whose corresponding output weight ratio sum is greater than the first weight ratio; In response to determining that it exists, the candidate object intention information whose corresponding output weight ratio sum is greater than the first weight ratio is determined as the object intention information.

7. The method according to claim 6, wherein: The generating the object intent information according to the obtained at least one initial object intent information further includes: In response to determining that there is no such information, determining at least one candidate object intention information whose corresponding output weight ratio is greater than a second weight ratio from the candidate object intention information set as at least one target candidate object intention information, wherein the second weight ratio is greater than the first weight ratio; Using the master node object intent recognition information generation model, selecting the object intent information from the at least one target candidate object intent information; Determine, from the plurality of sub-node object intent recognition information generation models, at least one sub-node object intent recognition information generation model whose corresponding output intent information is not the object intent information; Perform model retraining for the at least one child-node object intent recognition information generation model.

8. An information sending device, comprising: A first acquisition unit is configured to acquire security video for a target monitoring area; A cross frame extraction processing unit, configured to perform cross frame extraction processing on the security video to generate at least one frame image sequence; A first generating unit is configured to generate a video range for a target object according to the at least one frame image sequence; a segmentation unit configured to perform image sequence segmentation on each frame image sequence in the at least one frame image sequence according to the video range to generate a segmented image sequence, thereby obtaining at least one segmented image sequence; A second acquisition unit is configured to acquire video and voice in the security video within the video range and a distributed model set, wherein the model set includes: a main node object intention recognition information generation model based on multimodality, and a plurality of sub-node object intention recognition information generation models based on multimodality; A second generating unit is configured to generate object intention information for the target object based on the model set, the at least one segmented image sequence and the video voice; a labeling unit configured to, in response to determining that the object intention information is in the abnormal intention information table, label the security sub-video corresponding to the video range with a target object to generate a labeled video; The sending unit is configured to send the annotated video and the object intention information to an alarm terminal.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images